I almost made it a year without a blog post.

I recently started using Vagrant to test our auto-provisioning of servers with Puppet. Having a simple-yet-configurable system for starting up and accessing headless virtual machines really makes this a much simpler solution than VMware Fusion. (Although I wish Vagrant had a way to take and rollback VM snapshots.)

Unfortunately, as soon as I tried to really do anything in the VM my Mac would completely bog down. Eventually the entire UI would stop updating. In Activity Monitor, the dreaded kernel_task was taking 100% of one CPU, and VBoxHeadless taking most of another. Things would eventually free up whenever the task in the VM (usually apt-get install or puppet apply) would crash with a segmentation fault.

Digging into this, I found an ominous message in the VirtualBox logs:

AIOMgr: Host limits number of active IO requests to 16. Expect a performance impact.

Yeah, no kidding. I tracked this message down to the "Use host I/O cache" setting being off on the SATA Controller in the box. (This is a per-VM setting, and I am using the stock Vagrant "lucid64" box, so the exact setting may be somewhere else for you. It's probably a good idea to turn this setting on for all storage controllers.)

When it comes to Vagrant VMs, this setting in the VirtualBox UI is not very helpful, though, because Vagrant brings up new VMs automatically and without any UI. To get this to work with the Vagrant workflow, you have to do the following hacky steps:

1. Turn off any IO-heavy provisioning in your Vagrantfile
2. vagrant up a new VM
3. vagrant halt the VM
4. Open the VM in the VirtualBox UI and change the setting
5. Re-enable the provisioning in your Vagrantfile
6. vagrant up again

This is not going to work if you have to bring up new VMs often.

Fortunately this setting is easy to tweak in the base box. Open up ~/.vagrant.d/boxes/base/box.ovf and find the StorageController node. You'll see an attribute HostIOCache="false". Change that value to true.

Lastly, you'll have to update the SHA1 hash of the .ovf file in ~/.vagrant.d/boxes/base/box.mf. Get the new hash by running openssl dgst -sha1 ~/.vagrant.d/boxes/base/box.ovf and replace the old value in box.mf with it.

That's it. All subsequent VMs you create with vagrant up will now have the right setting.