All my fellow team-mates have fast workstations: quad core, 8 gigs of memory. Yay! BUT…..running our full test suite takes about 45 minutes. Boo! It’s a mix of Cucumber+webrat integration tests and unit tests. If you look at the cpu activity it doesn’t even spike a single core during the test. Memory consumption stays practically flat. That’s an extremely poor use of all that computing power. No wonder, all test are run sequentially. In our multi-core age that’s soo 90′s.
The solution is obvious: you need to parallelize the tests. Every integration test needs a dedicated environment to able to get predictable results. For most integration test this means exclusive access to resources like your database (Mysql), memory caching (Memcached) and/or full text search solutions (Sphinx | Solr). You can design your tests to be collision free, but like most multi-threaded programming that uses shared resource it’s quite difficult to get it right. And debugging weird threading issues will make you want to put pencils in your eyes. Trust me on that.
A more efficient way of creating a dedicated environment for every test is the use of virtual machines (vm). You replicate your integration test environment on a vm. Make several clones and your now have a pool of vm’s that can run your tests in parallel and guaranteed exclusivity.
The hard part of this solution;
- Cucumber and the unit test runner need to be modified to run tests distributed.
- Non-hypervisor virtualisation systems like Virtualbox and VMWare Server introduce a significant performance overhead. Hypervisor systems require a dedicated box.
- Provisioning of virtual machines can be a chore. Solutions like Vagrant can help with that.
But it will be worth it. Your CPU cores are worth it.