+972 52-548-6969
Fixing up the build process
There is a big problem in the RavenDB build process. To be rather more exact, there is a… long problem in the RavenDB build process.
As you can imagine, when the build process run for that long, it doesn’t' get run too often. We already had several runs of “let us optimize the build”. But… the actual reason for the tests taking this long is a bit sneaky.
To save you from having to do the math, this means an average of 1.15 seconds per test.
In most tests, we actually have to create a RavenDB instance. That doesn’t take too long, but it does take some time. And we have a lot of tests that uses the network, because we need to test how RavenDB works on the wire.
From that perspective, it means that we don’t seem to have any real options. Even if we cut the average cost of running the tests by half, it would still be a 30 minutes build process.
Instead, we are going to create a layered approach. We are going to freeze all of our existing tests, move them to an Integration Tests project. We will create a small suite of tests that cover just core stuff with RavenDB, and use that. Over time, we will be adding tests to the new test project. When that becomes too slow, we will have another migration.
What about the integration tests? Well, those will be run solely by our build server, and we will setup things so we can automatically test when running from our own forks, not the main code line.
Comments
Have you though about running your test suites in parallel? We're doing that quite heavily here. It's often difficulat to parallelize a single test suite, but if you have multiple of them you can easily run them in parallel.
MSBuild does a good job orchestrating that, there are a few caveats though (e.g. http://jorudolph.wordpress.com/2011/04/15/multiple-test-runner-scenarios-in-msbuild/)
I once had the idea of running a randomly chosen subset of 10% of the tests. That would cover most outrageous bugs during development.
How important is hitting the actual file system? Could you write to an in memory file system instead and sub in the real thing for a daily build?
Mocks could help with this ;)
It would be nice if you could specify in each test definition what kind of parallelism it could handle. Tests that just need an in-memory database with no http could probably run several at a time in parallel. Other tests, like those testing replication over specific http ports, would run serially. It would be better than forcing them all to run sequentially.
Making a build faster turns kittens into unicorns!
two questions here: 1. why the separate test project? can't you parameterize the test run, to only include the long-running tests when env=build-server or something of sorts?
lololo
two questions - 1 and 1.
by the time you're 35, and after having put two kids to bed and then reviewed and wrote code for a couple of hours, 11:30pm is late ..
Johannes, We have a lot of tests that require stuff that cannot be run concurrently. For example, all our tests that test the REST API require a port, etc.
Luke, We are a database, touching the FS is something that is quite important for us. An expensive test we have, for example, is what happens when we corrupt the files on the FS, how do we recover from that?
Ayende,
It seems Luke asked if you could use an in memory file system for such tests. You don't need a piece of metal to test the handling of corrupted files, right?
What I commonly do is have 2 "CI" builds. One that kicks off on check-in and runs in some short time span (say 5 mins or less). This is responsible for performing the most important quality checks against the changes (what is most important will be different from team to team, but it usually includes some sub-set of fast and/or important tests).
If/when that build succeeds another build gets kicked off immediately afterwards that runs all quality checks/tests, and this may take upwards of an hour.
This way the developer gets the most important feedback quickly, and then even more feedback shortly thereafter.
In complex projects if the 2nd build gets to be much more than an hour I may introduce a 3rd tier build that runs as a nightly build.
Comment preview