<< December 2008 | Home | February 2009 >>

Removing database dependancy from tests

At a recent internal tech day a project I had previosuly been involved in made a short report about some serious issues they were having with their test suite. They made heavy use of DBUnit to create a complex data set in an Oracle database. This was being setup and torndown for a great many of the tests. The result was the CI build could not run the test suite automatically. Instead the lead developer would manually initiate a test run overnight (I believe it was actually taking the entire night to run). Various of my collegues had had similar problems and suggested various solutions that had been succesful on their projects. These included the use of a dedicated super fast integration test machine and a the running of tests in parallel (hard to do for database dependent tests - there is often contention on the number of database connections allowed and any tests which extend beyond transaction boundaries interfere with each other).

The solution employed on my project (and advocated in many places, see links at bottom) successfully overcame these issues as is as follows.

On my project, we have a large set of unit tests (close to 2000 tests). These are pure unit tests in that there is no container (not even spring) and everything is mocked other than the class under test. There is no database dependency. These tests execute in under 5 seconds on our development PCs (which are dual quad core ubuntu boxes). We also have a smaller, but very important, set of integration and regression tests which execute inside the spring container and some use as much of the application stack as is possible. Only external system dependencies are mocked or stubbed. These tests did, for the first year of the project, use the database.

What we found (and I experienced this on the above troubled project as well) was that these tests were too fragile to be included in the automated build. They execute too slowly, cannot be run concurrently (as they access the same db resource) and if the build system dies it can leave a messy database which can be difficult to clear up. Even after several attempts to build robust setup and teardown scripts we still could not get integration tests that were reliable enough to run automatically. An even bigger problem was that the integration test suite would not execute on the build machine in less than half an hour. We build every time somebody commits and track failures religiously. Individuals are pressured never to break the build. A thirty minute build is unacceptable for us. This meant the integration tests were only run manually and with the best will in the world, people forgot to run them before checking in and they very quickly started showing false negatives. The broken window syndrome kicked in and the set of working integration tests diminished rapidly. We wasted a lot of time fixing the test suite in the first 12 months of the project.

I did some analysis of defects and was able to highlight to the client that some serious defects had made it out of development and cost serious time and effort in system test and UAT because of the lack of end to end test coverage which should have been provided by the regression test suite. Using this an ammunition it was easy to justify spending some time finding a solution. There is a well understood solution to this sort of issue and it involves the use of an in-memory hash table database replacement. I got one of the developers to implement a version of the DAOs which wrote objects to a hash table instead of the real database. As far as the users of the DAO were concerned, the system was working as normal. We already had some specific integration tests which did test the DAO database implementation so we were very confident they worked and did not need to be tested all the time. The hashcode database is easy to setup (we always had a rule that all non DAO tests had to perform database setup using the DAO methods rather than using a script solution like DBUnit, which I dislike). Tearing down the hashcode DB is very very easy. The regression test suite which was so slow it was unusable when hooked into Oracle now executes in 12 seconds. It has no external dependencies and so has been included in our automated CI build with great success. The DAO implementation test suite continues to be on a manual basis. Come to think of it, I haven't run it for a few weeks so I bet it's broken!

I should stress as well that we partitioned the tests from day 1 (unit, acceptance, externally dependent, integration and regression). The unit tests were always fast and we never had to disable them in the CI build.

This pattern does not work so well if you have data in your database which gets there without going through your DAO layer (e.g. reference data loaded through scripts, bulk loads via SQLLDR). This means you do not have a ready made way of creating all test fixtures. I would advocate the creation of a test helper DAO in this case to create those additional data structures that you require.

See this book for details of these approach.