Delivering software - testing tag

Security compliance without empirical evidence

Mon, 17 Aug 2009 20:01:13 GMT

As the project nears the final delivery I am having to complete a statement of compliance for group security (did you feel a shiver as you read that, it was justified). One of the values I have tried to instil is that we don't do any documentation or formal design with no clearly defined audience. When we do identify a subject that does need to be formally recorded I am keen that it is done well. The interactions between components OAUTH is one of those few key areas.

The OAUTH sequence diagram was correctly checked into the UML repository and was pretty good. Looking at it I was suddenly struck but a deep sense of unease. How was I supposed to know whether the implementation sitting on our servers bears any relation to the work of art being displayed on my screen? What value is my statement without real knowledge that we are secure? I know this is is something I have known for years and bang on about to anybody who will listen but it was a startling moment to be sitting there looking at the design and being asked to make a formal statement about its realisation without empirical evidence. I already knew from an audit of the acceptance test suite (end to end, automated, in-container tests) that one of the omissions was anything that exercised OAUTH. I decided that one of my priorities for tomorrow will be the completion of that test and that I wont be making a statement of compliance without it.

In container vs out of container testing

Mon, 18 May 2009 21:57:18 GMT

Ever since I started with Test Driven Development back in early 2004 I have always advocated out-of-container testing and regarded any test scheme that requires deployment before execution as flawed.
I still think this is largely true and there are many many good reasons why having a fast, automated and comprehensive test suite which runs without going near the full stack is essential.
I have come to believe that this is not the whole story though. Testing by humans is an expensive and time consuming activity, often executed poorly and with debatable results. Often the best thing you get out of a test team is somebody to blame if it doesn't work in production. That sounds harsh but how many times have you heard the developments team response to a defect be 'what the hell were the testers doing for two weeks?'?
If you really want to be able to remove humans from the equation then unless you are very confident that nothing in your execution environment can affect the functionality of your software then you need to create automated tests which run against a deployment in-container.
The added advantage to in-container tests is that they can be used to seed load tests which reflect the user stories being delivered rather than some cobbled together random sample by the expensive performance consultant the week before go live.
We are writing in container tests for a RESTful web service. The tests are developed using a home rolled framework designed to be as readable as possible to the test and analysis teams. The tests are driven by the user stories being delivered and are very easy to validate against the API documents we publish.

Removing database dependancy from tests

Thu, 29 Jan 2009 22:54:00 GMT

At a recent internal tech day a project I had previosuly been involved in made a short report about some serious issues they were having with their test suite. They made heavy use of DBUnit to create a complex data set in an Oracle database. This was being setup and torndown for a great many of the tests. The result was the CI build could not run the test suite automatically. Instead the lead developer would manually initiate a test run overnight (I believe it was actually taking the entire night to run). Various of my collegues had had similar problems and suggested various solutions that had been succesful on their projects. These included the use of a dedicated super fast integration test machine and a the running of tests in parallel (hard to do for database dependent tests - there is often contention on the number of database connections allowed and any tests which extend beyond transaction boundaries interfere with each other).

The solution employed on my project (and advocated in many places, see links at bottom) successfully overcame these issues as is as follows.

On my project, we have a large set of unit tests (close to 2000 tests). These are pure unit tests in that there is no container (not even spring) and everything is mocked other than the class under test. There is no database dependency. These tests execute in under 5 seconds on our development PCs (which are dual quad core ubuntu boxes). We also have a smaller, but very important, set of integration and regression tests which execute inside the spring container and some use as much of the application stack as is possible. Only external system dependencies are mocked or stubbed. These tests did, for the first year of the project, use the database.

What we found (and I experienced this on the above troubled project as well) was that these tests were too fragile to be included in the automated build. They execute too slowly, cannot be run concurrently (as they access the same db resource) and if the build system dies it can leave a messy database which can be difficult to clear up. Even after several attempts to build robust setup and teardown scripts we still could not get integration tests that were reliable enough to run automatically. An even bigger problem was that the integration test suite would not execute on the build machine in less than half an hour. We build every time somebody commits and track failures religiously. Individuals are pressured never to break the build. A thirty minute build is unacceptable for us. This meant the integration tests were only run manually and with the best will in the world, people forgot to run them before checking in and they very quickly started showing false negatives. The broken window syndrome kicked in and the set of working integration tests diminished rapidly. We wasted a lot of time fixing the test suite in the first 12 months of the project.

I did some analysis of defects and was able to highlight to the client that some serious defects had made it out of development and cost serious time and effort in system test and UAT because of the lack of end to end test coverage which should have been provided by the regression test suite. Using this an ammunition it was easy to justify spending some time finding a solution. There is a well understood solution to this sort of issue and it involves the use of an in-memory hash table database replacement. I got one of the developers to implement a version of the DAOs which wrote objects to a hash table instead of the real database. As far as the users of the DAO were concerned, the system was working as normal. We already had some specific integration tests which did test the DAO database implementation so we were very confident they worked and did not need to be tested all the time. The hashcode database is easy to setup (we always had a rule that all non DAO tests had to perform database setup using the DAO methods rather than using a script solution like DBUnit, which I dislike). Tearing down the hashcode DB is very very easy. The regression test suite which was so slow it was unusable when hooked into Oracle now executes in 12 seconds. It has no external dependencies and so has been included in our automated CI build with great success. The DAO implementation test suite continues to be on a manual basis. Come to think of it, I haven't run it for a few weeks so I bet it's broken!

I should stress as well that we partitioned the tests from day 1 (unit, acceptance, externally dependent, integration and regression). The unit tests were always fast and we never had to disable them in the CI build.

This pattern does not work so well if you have data in your database which gets there without going through your DAO layer (e.g. reference data loaded through scripts, bulk loads via SQLLDR). This means you do not have a ready made way of creating all test fixtures. I would advocate the creation of a test helper DAO in this case to create those additional data structures that you require.

See this book for details of these approach.