<< Death by developer | Home

In container vs out of container testing

Ever since I started with Test Driven Development back in early 2004 I have always advocated out-of-container testing and regarded any test scheme that requires deployment before execution as flawed.
I still think this is largely true and there are many many good reasons why having a fast, automated and comprehensive test suite which runs without going near the full stack is essential.
I have come to believe that this is not the whole story though. Testing by humans is an expensive and time consuming activity, often executed poorly and with debatable results. Often the best thing you get out of a test team is somebody to blame if it doesn't work in production. That sounds harsh but how many times have you heard the developments team response to a defect be 'what the hell were the testers doing for two weeks?'?
If you really want to be able to remove humans from the equation then unless you are very confident that nothing in your execution environment can affect the functionality of your software then you need to create automated tests which run against a deployment in-container.
The added advantage to in-container tests is that they can be used to seed load tests which reflect the user stories being delivered rather than some cobbled together random sample by the expensive performance consultant the week before go live.
We are writing in container tests for a RESTful web service. The tests are developed using a home rolled framework designed to be as readable as possible to the test and analysis teams. The tests are driven by the user stories being delivered and are very easy to validate against the API documents we publish.
Tags :

Add a comment Send a TrackBack