My team is responsible for several web service APIs that are used by external parties. The standard pattern for educating users about these interfaces was to provide an API document which listed the calls possibly with the occasional example. I have always found this approach to be less than useful and not particularly accessible.

I prefer an approach where the API specification is very much example based. This is makes more sense for RESTful web services than a programmatic API as there is no accepted mechanism for declaring such an interface. The same is not true if you are declaring, for example, a Java interface.

We supplement the document with JMeter projects pointed at a reference system which allow these interfaces to be exercised by the consumer and provides a step by step demonstration.

This has been an excellent mechanism which has resulted in very quick uptake of our interfaces but yesterday I was provided with an example of a potential pitfall:

We provided an API which upgraded widgets. Version was supplied with the manifest of the software package being supplied and was used again in the query string to the upgrade service. The reference data supplied by the system put the current version into a title metadata field so that testers could easily see that the upgrade had been formed. The version in the title matched exactly the title in the manifest.

Weeks after delivery and demonstration it stopped working. Why? The client was using the title rather than the metadata it was supposed to. Since we had used the same value for both the end to end use case appeared to work. As soon as they started using real data where the title was no longer equal to the version everything broke.

A valuable lesson here for me to relearn (I have hit it before in similar contexts). Always make sure reference and test data attributes are not interchangable with each other without an error occurring (and being detected). I should have made the title attribute 'The version of this widget is <x>'. That would have steered the client developer away from error and would have caused a validation error then first time he tried to submit it to our application.

My current client is a major telco. They have been developing high qaulity software for decades (literally). They have heavily entrenched waterfall process. They are used to doing everything to the letter of their methodology. When I first came on site two years ago I used to dispair of them ever doing anything agile.

Today, they seem to be doing agile pretty well. Its not been a universially posiitve experience and there are plenty of non-agile values and practices still floating around. On the whole though I have been impressed that rather than seeing Agile as an excuse not to write documents, they have embraced the disipline required by a leaner software process.

I have worked with clients who were much more reactive and used to being at the cuting edge. They embraced agile becuase it appeared to legitmise the lack of structure in their approach. They produced poor software when they produced up front designs and they continued to produce poor software when the developers were 'let off their leashes'.

Upon reflection I think my current client, with its long history of rigour and application of best practice has made far better use of the agile techniques. Maybe its becuase they continue to apply disiplince. Possibly its becuase they are very aware of the dangers of a less formal approach and are always trying to mitigate any risk that not having the waterfall in place might bring.

Ever since I started with Test Driven Development back in early 2004 I have always advocated out-of-container testing and regarded any test scheme that requires deployment before execution as flawed.
I still think this is largely true and there are many many good reasons why having a fast, automated and comprehensive test suite which runs without going near the full stack is essential.
I have come to believe that this is not the whole story though. Testing by humans is an expensive and time consuming activity, often executed poorly and with debatable results. Often the best thing you get out of a test team is somebody to blame if it doesn't work in production. That sounds harsh but how many times have you heard the developments team response to a defect be 'what the hell were the testers doing for two weeks?'?
If you really want to be able to remove humans from the equation then unless you are very confident that nothing in your execution environment can affect the functionality of your software then you need to create automated tests which run against a deployment in-container.
The added advantage to in-container tests is that they can be used to seed load tests which reflect the user stories being delivered rather than some cobbled together random sample by the expensive performance consultant the week before go live.
We are writing in container tests for a RESTful web service. The tests are developed using a home rolled framework designed to be as readable as possible to the test and analysis teams. The tests are driven by the user stories being delivered and are very easy to validate against the API documents we publish.