<< Tale of two SCRUM stand ups | Home | Concrete problems when developers opt out of TDD >>

Anti-pattern: The release vehicle.

At my current client site you cannot get a piece of compiled code into production unless you can find an appropriate 'release vehicle', i.e. a planned high ceremony release of the component which has been officially prioritised, scheduled and funded. (Note: The same does not apply to non-compiled code such as JSPs or XML templates containing complex XPath expressions).

Somebody very clever, who probably had a beard (Grady Booch?), once said that "Regular releases into production are the lifeblood of the software development process.". I agree. My current client also seems to be in agreement but cannot extract themselves from the constraints their existing processes.

The client in question has a successful agile adoption. Walking round the development teams you see task boards, burn downs and SCRUM meetings. Go to a management meeting and you'll hear them talk about two week iterations and the importance of continuous integration. At a strategic level, the organisation (which is very large) is still waterfall orientated. This has implications for the way in which work is financed. Funds for the development, testing and deployment of a certain application are released on waterfall inspired milestones. This, in conjunction with a legacy of long development cycles has led the this 'release vehicle' anti-pattern.

The organisation has an unwillingness to make a deployment of a component into production unless there is named and funded change request which covers its release. Activities within development, possibly funded internally as 'business as usual' do not have such CRs. Therefore, a development activity such as refactoring for technical debt reduction or improving performance might get engineering buy in but will not get released into production until some CR happens to touch the same application.

It is common to see refactorings made which then sit in source control for literally months as they wait for an excuse to go live. Medium to low priority defects or useful CRs which lack very high prioritisation from marketing never get executed because the programme manager does not have a release identified for the change.

The application suite can appear inert to external parties as it takes a considerable period for changes to make it through the full release cycle. This erodes confidence. If I was a product owner and saw that a team was taking six months to execute my minor change I am not going to be inclined to believe that the same team can turn around my big important changes quickly. I am going to be looking for other mechanisms to get my changes into production and earning money quickly. Once I find a route that works I am going to keep using it.

Why do people like the release vehicle?
  • It is the way the whole software lifecycle as exposed to the rest of the organisation works. The QA team don't test a component unless they have funding from marketing. Marketing won't be paying for something that has no role in a prioritised proposition. The Operations team won't support the deployment actives for our component if they don't have the cash from the same marketing team.
  • It looks like it is easier to manage for PMs. Releases (because they are infrequent) are a big deal, involve lots of noise, planning, disruption to everyday working pattern.
  • It reduces the infrastructure costs. It costs resource to make a release unless every aspect including testing and operational deployment is fully automated (and even then there is potential cost, dealing with failures etc.). It costs resource to automate a manual build process. Engineers appreciate that fully automated build processes are a priority because in the end they reduce costs and increase agility. It is that age old problem of trying to convince not just the build team, but the build team's manager and the build team's manager's manager that it is worth diverting resource in the short term to fix a problem in order to make a saving in the long term.
** This is a symptom of our strategic failure to get agile adopted beyond the development group. Until we do so, we will continue to hit these sort of issues.

What we should do instead:

We should schedule frequent (bi-weekly, ideally more frequent) updates in production from the trunk of source control for every component. We should not need an excuse for a release. The release process should be as cheap as possible, i.e. automated build, regression test, deployment and smoke test. The code in the trunk is supposed to always be production ready and the automated tests should keep it that way.

If we achieve this we should:
  • Reduce complexity in branch management (no merging changes made months ago).
  • Avoid a massive delay between development and deployment which is not cost effective and makes support very hard.
  • Increase our perceived agility and responsiveness.
  • Enable refactoring to improve non-functionals (stability, latency, dependency versions, capacity).
  • Prevent a release from being a 'special occasion' which requires significant service management ceremony.
If you release all the time everybody knows how to release. If you release twice a year every release involves re-education of the teams involved on deployment, load testing, merging etc.. etc. This increases the cost and risk that it fails.

Note: Having frequent, regular, low ceremony releases is greatly eased by having a fully automated build and deploy process but you can have one without the other. As stated above, having such a build process makes regular deployments to production cost effective but is an enabler rather than the justification for this change to working practice.


Re: Anti-pattern: The release vehicle.

Great example of a widely used/abused practice! I would like to add it to a design pattern catalog I am just now compiling: Deployment management design patterns for DevOps.

Re: Anti-pattern: The release vehicle.

Thanks for the positive feedback! Feel free to use the posting in any way you like. If you need any further information or clarifications just ask.

Andrew.

Re: Anti-pattern: The release vehicle.

Very interesting ideas! Just a few quick thoughts: Might this model actually disincentivize larger more invasive change that may sometimes be necessary for more strategic architectural changes? I'm all for automating as much as is possible (as opposed to manual qa) and having the ability to have deployable/tested version of a product on a regular basis -- but actually deploying on a very regular basis seems as if it may become a road block to large change after a product has been initially released. Also, I would imagine that if you cannot guarantee all consumers of a product are working with the same product version, having a new version every two weeks could become a nightmare for a support organization (imagine a support rep needing a hotfix for 20 releases of a product instead of 1 or 2). In my mind there are clear benefits to the model you suggest here but there are also pitfalls. Thoughts?

Re: Anti-pattern: The release vehicle.

I definitely agree but I could I suggest that you have a better definition of a 'release vehicle' to indicate that each release vehicle is a one-off. To borrow your transportation analogy, the solution you have suggested is a 'release train' where you have a series of regular releases scheduled well in advance that you can use to deploy the artefacts produced at the end of each sprint.

Re: Anti-pattern: The release vehicle.

This whole Article hits the NAIL ON THE H.E.A.D. I am so tired of what's happening in the software industry. I really am exhausted with all the crap. All i want to do is get stuff out there. Stuff that really works. Stuff that'll make the customer happy, quickly. I HATE process. A bit is ok. But it always goes too far in these large crappy companies. I'm not an automaton. Parts of this article are worth their weight in gold. Thank you for writing this. I wish people could understand.

Add a comment Send a TrackBack