<< July 2009 | Home | September 2009 >>

Value and cost in hardware and software

Neal Ford's presentation on emerging architecture contained a reference to the controversial Intel practice in the 1990s regarding the 386-SX math co-pro. Intel were producing two variants of the 386 chip at the time:
  • The 386-DX which featured a math co-processor on board which allowed faster execution of floating point maths required in financial applications, graphics packages etc..
  • The 386-SX which was cheaper but did not feature the math co-processor and therefore was less 'powerful'. It had enough to differentiate itself from the previous generation of 286 chips but was regarded as distinctly inferior to its more expensive sibling.

This all sounded fair enough until it became public that the 386-DX and 386-SX shared the same manufacturing process including the construction of the maths co-processor. Where the process differed was that at some point the math copro in the SX chip was destroyed via some mechanical process. Suddenly the perception of the SX went from a lower spec product to a broken product. In Neal Ford's presentation he described the whole process as Intel selling customers their trash, like the SX was a defective unit being offloaded to unsuspecting users as a working chip.

At a very low technical level Neal's statement is true but not at a commercial or practical level. Intel engineers were given a change in requirements by their marketing department: Produce a chip that is going to enter the budget market to complement but not fully compete against the 386-DX. They looked at their system and determined the most efficient way to achieve this end was to 're-configure' existing 386-DX chips. This was likely much much cheaper than setting up a whole new production line and testing the brand new chip it produced. To do otherwise would be against the pragmatic engineering ideals that Agile is supposed to champion. If you flip the argument around and say: Should we redesign from scratch the entire process to achieve the same end result but at much higher cost just so that we claim that the chip contains no parts that it doesn't need. Maybe we find this so objectionable because a chip is a tangible entity and we are used to associating (rightly or wrongly) the cost of raw materials and manufacturing with the value of such items. Maybe we don't factor in the cost of design and marketing, which I suspect are massive for a complex, consumer electronic product like a cutting edge CPU.

This pattern raised its head again, but with less fuss, a couple of years ago when HP shipped high end servers with multiple CPUs. Some of the CPUs were disabled upon delivery. If the customer's processing requirements increased over time (which they always do) then they could pay HP who could then remotely enable the additional processors without the customer incurring the cost of an engineer on site, downtime for installation etc. etc.. Again, this early step towards todays processing on demand cloud computing concept raised some eyebrows. Why should customer's pay for something that was going to cost the supplier nothing? Again, this is a preoccupation with the physical entity of the manufactured component. If the additional CPUs had been sitting idle in a HP server farm rather than at the customer's site and purchasing them involved work being sent across the network my suspicion is that nobody would have had any objections.

We use a UML design tool at my current client site called Visual Paradigm. It has a number of editions, each with a different cost. It has a very flexible license purchase system which we have taken advantage of. We have a large number of standard level licenses because the features that this edition gives will support most of our users most of the time. Occasionally we need some of the features from a more expensive edition. Its not that only one or two individuals require these features, we all need them, just very occasionally. The Visual Paradigm license model supports this beautifully. We have a couple of higher edition licenses. On the rare occasion that users need the extra features they just start the program in the higher edition mode. As long as no other users connected to our license are using the higher edition at that time, there is no issue. The similarity with the examples above is that there is only one installation. We don't need to install a different program binary every time we switch edition. We love this as it makes life easy. I am sure Visual Paradigm like it as well as it simplifies their build and download process.

Two me the two scenarios, software and hardware appear pretty much identical. Everybody appreciates that the cost of creating a copy of piece of software is so close to zero that it is not worth worrying about. Therefore we don't mind when a supplier gives us a product with bits disabled until we make a payment and get a magic key. It's harder to think of hardware in the same way, that the build cost doesn't matter. That there might be no difference in manufacturing costs for two products with very different customer prices. The cost of delivering the product, like delivering software, includes massive costs which have nothing to with creation of the physical artifact.  

Maybe this wasn't the point in the above presentation but I guess the thing that startled me was that my natural inclination was to immediately associate the value with the tangible item. In my head this is all getting mixed up with free (as in speech) software and the idea that it is unproductive and unethical to patent / own / charge for ideas.

Agile 2009

I have spent the week at the Agile2009 conference in Chicago. This annual conference, now in its eighth year, is the premier international gathering for agilsts. It caters for a whole spectrum of experience from new comers to the discipline to the gurus who are leading the way.

I attended some very thought provoking sessions as well as presenting my own experience report on techniques for technical architecture in an agile context. My colleague from Valtech US, Howard Deiner, battled hardware and network issues to present a well received demonstration of Continuous Integration. Both sessions got reasonable attendances in the face of stiff competition from competing presentations being held in parallel. Both received very positive feedback from attendees (mine scored 80% for command of topic and 78% overall). I got lots of positive feedback for my session in conversations with conference attendees throughout the week. This was very much appreciated.

My presentation is backed up by an IEEE Report which was published in the conference proceedings. The report's premise is that incumbent waterfall software development processes force technical architects into a position of isolation and ineffectiveness (the ivory tower). The challenge I (any many many other TAs) have faced is how to deliver the guarantees of technical correctness and consistency that clients (especially those moving from waterfall to agile) demand when some of the most widely used conventional techniques for architecture have been discredited. I am thinking primarily of the emphasis placed on up front detailed design and architectural review.

The report details architectural problems during scale up of a previously successful agile project. The report then describes and evaluates a number of techniques employed on the project to deliver the technical architecture without ascent of the ivory tower. The conclusions include the justification that documentation is not an effective tool for technical governance and that the architect must target activities which bring them closer the the actual implementation. This mirrors Neal Ford's point in his Emerging Architecture presentation that we need to accept that the real design is the code, not the summaries and abstractions of the code presented via the numerous tools (UML, narrative documents, whiteboard sessions) at our disposal. Other conclusions include the identification of automated tests as an architect's, not just a tester's, most effective tool for delivering a correct solution. The paper also identifies that soft skills around communication and people management, often the anathema of the conventional architect are critical to success. Finally the report concludes that utilizing the most cost effective techniques (rather than just the most technically powerful) were key. (That does not mean you cannot justify the use of expensive techniques, just that they may only be justifiable on the most important components in the system).

Agile 2009 was a great balance of real world experiences (such as my session) and more philosophical, academic sessions. There also the chance to listen to some insightful keynotes and take part in some exciting expert sessions which challenged the way we work. It is always easier to learn in a community of professionals with real experience and this was definitely the case at this conference. I learned as much over dinner and in break out sessions as I did in the formal seminars.

I am going to blog what I learned in some of sessions in the next couple of days, possibly earlier as I am stuck at Chicago O'Hare for eleven hours after a 'mechanical issue' with our plane!

Use of best of breed open source

Over the last two years of my current project I have noticed a recuring pattern. On several occasions we have identified an implementation pattern which commonly appears on many enterprise projects. That pattern is common enough that there is a well known (i.e. at least one person in the team has heard of it) open source solution which appears to be recognised by the community as being best of breed. In order to reduce risk and increase our velocity we use that open source component, possibly making changes to the design to more effectively incorporate the ready to run package. The theory (and one I fully buy into) being that be using the open source library we free up time to concentrate on the part of our solution which are truly unique and require bespoke software.

The recurring pattern I see is that on at least four occasions the best of breed package has proven to be severely sub-optimal. What is worse is that most of the time these deficiencies occur when we move into high volume load test in a cluster. It seems only then that we discover some limitation. Typically this is caused by a particular specialism required for our application which then exercises some part of the library that is not as commonly utilised as others and therefore less stable. Some times the limitation is so bad that the library has to be refactored out before launch and other occasions the issue becomes a known restriction which is corrected at the next release. All of the significant refactorings have involved replacement of the large, generic, well known library with a much smaller, simpler, bespoke piece of code.

I am undecided whether this is a positive pattern or not. On one hand using the standard component for a short period helped us focus on other pieces of code. On the other, the identification of issues consumed significant resource during a critical (final load test) period. The answer probably is that it is okay to use the standard component as long as we put it under production stresses as quickly as possible. We then need to very carefully take account of the effort being consumed and have an idea of the relative cost of an alternative solution. When the cost of the standard component begins to approach the cost of the bespoke then we must move swiftly to replace it. The cost should also factor in maintenance. We need to avoid the behaviour where we sit round looking at each other repeating "This is highly regarded piece of software, it can't be wrong, it must be us." for prolonged periods (its okay to say this for a couple of hours, it could be true). I used to work for a well known RDBMS provider. I always felt that the core database engine was awesomely high quality and that anybody who claimed to have found a defect was probably guilty of some sloppy engineering. I knew however, from painful experience, that you did not have to stray far from the core into the myriad of supported options and ancillary products to enter a world of pure shite. The best of breed open source components are no different.

Some of the problem components:

ActiveMQ (2007) - We thought we needed an in memory JMS solution and AcitveMQ looked like an easy win. It turned out that at that release the in-memory queue had a leak which required a server restart every ten to fifteen days. It also added to the complexity of the solution. Was replaced by very few lines of code utilising the Java 5 concurrency package. I would still go back to it for another look, but only if I was really sure I needed JMS.

Quartz (2007) - The bane of our operations team's life as it would not shutdown cleanly when under load and deployed as part of a Spring application. Replaced by the Timer class and some home grown JDBC.

Quartz (2009) - Once bitten, twice shy? Not us! The shutdown issue was resolved and we needed a richer scheduling tool. Quartz looked like the ticket and worked well during development and passed the limited load testing we were able to do on workstations. When we moved up to the production sized hardware and were able to put realistic load through we discovered issues with the RAMJobStore that were not present with the JDBC store (which we didn't need). It just could not cope with very large (100 000+) numbers of jobs where new jobs were being added and old ones deleted constantly.

Security compliance without empirical evidence

As the project nears the final delivery I am having to complete a statement of compliance for group security (did you feel a shiver as you read that, it was justified). One of the values I have tried to instil is that we don't do any documentation or formal design with no clearly defined audience. When we do identify a subject that does need to be formally recorded I am keen that it is done well. The interactions between components OAUTH  is one of those few key areas.

The OAUTH sequence diagram was correctly checked into the UML repository and was pretty good. Looking at it I was suddenly struck but a deep sense of unease. How was I supposed to know whether the implementation sitting on our servers bears any relation to the work of art being displayed on my screen? What value is my statement without real knowledge that we are secure? I know this is is something I have known for years and bang on about to anybody who will listen but it was a startling moment to be sitting there looking at the design and being asked to make a formal statement about its realisation without empirical evidence. I already knew from an audit of the acceptance test suite (end to end, automated, in-container tests) that one of the omissions was anything that exercised OAUTH. I decided that one of my priorities for tomorrow will be the completion of that test and that I wont be making a statement of compliance without it.

Valtech Blog is live

The Valtech blog is live, yay! It incorporates posts from various Valtech consultants and covers all things Agile and software development in general. My submissions are featured so no more swearing...

SpringSource in the cloud?

SpringSource has been acquired by VMWare!

Interesting that VMWare paid more for SpringSource than Redhat paid for JBoss, even in the midst of the recession.

Spring has got to be the tool of choice for the development of enterprise Java applications right now. I wonder if in the near future deploying Spring applications to the VMWare cloud (or using Spring beans already deployed in the cloud) will be as easy as deploying to Tomcat or Jetty?