Empirical Management

Aliases:

… A company that has been following the Waterfall methodology wants to begin delivering software better, faster, and cheaper.

•••

A number of organizations use the Waterfall processes for software development. Such processes require that all requirements be gathered at the start of a project. Managers in such organizations implicitly assume that:

• Requirements can be well defined by the users of a non-existent product who know what they want.
• Requirements are static and will not change.

The subsequent creation of a detailed design then supposes that the solution to a poorly-stated problem (in the previous stages of the SDLC) can be defined with precision.

The waterfall method does not explicitly encourage prototyping at the design stage. This implies that design can be conducted in a purely abstract space, or put another way that trial rarely leads to error.

Eventually, the team starts to implement the design that has not yet been proven. The underlying assumption is that the technology will all fit nicely into place when the time comes.

More often than not, projects flounder at the integration stage as the previous implicit assumption is rarely true.

Companies that follow a disciplined software development process to ensure predictability and efficiency usually end up with projects that fail outright (cancelled prior to completion or delivered and never used) or are challenged (fewer features delivered than agreed and/or late).

Details on the state of project delivery can be found on the Standish Group Web site or from an interview with Jim Johnson, the Chairman of the Standish Group. The Standish Group has been conducting annual surveys of hundreds of companies and projects for the past dozen years.

Therefore:

Use “agile” management techniques that are characterized by quickness, lightness, and fluidity (allow for changes in the business environment with minimal fuss).

While agile techniques vary in practices and emphasis, they share common characteristics, including iterative development and a focus on interaction, communication, and the reduction of resource intensive intermediate artifacts. In addition all agile methods recognize that software development is inherently creative in all its aspects.

•••

Defined processes can be characterized as “Plan, Plan, Plan, Do” processes. They can be characterized by:

1. Plan what you expect to happen throughout the project
2. Enforce that what actually happens is the same as what had been originally planned
3. Use change control mechanisms to manage changes in users needs and business requirements

Agile processes on the other hand spread the planning throughout the project lifecycle. Rather than heavy up-front planning, agile methods rely on frequent course correction during the project.

Because traditional methodologies invest so much in up-front planning and up-front design, they are naturally biased against change. Waterfall based methodologies assume that variations are the result of errors (in the definition of requirements) and these incorrectly specified requirements lead to incorrect designs and eventual project troubles.

Agile methods, however, assume that the external environment causes critical variations. Agile methods welcome change. These processes adapt and thrive on change, even to the point of changing themselves. Changing requirements often imply a better understanding of the market, the product, and the technology being used. These changes allow the company and team to fine-tune what they are doing and delivering.

• Developing in iterations allows the team to adapt quickly to changing requirements.
• Working in close location and focusing on communication helps teams make quick decisions and act on them immediately, rather than wait on written communication.
• Reducing intermediate artifacts that do not add value to the final deliverable frees resources that can be devoted to the development of the product itself which can be finished sooner.

The problems due to the Waterfall process are listed below (a non-exhaustive list):

• Projects are significantly “challenged” — late and with reduced scope — and the problem is partially attributable to a subjective assessment of project status.
• Pushing integration and user feedback to the end ensures that deficiencies and errors will not be caught soon enough to be fixed.
• Business perception of long lead times for delivering value.
• Arbitrary order of feature delivery — no focus on delivering high business value items first.
• Applications delivered do not meet users’ needs because feedback from users and subsequent adaptation is delayed due to the sequential nature of the process.
• Focus on hand-offs (comprehensive up-front) rather than collaboration and quicker delivery. Written communication suffers from the inability to quickly clarify what was stated; this ambiguous and imprecise communication leads to undetected inconsistencies in requirements, designs, and implementations.
• Failure to attack risk early. Waterfall projects delays the mitigation of risks — « Big Bang » implementation and integration effort just before delivery date. Problems unearthed then are usually very expensive to fix as they have a cascading effect on other features, which also need to be modified.

Note that the problem is not with the Waterfall method itself. Defined processes have been used for decades with success in the engineering disciplines. The problem is the improper selection and inappropriate use of the method — you can use a fly-swatter and a baseball bat to strike objects, but the implement used varies depending on the circumstances. Any process properly applied in the correct domain will result in the desired outcome; our problem is that traditional predictive methods do not work well in most software development domains. Unlike most manufacturing and engineering disciplines, software development is more akin to continuous learning and invention. When you can’t define things well enough up-front so that they run unattended and produce repeatable, acceptable quality output (activities are not predictable, are non-linear, and are too complex to define in repeatable detail); the only solution is to control the process through inspection and adaptation.