I recently saw a question posed as to whether Agile is dead.
Agile isn’t dead. However, Agile no longer addresses the main bottleneck to delivering value. The bottleneck has shifted. Let’s look at why that is the case and to where the bottleneck has shifted.
First let’s understand what problem Agile was intended to solve. The Agile Manifesto is an artifact of a specific period of time and context. In the 1990s and early 2000s, software delivery problems were rampant (largely due to the heavyweight processes used). Many initiatives were outright failures, and it wasn’t unusual for products to be completed years late or with significant quality problems. Just getting software shipped out the door was the main bottleneck to being able to deliver business value.
The Agile Manifesto and frameworks like XP and Scrum were a response to this environment. Ideas like working software, iterative development, early integration and frequent delivery were emphasized in order to address this bottleneck of shipping software.
Software developers spent much of the next 15 years improving software delivery by developing and honing techniques like unit testing, test-driven development, behavior-driven development, continuous integration, continuous delivery, continuous deployment, DevOps, etc.
As a result, teams are pretty good at delivering software now. The Agile Manifesto set out with a goal of “uncovering better ways to build software”. In that it has succeeded; software delivery today is greatly improved over the state of the art when the Agile Manifesto was created. In most cases building and shipping software is no longer the bottleneck to delivering business value. The bottleneck has moved, and as a result the problem that Agile addresses is no longer the fundamental system constraint.
The fundamental constraint to delivering value has shifted to building the right thing. A product is more likely to fail today because no one wants it rather than because of a failure to deliver it or because of quality problems. In other words, most failures today are a failure to reach product/market fit. That is, they are a failure to build the right thing.
Agile doesn’t help to address this new constraint. As Gojko Adzic, Dan North and others have pointed out, Agile isn’t concerned with what to build; it only addresses how to build things. In other words, Agile is a local optimization. It only addresses a subset of the value creation process: software delivery. Agile principles and practices won’t get you to product/market fit.
Successful organizations in the 21st century will recognize this fundamental shift in the constraint to value delivery and will adapt. The shift is underrecognized. Many organizations are still working on ‘Agile transformations’ without realizing that they are targeting last decade’s problems. What’s required now is a transformation of the product development process into an end-to-end framework enabling building the right thing through a process of product discovery rather than product prescription.
The traditional product development approach is for an organization to target a market, define a product upfront, build it, release it and then see whether it is a success. It’s very difficult to reach a product/market fit with this prescriptive approach. Perhaps it may have worked just enough for organizations with a lot of capital in less competitive environments when business cycles were slower, but today it’s a source of enormous waste and opportunity cost.
How then can we address this new constraint and try to build the right thing? The first step in building the right thing is to avoid building wrong things along the way. This of course requires the ability to recognize something as being the wrong thing. In other words, it requires feedback loops, learning, and adaptation.
How can we recognize that something is a wrong thing? A wrong thing doesn’t conform with the assumptions underlying our idea. Therefore we derisk our idea by explicitly enumerating the assumptions that we’re making and then - in order of highest risk - devising inexpensive experiments to try to invalidate each assumption. A failure to invalidate the hypothesis means that further investment in the idea may be warranted.
This transition requires adopting an experimental mindset. Rather than thinking in terms of requirements, we think in terms of hypotheses and in terms of investments in experiments. We iteratively invest a little to validate ideas before making larger investments in ideas. We orient around goals and outcomes rather than around the delivery of features. We use adaptive planning rather than predictive planning.
In essence what is needed is the adoption of a Lean Startup mindset and the creation of a toolbox of techniques for (in)validation. The toolbox includes items such as prototyping, customer & user interviews, market research surveys, usability testing, etc. It’s a process of product discovery achieved by iterating and adapting toward a product/market fit.
Transitioning away from product prescription to product discovery is challenging. Product development transformation is more far-reaching than applying Agile practices and thus is harder to achieve. It requires widespread cultural change, and cultural change is hard. It requires change at levels and from roles in the organization that have sometimes considered change to be something that applies to others in the organization. For example, changes in governance and in budgeting are required, because an adaptive approach has zero chance of survival in a traditional prepare-a-business-case, pitch it, get approval and budget, measure-deviation-from-plan governance environment.
It’s challenging for an organization and the individuals in it to recognize that this shift in the flow of value has occurred (particularly when preserving the status quo depends on not recognizing the shift). When recognition does occur, it is then challenging to coach and educate the organization in a completely different product development mentality involving significant learning curves for new techniques.
However, therein lies both the challenge and the opportunity. These changes are not simple, but they are also not optional for organizations that want to stay relevant in the 21st century.
So where does this leave Agile? It’s not dead. Agile still has its place, and its practices and principles are still as useful as ever. Agile has done its job and continues to do its job and to evolve. But Agile isn't about building the right thing - it's about building the thing right. Building the right thing is the job of a product development process focused on discovering a product/market fit in a good market.
Update: There's been some good discussion on Reddit
