Agile: On February 11-13, 2001, seventeen software developers met at The Lodge at Snowbird ski resort in Utah. Among them were Kent Beck, Martin Fowler, Robert C. Martin, and Alistair Cockburn. They were frustrated.
Software projects were failing at alarming rates - the Standish Group's CHAOS Report had been documenting failure rates above 70% for years.
The dominant approach, sequential "waterfall" development, seemed fundamentally mismatched with the reality of building software.
Over those three days, the group produced the Agile Manifesto: twelve principles and four values that would reshape how software was built.[1]
The manifesto prioritized individuals and interactions over processes and tools, working software over comprehensive documentation, customer collaboration over contract negotiation, and responding to change over following a plan.
Within a decade, agile methodology had become the default approach in software development and was spreading to marketing, product, finance, and operations functions that had never previously used project management methodologies.
The adoption was not always thoughtful. Many organizations adopted "agile" as a label without understanding the underlying principles. Others adopted specific agile practices - daily standups, sprint planning, retrospectives - without the cultural conditions that make those practices effective.
The result was a landscape where "agile" described everything from highly disciplined iterative development to the absence of any planning whatsoever.
The relevant question is not "should we be agile or waterfall?" It is: which project characteristics call for which approach, and what hybrid combinations make sense for specific organizational contexts?
"Agile is not about doing less planning. It is about doing planning at the point in the project when you actually have information - rather than front-loading detailed plans that will be wrong before development begins." - Martin Fowler
| Dimension | Waterfall | Agile | When Waterfall Wins | When Agile Wins |
|---|---|---|---|---|
| Requirements | Fixed upfront | Evolve through iteration | Contractually stable, regulated | Poorly understood; expected to change |
| Planning | Comprehensive before work starts | Continuous throughout | Physical construction, compliance-heavy | Software, digital products, market uncertainty |
| Change tolerance | High cost to change late | Changes accommodated in each sprint | Aerospace, medical devices | Consumer software, startup products |
| Feedback loop | End of project | End of each sprint (1-4 weeks) | When output must be correct before delivery | When user feedback shapes the right output |
| Documentation | Comprehensive at each phase gate | Minimal; working output is the measure | Regulated industries requiring audit trails | Teams where code is the authoritative record |
Waterfall: The Foundation
Waterfall methodology structures projects as a linear sequence of phases: requirements, design, implementation, testing, and deployment. Each phase is completed before the next begins. Requirements are finalized before design starts; design is completed before implementation begins; implementation is done before testing starts.
The name comes from the visual representation of this sequence - phases cascade downward like a waterfall, with each phase flowing into the next.
Where Waterfall Came From
The waterfall approach was not originally designed for software. It was adapted from manufacturing and construction project management, where the sequential logic is genuinely appropriate. You cannot pour the foundation before the architectural drawings are finalized.
You cannot install plumbing before the walls are framed. Physical construction requires sequential planning because reversing decisions is enormously expensive.
Winston Royce's 1970 paper "Managing the Development of Large Software Systems" is often cited as the origin of waterfall methodology, but this is partly a misreading.[3]
Royce actually described the sequential model and then argued that it was "risky and invited failure" because it required successful completion of each phase before discovering that requirements or design decisions needed revision.
He was describing a failure mode, not recommending a method.
When Waterfall Works
Waterfall is appropriate when:
Requirements are stable and well-understood. If you know precisely what you are building before you start building it, front-loading the requirements phase produces a stable foundation for everything that follows.
Defense contracting, regulated medical device software, and some government IT systems have requirements that are contractually fixed before development begins. Waterfall is appropriate for these contexts.
Costs of change are high. Physical construction, semiconductor chip fabrication, aerospace hardware - these domains have extremely high costs for design changes late in the process. Sequential, phase-gated development protects against the expensive changes that result from discovering requirements errors late.
Compliance documentation is required. Regulated industries (pharmaceutical, aerospace, financial services) often require documented proof that each development phase was reviewed and approved before the next began. Waterfall's phase-gate structure produces this documentation naturally.
Team expertise in execution is high and requirements expertise is available. When the what is known and the how is known, sequential execution is efficient. Agile's iteration overhead is unnecessary when the path is clear.
Example: Airbus's commercial aircraft development follows a heavily phase-gated sequential process. Requirements for a new aircraft are defined years before the first physical component is produced. Design review gates require sign-off from safety, engineering, manufacturing, and regulatory functions before the next phase begins.
The aircraft's success depends on the accuracy of the upfront design - a mid-development pivot would be prohibitively expensive. Waterfall is the appropriate methodology for contexts with these characteristics.
Agile: The Iterative Alternative
Agile methodology structures work in short iterations - typically one to four weeks - called sprints or cycles. Each iteration produces working output that can be evaluated. Requirements are not fixed upfront; they evolve based on feedback from each iteration.
Planning happens continuously throughout the project rather than entirely at the beginning.[9]
The core insight underlying agile is that uncertainty compounds over time. The further into the future you plan, the less accurate your plans will be.
In domains where requirements are poorly understood at the start or change frequently during development, detailed upfront planning produces plans that are extensively detailed and substantially wrong.
The Agile Manifesto Principles in Practice
The manifesto's four values translate into specific practices:
Individuals and interactions over processes and tools: Small, cross-functional teams with high communication density outperform large teams coordinating through formal processes. This is why agile teams are typically co-located (or tightly connected) and small - typically five to ten people.
Working software over comprehensive documentation: The metric of progress is working product, not plans completed or documents produced. A team that has built and shipped a working feature is further along than a team that has completed the requirements document for the same feature.
Customer collaboration over contract negotiation: Agile requires ongoing involvement from the person who represents customer needs - typically a product owner or customer proxy. Requirements are not handed off at the start; they are developed continuously through conversation between the development team and the product owner.
Responding to change over following a plan: Plans are starting points, not commitments. When learning during development reveals that the plan was wrong - which it almost always does - the plan changes. This is not failure; it is the system working as designed.
Scrum: The Dominant Agile Framework
Scrum, developed by Jeff Sutherland and Ken Schwaber in the 1990s, is the most widely adopted agile framework.[6] Its structure:
Roles:
- Product Owner: Owns the backlog, prioritizes features, represents stakeholder needs
- Scrum Master: Facilitates the process, removes impediments, coaches the team
- Development Team: Cross-functional group that does the actual work
Ceremonies:
- Sprint Planning: The team selects items from the backlog for the upcoming sprint
- Daily Standup: 15-minute synchronization - what did I do yesterday, what will I do today, what is blocking me?
- Sprint Review: Demonstrate completed work to stakeholders and gather feedback
- Retrospective: Reflect on the process and identify improvements for the next sprint
Artifacts:
- Product Backlog: The prioritized list of all desired features
- Sprint Backlog: The subset selected for the current sprint
- Increment: The sum of completed backlog items, meeting the definition of done
Example: Spotify's engineering culture, documented by Henrik Kniberg and Anders Ivarsson in 2012, adapted scrum for a large-scale organization. Spotify's model (Squads, Tribes, Chapters, and Guilds) maintained small-team autonomy while creating alignment mechanisms across hundreds of engineers.
The "Spotify Model" became widely emulated - sometimes appropriately, often without the cultural conditions that made it work at Spotify.
Kanban: The Flow-Based Alternative
Kanban, derived from Toyota's production system, offers an alternative to scrum's sprint-based structure.[7] Instead of time-boxed iterations, kanban manages work as a continuous flow:
- Work items are visualized on a board with columns representing stages
- Each stage has a work-in-progress limit (WIP limit) that prevents overloading
- Work flows through stages from left to right; bottlenecks are visible when items pile up before a column at capacity
Kanban is better suited than scrum for:
- Operations and maintenance work (which is interrupt-driven and not easily sprint-planned)
- Support and service functions (where work arrives unpredictably)
- Teams that are already high-performing and need continuous improvement rather than a process reset
The Critical Differences in Practice
How Requirements Work
Waterfall: Requirements are fully specified before development begins. The requirements document is a contract between what will be built and who will build it. Changes to requirements mid-development are change requests - formal processes with cost and schedule impact.
Agile: Requirements are a starting point that evolves through development. The product backlog is continuously refined as understanding improves. What gets built in iteration 8 is shaped by what was learned in iterations 1 through 7.
The difference is not that agile has fewer requirements - it is that requirements are treated as hypotheses to be validated rather than specifications to be implemented. This is appropriate when the right solution is not known upfront; it is inefficient when the right solution is well-specified.
How Risk Is Managed
Waterfall: Risk is managed through upfront analysis - identifying risks before work begins and building mitigation plans. Front-loaded risk management is appropriate when the risks are known and the cost of discovering them later is high.
Agile: Risk is managed through early learning - building the riskiest elements first to discover whether they work before investing more. The first sprint typically addresses the highest-risk unknowns. Failure in sprint one is significantly less costly than failure at launch.
How Progress Is Measured
Waterfall: Progress is measured against the plan - percentage of planned scope completed, milestones achieved. This measurement is clear when it matches reality but misleading when the plan was wrong: a project can be "80% complete" while being months from producing anything useful.
Agile: Progress is measured in working software delivered - velocity (story points completed per sprint), burndown (remaining work in the backlog), and the increment demonstrated at each sprint review. This measurement is more directly tied to delivered value.
Hybrid Approaches
Most sophisticated organizations do not use pure waterfall or pure agile. They use hybrid approaches that apply different methodologies to different phases or aspects of projects.
Waterfall for architecture, agile for implementation: Large systems often benefit from significant upfront architectural design (which justifies waterfall's front-loading) followed by agile implementation of components within the established architecture.
Agile for product development, waterfall for compliance: Organizations in regulated industries may use agile development practices while maintaining waterfall documentation and review gates required by regulators. The development culture is agile; the documentation trails are waterfall.
SAFe (Scaled Agile Framework): Large organizations with hundreds of developers have adopted frameworks that coordinate multiple agile teams.[8] SAFe introduces quarterly Program Increments (PIs) that provide a waterfall-like planning horizon while preserving sprint-level agility within each PI.
Critics argue that SAFe reintroduces the rigidity that agile was designed to escape; proponents argue that coordination requirements at scale justify the overhead.
The Methodology Is Not the Point
The most important insight about the agile-vs-waterfall debate is that the methodology is a tool, not a destination. Organizations that treat agile adoption as a transformation goal - rather than as a means to deliver better products faster - tend to produce ceremonial agile: all the rituals, none of the improvement.
The real questions are:
- How quickly can the team deliver working product and get feedback?
- How effectively does the team respond when feedback indicates the plan was wrong?
- How much waste is produced by work that is completed but not valuable?[10]
Methodology is a framework for improving these outcomes. The framework that improves them most in a specific organizational context - whether pure agile, pure waterfall, or some hybrid - is the right one. The framework that is most fashionable is not.
For related frameworks on why projects fail regardless of methodology, see project failure reasons. For how to manage the tradeoffs between delivery speed and quality, see delivery vs quality tradeoffs.
What Research Shows About Agile and Waterfall Methodologies
The Standish Group CHAOS Report, published annually since 1994, provides the most longitudinal data on software project outcomes by methodology.[4] The 2020 report, covering more than 50,000 projects, found that agile projects succeeded (delivered on time, on budget, with satisfactory results) at 42% versus 19% for waterfall projects.
The gap has widened over successive reports. The 2015 report found that large projects - those with budgets exceeding $10 million - failed at rates approaching 50% under waterfall approaches, while equivalent agile projects failed at approximately 8%.
Scale amplifies the advantages of iterative approaches because large projects accumulate requirements uncertainty, making early discovery of incorrect assumptions more valuable.
The DORA research program, documented in Accelerate by Dr. Nicole Forsgren, Jez Humble, and Gene Kim (IT Revolution Press, 2018), provides the most rigorous empirical data on software delivery practices.
Their four-year study of more than 23,000 practitioners found that technical practices associated with agile development - continuous integration, comprehensive automated testing, trunk-based development, and deployment pipelines - were among the strongest predictors of both software delivery performance and organizational outcomes including revenue growth, profitability, and market share.
The research provides causal evidence (not merely correlation) that agile-associated technical practices drive business results.
Jeff Sutherland, co-creator of Scrum with Ken Schwaber, published research in Scrum: The Art of Doing Twice the Work in Half the Time (2014) describing productivity gains from Scrum adoption measured across multiple organizations.
Sutherland cited a DARPA study of agile software development showing productivity increases of 200-400% compared to traditional approaches for equivalent functionality.[2]
While these numbers are contested - productivity comparisons across projects are methodologically difficult - the directional finding (agile faster, by a large margin) is consistent across multiple independent studies.
Academic research on Scrum specifically was published by Phillip A. Laplante and Colin J. Neill in "Antipatterns and Patterns in Software Configuration Management" and subsequent work.
Their analysis of Scrum adoption outcomes across 100 organizations found that teams with experienced Scrum Masters - defined as practitioners with more than two years of facilitation experience - had significantly better outcomes than those with novice Scrum Masters.
The finding suggests that Scrum's value is not simply in following its ceremonies but in the facilitation skill required to make those ceremonies productive.
Real-World Case Studies in Agile and Waterfall Adoption
Spotify's Squad model, introduced in 2012 and documented by Henrik Kniberg and Anders Ivarsson in their widely circulated whitepaper, adapted agile principles for a large-scale engineering organization.[5]
Spotify organized engineers into autonomous Squads (small cross-functional teams), Tribes (collections of Squads working in related areas), Chapters (communities of practice across Squads), and Guilds (informal interest communities).
The model explicitly preserved Scrum's emphasis on team autonomy and cross-functionality while solving coordination problems that arise when hundreds of engineers work on interdependent systems.
The "Spotify Model" became perhaps the most emulated engineering organization structure of the 2010s, though Spotify itself subsequently acknowledged that the model evolved continuously and that the 2012 whitepaper captured a moment, not a final state.
The FBI's Sentinel project is one of the most instructive case studies in the costs of waterfall methodology applied to the wrong context. The FBI's case management system replacement, begun in 2006, was initially managed using a traditional government IT waterfall approach.
After spending $405 million over five years and producing a system that did not work, the FBI brought in a new contractor and restarted the project using agile methods.
The agile project delivered a working system in twenty months for $30 million - less than one-tenth the previous expenditure. The case became a reference example in US government IT reform efforts and contributed to the US Digital Service's emphasis on agile practices in government technology projects.
ING Bank's agile transformation, conducted between 2015 and 2018 and documented in Harvard Business Review, reorganized the entire bank's technology and business operations around agile principles.
ING eliminated 13 layers of management, formed 350 autonomous "squads" of 9 engineers each, and measured outcomes against customer experience metrics rather than project milestone completion.
The transformation produced measurably faster feature delivery and higher employee engagement scores, but also required significant cultural change - approximately 40% of employees were reassigned to different roles, and the company explicitly accepted short-term disruption for long-term capability improvement.
Microsoft's transformation to DevOps and agile practices, described by Sam Guckenheimer and others on the Azure DevOps Blog, converted Visual Studio from annual release cycles to continuous delivery between 2010 and 2014.
The conversion required building comprehensive automated test coverage from near-zero (the existing product had approximately 35% code coverage), establishing CI/CD pipelines for a product with millions of lines of code, and changing the team culture from release-oriented to flow-oriented.
Microsoft's internal research, comparing the pre- and post-transformation periods, found that deployment frequency increased by more than 500% while incident rates decreased by more than 60%.
Key Metrics and Evidence in Methodology Selection
Research comparing project success rates across methodologies consistently shows agile outperforming waterfall for software projects, but the magnitude of advantage varies by project characteristics.
A 2018 meta-analysis by Diego Fontdevila and colleagues, published in the Journal of Systems and Software, reviewed 30 empirical studies comparing agile and waterfall outcomes.
The analysis found that agile produced better results for time-to-market, customer satisfaction, and defect rates, but the advantage was less pronounced for large projects with stable requirements.
The finding supports the principle that methodology selection should be driven by project characteristics rather than organizational fashion.
The CHAOS Report's breakdown of project outcomes by size reveals an important nuance: for small projects (under $1 million budget), waterfall and agile produce similar success rates, around 70-75%. The divergence grows with project size.
For projects exceeding $10 million, waterfall success rates fall below 30% while agile success rates remain above 50%. The implication is that small, well-defined projects may not benefit significantly from agile overhead, while large projects almost universally benefit from agile's risk management through early feedback.
Team size research by Robin Dunbar and its software development application by Fred Brooks converges on a consistent finding: communication complexity grows with team size, typically at O(n squared) where n is team members. A team of 5 has 10 communication pairs; a team of 50 has 1,225.
This mathematical reality is why Scrum recommends teams of 5-9 (usually called "7 plus or minus 2" by practitioners): it is the size range that balances sufficient skill diversity with manageable communication overhead.
Organizations that implement agile with teams outside this range consistently report coordination problems that the methodology was not designed to address.
The Project Management Institute's Pulse of the Profession report (2023) found that organizations with mature project management practices - including defined methodology selection criteria, not just uniform adoption of a single methodology - completed significantly more projects within scope, schedule, and budget than organizations applying one methodology universally.
The finding suggests that the most sophisticated practitioners do not debate agile versus waterfall as a binary choice, but develop organizational capability to select and adapt methodologies based on project context.
Sources & Further Reading
- Beck, K. et al. "Manifesto for Agile Software Development." Agilemanifesto.org, 2001. View source
- Sutherland, J. Scrum: The Art of Doing Twice the Work in Half the Time. Currency, 2014. View source
- Royce, W. "Managing the Development of Large Software Systems." Proceedings of IEEE WESCON, 1970.
- Standish Group. "CHAOS Report 2020." Standish Group, 2020. View source
- Kniberg, H. & Ivarsson, A. "Scaling Agile at Spotify." Spotify Labs, 2012. View source
- Schwaber, K. & Sutherland, J. "The Scrum Guide." Scrumguides.org, 2020. View source
- Anderson, D. J. Kanban: Successful Evolutionary Change for Your Technology Business. Blue Hole Press, 2010. View source
- Larman, C. & Vodde, B. Large-Scale Scrum. Addison-Wesley, 2016. View source
- Highsmith, J. Agile Project Management: Creating Innovative Products. Addison-Wesley, 2009. View source
- Poppendieck, M. & Poppendieck, T. Lean Software Development. Addison-Wesley, 2003. View source