Why Agile is Here to Stay
Why Agile? Throw the question out to programmers, consultants, or even Google, and you’ll get tributes or tirades, with little in between. It’s not just software developers that use “Agile” ways of working. Now there are even agile marketing and agile HR teams. In fact, 90% of organizations surveyed for the State of Agile Report say they practice agile with most of their teams.
So why is Agile so popular among teams, companies, and stakeholders?
This resource seeks to explain why Agile – an adaptive approach for project management – is loved by some, loathed by others, but destined to stick around.
What is Agile? 🤷🏽♀️
Agile is a project management approach that prioritises short work iterations over long-term project deadlines. Agile is built around a few core principles, such as working iteratively, putting the customer first, only committing to building small increments at a time, continuous improvement, and working in a lean way.
Over the two decades since Agile emerged, it has exploded in popularity, with many companies attempting to “go Agile” or launching “Agile transformations” to help them build better products faster.
The Agile Manifesto and its principles
The story of Agile begins in 2001, when 17 developers met in Utah and crafted The Agile Manifesto.
Its principles inspired people everywhere to create new practices for their day-to-day work.
The Agile Manifesto is only a few lines long but has had a huge impact on the world:
We are uncovering better ways of developing software by doing it and helping others do it.
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a planThe Agile Manifesto
That is, while there is value in the items on the right, we value the items on the left more.
Alongside the Agile Manifesto are 12 Agile principles to guide would-be Agilists in their implementations.
When you read the manifesto and its principles, two things stand out:
- They are very short
- They are open to interpretation
The Agile manifesto doesn’t give any practical guidance or agile process for implementing its ideas. That’s why many agile frameworks have emerged, like Scrum, Extreme Programming (XP), and SAFe that offer options for putting Agile principles into practice.
Whereas the Agile Manifesto only provides the why and what, these methods provide the how of agile working. They describe things like which roles to create on your teams, the type of meetings to plan, and how to run them.
💡 Fun fact: Did you know that the Scrum framework pre-dates the Agile Manifesto, even though it is now the most popular “agile framework”. 81% of agile teams using some version of Scrum, including Scrumban or a hybrid Scrum model.
Why Agile is so popular ❤️
Agile offers companies a way to deal with the fast-changing world around us – including shifting market conditions, customer preferences, and technology. Today’s teams also expect more autonomy in their work. Agile fits these requirements and improves customer satisfaction, too.
Project managers, teams, and organizations often want to “go Agile” because it:
- Helps companies to go to market more quickly
- Ensures the company is always building something the customer wants
- Prevents surprises on “launch day” through continuous releases and continuous testing
- Gives people autonomy in their work and creates time for deep work
- Focuses on the customer’s needs first
- Empowers teams to inspect, adapt, and improve their working process regularly
- Gives companies a competitive edge
💡 Fun fact: 93% of companies in one study agreed their agile business units outperformed others on customer satisfaction during the COVID-19 pandemic.
Agile frees teams from bureaucracy and micro-management
Before agile software development, few people were happy with the way software projects were executed.
Programmers spent much of their days writing documentation instead of code or were stuck in meetings. While product owners hyperventilated in a corner somewhere as launch deadlines arrived without much evidence that the product would actually work.
Agile changed all that by inspiring new, iterative ways of working that lead to high-quality outcomes.
Agile advocates often discuss it as a mindset and a way of living – one that cuts through the bureaucracy of traditional teaming and makes work more meaningful, efficient, and empowering for everyone involved.
Data on Agile popularity
- HR leaders rate “role clarity” and “more focus” as the top benefits of using agile in their function. (Source: State of Agile HR)
- Teams that adopt Scrum well can improve their productivity by 300% to 400%. The best teams can achieve productivity increases of up to 800% and replicate that success over and over again. They also end up more than doubling the quality of their work. (Source: Jeff Sutherland)
- 85% of Scrum practitioners state that it improved the quality of their work life. (Source: Scrum Alliance)
Agile project management is often better than waterfall
Before Agile, teams would manage the software development process based on a waterfall model – the knowledge-work equivalent of a conveyor belt. In such a traditional project model, every phase and task moves sequentially. Nothing can happen in parallel. Think: Gantt charts.
As with the factory conveyor belt, one problem can hold up the entire project, and the customer only gets to see the finished product at the end. By that time, they might not even want that product anymore.
The waterfall model doesn’t leave any room for changes or additions once development is underway. Sales reps and customers would bicker over the scope of the contract, agreeing on what would and would not be built. The result: bloated project budgets and over-runs.
As Lean manufacturing transformed factories, Agile transformed software projects and other knowledge work. One of the best real-world examples of Agile’s effectiveness compared to waterfall is the turnaround of the FBI’s Sentinel project.
Sentinel started in 2005 to merge the FBI’s internal data and make it shareable across the organization. By 2010, using traditional software development methodologies, the project was a year behind schedule, only halfway finished, and close to blowing past its $451 million budget.
Then Chad Fulgham, the FBI’s chief information officer (CIO), brought in Jeff Johnson, a Scrum expert. He reduced the project’s team size by 80%, spent just $20 million, and finished the project within 18 months.
Agile vs Waterfall data
- 78% of Scrum practitioners would recommend the framework to colleagues, friends, or other professionals. (Source: Scrum Alliance)
- Agile adoption in software teams increased from 37% in 2020 to 86% in 2021. (Source: Digital.ai)
- 91% of organizations state that it is a strategic priority to adopt Agile. (Source: KPMG)
- By 2017, 80% of U.S. federal IT projects were characterized as agile or iterative, as compared with 10% in 2011. (Source: Deloitte)
- 70% of agile organizations say that agile helps them achieve a faster time to market. (Source: JCURV)
- Business units that fully adopted agile before the Covid-19 pandemic outperformed units that hadn’t on customer satisfaction (93% agreed), employee engagement (76% agreed), and operational performance (93% agreed). (Source: McKinsey)
Getting started with Agile methods is easy️ and delivers immediate benefits
Two of the most popular Agile frameworks are Scrum and Kanban. While mastery requires longer – and help from an Agile Coach is recommended – teams can be “doing Agile” within hours with these approaches, which is why they’ve become so widespread. The easiest starting point is simply running regular retrospective meetings to reflect and improve on work.
Woody Zuill, Agile Coach
“If you adopt only one agile practice, let it be retrospectives. Everything else will follow.”
Getting started with Scrum
Scrum, for example, originates from the official Scrum Guide, a short PDF you can read in less than an hour. It describes a few roles and meetings your team can adopt immediately, without further cost or training. Understanding and setting up a Scrum board or starting retrospectives takes less than an hour. You can even do it without disrupting your team’s existing methods and practices.
Getting started with Kanban
Kanban’s central element is the Kanban Board, a board in which columns reflect the different phases of a team’s working process. Cards representing work items move through these columns, from idea to completion. This visualization of work helps reduce bottlenecks and gives a “status update” to anyone who looks at the board.
Agile effectiveness data
- Teams that have regular sprint retrospectives have 24% more responsiveness and 42% higher quality with less variability than teams with infrequent or no retrospectives. (Source: CA Technologies)
- 76% of respondents reported that Kanban was “effective” or “much more effective” than other methods/frameworks that they have used. (Source: Kanban University)
- Kanban boards for workflow management are now used by 61% of respondents, steadily increasing from 6% in the first survey. (Source: Digital.ai)
Why Agile fails 😬
Despite its soaring popularity, many managers dislike Agile’s mantra of decentralization and self-management.
Others fear it will cause organizational anarchy or will make their roles redundant.
When Agile is practiced incorrectly it can also become miserable.
Daily Scrum meetings can turn into endless ordeals every day of the week if they’re not properly time-boxed. It’s also all too common for teams to find they’re still practicing a half Agile half waterfall medley that causes a lot of stress and insecurity.
Many companies also use Agile as an excuse to introduce more metrics, processes, and control from the top down rather than empowering teams to arrange these for themselves.
Rigid rules for how to do Agile
Some Agile Coaches and consultants from the so-called “Agile-Industrial Complex” alienate teams with rigid rules presented as Agile laws.
They proclaim right and wrong ways of doing Agile instead of teaching its fundamentals, so teams can adapt it to their specific needs.
In its most innocent form, a well-meaning but inexperienced Scrum Master doesn’t allow any deviation from the Scrum Guide because it’s meant to be “immutable” and they’re insecure about what happens when they stray from the official path.
In darker scenarios, some companies purposely dictate a complex, rigid framework labeled as Agile to demand a hefty licensing bill and hourly fees to “make it work.”
The words agility and rigidity are essentially opposites. There’s nothing wrong with using a particular method, but it should be a starting point and guide, not dogma.
Agile’s aim is continuous improvement — to constantly experiment and adapt in service of the customer, team, and organization. Not to blindly follow one framework to the letter because “the guide said so”.
Agile can create confusion and frustration
When organizations misunderstand Agile, teams do daily standup meetings, write user stories, and call projects sprints without knowing why. Worse, managers let them go through these motions on top of their existing practices, causing burnout.
The result? “Conformance Scrum” (or Zombie Scrum for horror movie fans) and “waterfall development with additional overheads like sprints and iterations.”
Working in an Agile way is meant to reduce bureaucracy by making space for deep work. When misapplied, it creates more bureaucracy that reduces productivity and increases frustration – the opposite of what Agile should achieve.
Max Andaker, co-founder and chief product officer (CPO) at CoScreen (now Datadog), told us:
“Many organizations have gotten so lost in their pursuit of the ideal agile workflow and processes that they are missing the point of what the original manifesto and principles were trying to convey.”
Confusion around Agile terminology also doesn’t help Agile’s case: Agile, agile, Scrum, daily scrums, and daily standups, to name a few confusing agile terms. When even experienced practitioners use different words for the same thing or the same word for different things, you can be sure everyone else gets confused, too.
A simple internet search doesn’t always fix things either. As Monica Georgieff, Agile coach and trainer at Agile Sherpas, told us:
“There’s so much information on the internet about Agile right now that it’s a bit chaotic to figure out what it is and what it isn’t.”
This confusion creates divisions among developers and between development teams and business folks. They no longer speak the same language and misunderstand each other.
Teams apply Agile when they shouldn’t
Agile is not a magic solution that solves every organization’s problems. Yet, ignorance of Agile’s strengths and weaknesses, its soaring popularity, and commercial interests from consultants and coaches, means Agile often gets applied in cases where it shouldn’t.
Here’s why Agile doesn’t work for some teams:
- Agile is deadline-averse. When there’s external pressure to complete a specific thing at a particular time, Agile might not be the right choice. Planning an event is one such case or when filing legal or other compliance documents.
- ️ Agile shuns predictability and rigidity. Agile isn’t a great fit when projects recur and follow the same structure – think accounting, maintenance, and legal work.
- Agile needs a welcoming host. Change-averse organizations shouldn’t adopt Agile. When leaders refuse to support teams to become self-managed or ditch other meetings in favor of Agile ceremonies, an agile approach will just frustrate team members and fail.
Maarten Dalmijn, Serious Scrum ambassador, told us he sometimes doesn’t pick an agile approach for projects he’s involved in:
“There have been cases where I specifically decided to not work in an Agile way to achieve the best results. There isn’t one size fits all, you need to pick the best approach given the context you’re operating in. Agile is hot and popular, but that doesn’t mean it is always the best solution.”
Why Agile is here to stay 🏰
Agile teams thrive when they understand its principles and apply them in the proper context. It works particularly well amid rapidly changing circumstances and unpredictability – precisely what the future holds for most companies doing complex knowledge work.
Agile creates sustainable team productivity
Agile frameworks like Scrum create a sustainable work rhythm. Instead of thinking, “I can sit back because the deadline is in six months,” or being overwhelmed by the enormous task ahead, you work toward smaller, shorter-term goals that prevent project stagnation and all-nighters at the end.
You also have more time for deep work because meetings are fixed at scheduled times and places, reducing ad hoc distractions.
These shorter, focused cycles also allow agile teams to iterate faster. They constantly ship working features, which keeps them close to the customer.
As Jeff Sutherland, co-creator of Scrum and a signatory of the Agile Manifesto, says, Scrum helps teams “do twice the work in half the time.” He regularly sees Scrum teams improve their productivity by 300%, sometimes even 800%.
Now, Scrum’s creator might be biased about his own method. But research by management consultancy JCURV found that 70% of agile organizations think that agile helps them achieve a faster time to market. Such performance improvements mean more companies will adopt agile or keep it around for the long term, regardless of who loves or loathes it.
Agile is meant to change and adapt
Agile is… agile. Its principles and frameworks are flexible rather than rigid, resilient instead of fragile. These qualities give organizations and teams the ability to adapt and evolve their processes along with projects, customers, and the larger world.
For example, during Sprint Retrospectives – a meeting type in the Scrum framework – teams evolve by scrutinizing their own performance. Jordan Husney, CEO of Agile meeting platform Parabol, says:
“Adopting agile can feel worse before it feels better. Problems and tensions often become more visible, which can be uncomfortable for some teams. But when tensions are more visible, you have a better opportunity to solve them.”
Agile’s evolution is a testimony to this ability to change. There are now dozens of frameworks based on agile principles, and new ones appear every year. Some, like Scrum or Kanban, are widely adopted. Others, like the so-called “Spotify Model” for doing Agile, are tailored to specific organizations. Fredrik Carleson, an IT professional, says:
“The Spotify model was customized for Spotify. Organizations will have to find framework experts who can help create their own ‘pimped’ agile framework. There will be a need for many new frameworks.”
Agile has spread far beyond its software origins. National Public Radio uses it to create new programs, John Deere to develop machines, and Saab to make fighter jets. It’s also used across disciplines like marketing, HR, and executive management. For example, in recent research by Agile Sherpas, 51% of marketing teams considered themselves agile.
Is Agile right for you? 🚦
Agile isn’t a silver bullet that solves every challenge under any circumstance. But it might be a good fit if you:
- Deliver complex and unpredictable work because requirements for your teams change often, at short notice, or both.
- Want to shorten feedback loops between your teams and customers to reduce mismatches between what you create and what your customer needs.
- Believe good ideas come from people close to the customer rather than from distant decision-making by senior managers and executives.
- ️See a lot of waste in your company in the form of bureaucracy and overhead, and you’re looking for ways to streamline work.
- Don’t get results from your approach to planning and execution, often signaled by projects or products that take forever to complete.
- Have an interest in going agile! Perhaps you’ve seen one or more other teams with Agile practices, and their results have inspired you and others in the organization.
Do one or more of these points apply to your situation? Consider diving deeper into agile with the guides and resources listed below.
So you want to “go Agile?” But which framework should you choose? Here’s an overview of all essential agile frameworks. We spoke with several Agile coaches to help you decide which framework is right for your team or organization. And we organized the most relevant frameworks according to team size and organization type.
An agile meeting is typically considered any event that’s part of an Agile framework, method, or theory. This resource catalogs them all.
A step-by-step overview of what to consider before embarking on an agile journey and how to implement one of its frameworks when you do.
Scrum is the most popular agile framework. It’s easy to get started with and the most recommended take-off point for teams new to agile. This guide covers everything you need to know.
An in-depth overview of Scrum’s five events: Sprint Planning, Daily Scrum, Sprint Review, Sprint Retrospective, and Backlog Refinement. The article includes guidance for remote and cross-functional teams.
A conversation about Scrum and Agile can quickly get confusing, even for the initiated. This resource covers and clarifies the nine most confusing terms.
Want to get up to speed with all the Agile lingo? This comprehensive glossary is worth the whole team bookmarking so you can stay on the same page.
Perhaps you’re trying to figure out whether Agile is right for you, or justify it to your team. This resource is the most comprehensive list of Agile and Scrum statistics on the internet. You’re sure to find something useful.