Agile Software Development Ecosystems [Paperback]

From February 11 to 13, 2001, at the Lodge at Snowbird ski resort in the Wasatch Mountains of Utah, 17 people met to talk, ski, relax, and try to find common ground. What emerged was the Agile Software Development movement. Representatives from Extreme Programming (XP), Scrum, the Dynamic Systems Development Method (DSDM), Adaptive Software Development (ASD), Crystal Methods, Feature-Driven Development (FDD), Pragmatic Programming, and others sympathetic to the need for an alternative to document-driven, rigorous software development processes convened. What this meeting produced--The Manifesto for Agile Software Development, signed by all 17 of the participants--was symbolic. It declares that:

We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:

• Individuals and interactions over processes and tools
• Working software over comprehensive documentation
• Customer collaboration over contract negotiation
• Responding to change over following a plan.

That is, while there is value in the items on the right, we value the items on the left more.

This value statement has a number of fascinating aspects, not the least of which was getting 17 people to agree to it. Although this was a group of experienced and recognized software development "gurus," the word uncovering was selected to indicate that the authors don't have all the answers and don't subscribe to the silver-bullet theory.

The phrase "by doing it" indicates that the authors actually practice these methods in their own work. Ken Schwaber told the story of his days of selling tools to automate comprehensive, "heavy" methodologies. Impressed by the responsiveness of Ken's company, Jeff Sutherland asked him which of these rigorous methodologies he used for internal development. "I still remember the look on Jeff's face," Ken remarked, "when I told him, ‘None. If we used any of them, we'd be out of business.'" The authors want to help others with Agile practices and to further our own knowledge by learning from those we try to help.

The value statements have a form. In each statement, the first segment indicates a preference, while the latter segment describes an item that, while important, is of lesser priority. This distinction lies at the heart of agility, but simply asking people to list what's valuable doesn't flesh out essential differences. Roy Singham, CEO of ThoughtWorks, put it well when he said that it's the edge cases, the hard choices, that interest him. "Yes, we value planning, comprehensive documentation, processes, and tools. That's easy to say. The hard thing is to ask, ‘What do you value more?'" When push comes to shove--and it usually does--something must give, and we need to be clear about what stays and what gives.

The first value statement recognizes the importance of processes and tools but stresses that the interaction of talented individuals is of far more importance. Rigorous methodologies and their business process reengineering brethren place more emphasis on process than people. Agile development reverses this trend. If individuals are unique, and we believe they are, then processes should be melded to individuals and teams, not the other way around.

Similarly, while comprehensive documentation is not necessarily harmful, the primary focus must remain on the final product--working software. This means that every project team needs to determine, for itself, what documentation is absolutely essential to delivering working software. Working software tells the developers and sponsors what they really have in front of them--as opposed to promises of what they might have in front of them. Working software can be shipped, modified, or scrapped, but it is always real.

Contract negotiation, whether through an internal project charter or an external legal contract, isn't a bad practice, just a seriously insufficient one. Customers want a product that conforms to their needs--at the time of delivery. "Contract negotiation encourages the addition of buffers contingency," says colleague Ron Holliday. "This makes projects take even longer, drives up costs, and reduces responsiveness to change. A collaborative arrangement is a team effort between customer and supplier to deliver the best possible solution." Contracts and project charters provide boundary conditions within which the parties can work, but only through ongoing collaboration can a development team hope to understand and deliver what the client wants.

No one can argue that following a plan is a good idea--right? Well, yes and no. In a world of business and technology turbulence, scrupulously following a plan can have dire consequences, even if it's executed faithfully. However carefully a plan is crafted, it becomes dangerous if it blinds you to change. As you will see in several of the case studies presented in this book, few, if any, of the projects delivered what was planned for in the beginning. And yet they were successful because the development team was Agile enough to respond again and again to external changes. In the Information Age, planning is important, but accepting that deviations from any plan are "normal" is even more important.

The meeting at Snowbird was incubated at a spring 2000 gathering of Extreme Programming leaders organized by Kent Beck. At that meeting in Oregon, a few "outsiders" but "sympathizers," such as myself, were invited, and attendees voiced support for a variety of "light" methodologies. During 2000, a number of articles referenced the category of "light" or "lightweight" practices. In conversation, the soon to be "Agile" authors didn't like the moniker "light," but it stuck at that time.

In September 2000, "Uncle Bob" Martin of Object Mentor in Chicago started what was to become the Agile ball rolling with an email: "I'd like to convene a short (two-day) conference in the January to February 2001 time frame here in Chicago. The purpose of this conference is to get all the lightweight method leaders in one room. All of you are invited, and I'd be interested to know who else I should approach." Bob set up an online group site, and the discussions raged. Early on, Alistair Cockburn weighed in with an epistle that identified the general disgruntlement with the word "light": "I don't mind the methodology being called light in weight, but I'm not sure I want to be referred to as a lightweight attending a lightweight methodologists meeting. It somehow sounds like a bunch of skinny, feeble-minded lightweight people trying to remember what day it is."

The fiercest debate was over location! There was serious concern about Chicago in wintertime--cold and nothing fun to do. Someone suggested Snowbird, Utah--cold, but fun things to do, at least for those who ski on their heads, as Martin Fowler tried on the first day. Someone else mentioned Anguilla in the Caribbean--warm and fun, but time consuming to get to. In the end, Snowbird and skiing won out.

The Agile Manifesto value statements represent a deeper theme that drives many, but not all, of the Manifesto's authors. At the close of the two-day meeting, Bob Martin joked that he was about to make a "mushy" statement. Although tinged with humor, no one disagreed with Bob's sentiments--that we all felt privileged to work with a group of people who held a set of compatible values, based on trust and respect for each other, promotion of organizational models based on people, and the desire to build collaborative communities in which to work. At the core, I believe Agile developers are really about mushy stuff--about delivering products to customers while operating in a vibrant, sustaining workplace. So in the final analysis, the meteoric rise of interest in--and criticism of--Agile Software Development is about the mushy stuff of values and culture.

For example, I think that, ultimately, XP has mushroomed in use and interest not because of pair programming or refactoring but because, taken as a whole, the practices define a developer community freed from the baggage of Dilbertesque corporations. Kent Beck tells the story of an early job in which he estimated a programming effort to be six weeks for two people. After his manager reassigned the other programmer at the beginning of the project (effectively halving the resources), Kent completed the project in twelve weeks--and felt terrible! The boss, of course, harangued Kent throughout the second six weeks. Somewhat despondent because he was such a "failure" as a programmer, Kent finally realized that his original estimate of six weeks was extremely accurate--for two people--and that his "failure" was really his manager's failure to take the responsibility for removing project resources. This type of situation goes on every day with individuals who don't want to make hard tradeoff decisions, so they impose irrational demands.

The Agile movement is not anti-methodology; in fact, we seek to restore credibility to the concept of methodology. We want to restore a balance. We accept modeling, but not in order to file some diagram in a dusty corporate repository. We accept documentation, but not hundreds of pages of never-maintained and rarely used tomes. We plan, but recognize the limits of planning in a turbulent environment. Those who would brand proponents of FDD or Scrum or any of the other Agile approaches as "hackers" are ignorant of both the approaches and the original definition of the term hacker--a programmer who enjoys solving complex programming problems, rather than one who practices either ad hoc development or "cracking." Agile Software Development incorporates proven software engineering techniques, but without the overhead of traditional methodologies.

The Agile Alliance was born in early 2001, but the history of the various approaches and the people who developed them goes back 10 to 15 years. This book describes these practices and the principles behind them, but more important, it delves into the people--the people who are developing the practices and the people who use them to deliver business value to their customers.

The left side of the Agile Manifesto value statements indicates what Agilists consider more important than the items on the right side. This should not be construed as indicating that tools, process, documentation, or contracts are unimportant. There is a tremendous difference between one thing being more or less important than another and being unimportant. Judicious use of tools is absolutely critical to speeding software development and reducing its costs. Contracts are one vital component to understanding developer-customer relationships. Documentation, in moderation, aids communication, enhances knowledge transfer, preserves historical information, and fulfills governmental and legal requirements.

But Agilists take a certain perspective on these topics. Try this exercise. For each of the Manifesto value statements, construct two questions along the lines of the following:

• Could we have a successful project by delivering documentation without working software?
• Could we have a successful project by delivering working software without any documentation?

By looking at these two end points, we can better grasp relative importance. Although there has to be a balance--documentation and working software, contracts and collaboration, responsiveness and planning, people and process--we have to delineate the extremes, the end points, so that organizations, teams, and individuals can find their own balance points. If we start out trying to find the middle ground, we never will.

One of the great contributions of XP's "extremoes"--Kent Beck, Ron Jeffries, and Ward Cunningham--is that they have staked out positions that have stirred debate in ways that taking moderate positions never would. When Ron says, "Great designs emerge from zero anticipatory design and continuous refactoring," he is challenging himself, and us, to rethink our assumptions about software development. We have to understand the limits before we can understand the balance points.

So although I realize the value of documentation, contracts, plans, and processes, there are numerous sources of material about these subjects. My intention is to identify and define Agile Software Development, to articulate its practices and principles, so you can make your own decision about where on the spectrum you, or your organization, need to be.

There are three fundamental questions that this book answers: (1) What kinds of problems does agility solve best? (2) What is agility? and (3) What are Agile Software Development Ecosystems (ASDEs)?

Problems characterized by change, speed, and turbulence are best solved by agility. Some people argue that good practices are good practices (pair programming, customer focus groups, or feature planning, for example) and therefore ASDEs should not be "limited" to a particular problem type. While true in part, the question asks what problems Agile practices best solve, not just solve. So, while XP, Crystal, or Scrum can surely be used for a wide range of projects, they are particularly relevant to extreme or complex projects--those that have an accelerated time schedule combined with significant risk and uncertainty that generate constant change during the project.

As the level of change caused by rapidly evolving technology, business models, and products increases and the need for delivery speed accelerates, ASDEs' effectiveness increases quickly over rigorous methodologies.

Agility, like any other complex concept, has a number of definitions, but for me, the most clearly focused definition is

Rather than shrink from change, Agile organizations harness or embrace change by being better than competitors at responding to changing conditions and by creating change that competitors can't respond to adequately. However, companies must determine what level of agility they require to remain competitive. Agility is only an advantage relative to competitors--a copper mining company doesn't need to be as agile as a biotechnology firm.

Other aspects of agility are also important: nimbleness or flexibility on the one hand, and balance on the other. Agile organizations are nimble (able to change directions quickly) and flexible (able to see how things that worked for them last week may not work as well next week). An Agile organization also knows how to balance structure and flexibility. If everything changes all the time, forward motion becomes problematic. Agile organizations understand that balancing on the edge between order and chaos determines success.

I began writing this book about Agile Software Development methodologies, but I kept worrying about the word "methodology" because it didn't fit with the focal points of Agile development--people, relationships, and uncertainty. Furthermore, by using the word "methodology," Agile practices are instantly compared to traditional software development methodologies--thereby using the wrong measuring stick for comparison. So I use the term "Agile Software Development Ecosystem" to describe a holistic environment that includes three interwoven components--a "chaordic" perspective, collaborative values and principles, and a barely sufficient methodology--and the term Agilists to identify those who are proponents of ASDEs.

Some people think that "Agile" means fewer processes, less ceremony, and briefer documents, but it has a much broader perspective, which is the primary reason for using the word ecosystem rather than methodology. Although fewer processes and less formality might lower development costs, they are not enough to produce agility. Focusing on people and their interactions and giving individuals the power to make quick decisions and to self-adapt their own processes are key to Agile ecosystems.

The American Heritage Dictionary defines ecosystem as "organisms and their environment: a localized group of interdependent organisms together with the environment that they inhabit and depend on." The Oxford English Dictionary extends this definition to include a constant interchange within the system, including both organic and inorganic elements. The word methodology conjures up a vision of things--activities, processes, tools. These are not inconsequential elements, but incomplete ones. The word ecosystem conjures up a vision of living things and their interactions with each other. Within an organizational context, an ecosystem can then be thought of as a dynamic, ever-changing environment in which people and organizations constantly initiate actions and respond to each other's actions. The word ecosystem focuses us on the dynamic interactions of individuals and teams rather than on the static lines on organization charts.

To fully understand ASDEs, we need to understand each of the three components and how they relate to each other. First, Agilists share a view that organizations are chaordic--that every organization exhibits properties of both chaos and order that defy management through the use of linear, predictive planning and execution practices. Viewing organizations as chaordic means understanding that the predictability upon which traditional project management and development life cycle practices are built is a root cause of dysfunctionality between customer, management, and development organizations. A chaordic perspective impacts both how we respond to change and how we manage project teams and organizations.

In day-to-day project work, a chaordic perspective creates two outcomes that are 180 degrees out of sync with rigorous methodologies.

• Product goals are achievable, but they are not predictable.
• Processes can aid consistency, but they are not repeatable.

Although ASDEs involve careful planning, the fundamental assumption remains that plans, in a turbulent environment, are not predictable, at least at the level of project scope, schedule, and cost. Plans are hypotheses to be tested rather than predictions to be realized. However, the product goals of the business are achievable, in large part because Agile people adapt. They can "adapt" to an articulated vision and a schedule, scope, or cost goal through tradeoffs in the other two dimensions. Second, while process can aid people in working together, in volatile environments the idea of driving out process variation through measurement and correction--statistical process control--becomes an unworkable hypothesis. Changes that are the result of knowledge gained during the project, knowledge not discernable early in the project, require processes that can respond to change, not ones that attempt to eliminate it.

As Martin Fowler points out, the two fundamental characteristics of ASDEs are their focus on adaptability rather than predictability and on people rather than process (Fowler 2000). As you read through this book, you will see just how fundamental--and challenging to the status quo--these two principles really are. Being Agile means accepting that outcomes are not predictable and that processes are not repeatable. It even implies that as process repeatability increases, project predictability decreases.

Peter Senge uses the term "mental model" to identify the perspective, set of assumptions, stories, and beliefs that each of us carries in our mind that provide a context for thinking (Senge 1990). In organizations, the collective set of mental models defines an overall cultural context. Companies that are heavily sales oriented differ from those that are heavily engineering oriented. Companies whose driving strategy is customer intimacy differ from those whose driving force is product innovation. Companies whose mental model includes linearity, cause and effect, hierarchy, predictability, and control will operate very differently from one whose mental model includes collaborative networks, emergence, decentralization of power, and acceptance of unpredictability. One is Newtonian, the other chaordic.

A chaordic perspective draws on a complex adaptive systems model in which decentralized, independent agents (each of us) interact in self-organizing ways, guided by a set of simple, generative rules that create complex, emergent results. This perspective is examined in detail in my book Adaptive Software Development (Highsmith 2000) and is explicitly embraced in the philosophies of Agilists Ken Schwaber, Bob Charette, and Kent Beck.

XP provides an example of how methodology and culture fit together. At one level, XP is defined by a system of practices--pair programming, test-first development, refactoring, coding standards. But the values and principles of XP define a collaborative culture--how developers work together with customers, how individuals interact and treat each other as human beings.

The second piece of the interconnected web that defines ASDEs is the statement of collaborative values and principles. While it is difficult to characterize the Agile Manifesto in one word, "collaborative" seems to be the best single adjective. Values and principles shape the ecosystem. Without a set of stated values and principles, an ecosystem is sterile, reflecting practices but not the people who interact within it.

A collaborative culture includes people and their relationships within a development team and with customers, management, and partnering teams within or external to their own company. Human dynamics, communications, and collaboration may be known as the "soft" sciences, but in practice, they may be the hardest to master. Principles and values help define a culture--the environment in which people want to work.

The final component of an ASDE is methodology. The traditional definition of methodology includes things such as roles, activities, processes, techniques, and tools. Alistair Cockburn summarizes these components when he defines methodology as "the conventions we agree to"--the ways in which people work together on a project. In The Social Life of Information, John Seely Brown and Paul Duguid (2000) discuss the major differences between process (as used by the business process reengineering movement) and practice. Processes are described in manuals; practices are what happen in reality. Process centrists relegate people to second place; practice centrists place people first. Process centrists focus on explicit (written down) knowledge, while practice centrists focus on tacit (internal) knowledge. The ASDE model provides a practice-centered, rather than a process-centered, approach to methodology.

There are two reasons to pursue barely sufficient methodologies: value and innovation. Streamlined methodologies concentrate on those activities that create value and ruthlessly eliminate non-value-adding activities. Programming usually adds value; process management often adds overhead. Bare sufficiency means keeping the former and eliminating the latter. Second, innovation and creativity flourish in chaordic environments, not orderly ones. Barely sufficient methodologies are cauldrons for breeding innovation.

Methodology also relates to organizational model. Agile methodologies contain minimal processes and documentation and reduced ceremony (formality). Agile methodologies are labeled "barely sufficient" (Cockburn) or "a little bit less than just enough" (Highsmith), or "minimal" (Charette). However, this streamlining of methodology isn't based just on reducing work effort but, more important, it is based on understanding the chaordic world view--one in which emergent (innovative) results are best generated at the "edge of chaos," perched midway between chaos and order.

Practices (or techniques) are the lifeblood of methodology. Whether it's pair programming, Scrum meetings, customer focus groups, or automated testing, the practices of ASDEs, carried out by talented and skilled individuals, produce results.

In the software profession, we've used the words "methodology" and "process" for so long that they roll off the tongue and the pen without trouble. "Ecosystem" will take getting used to, but then, that's why I'm using the word--to foster a different perspective. "There is accumulating evidence that corporations fail because the prevailing thinking and language of management are too narrowly based on the prevailing thinking and language of economics," says Arie De Geus (1997). "They forget that their organizations' true nature is that of a community of humans."

To change thinking, we must change the language we use, so I use the word "ecosystem" to change our thinking about how software projects should be viewed. A chaordic perspective, a collaborative set of values and principles, and a barely sufficient methodology all combine and interact to form an Agile ecosystem. We cannot separate the three, at least in my mind, and I think most Agilists would agree. One could have a streamlined methodology but a linear, Newtonian view of organizations, and the result would not be Agile. One could have a streamlined methodology but a hierarchical, control-based work culture, and it would not be Agile. One could have a collaborative, people-oriented work culture but a rigid, predictive approach to planning and managing projects, and it would not be Agile. Agility requires all three.

The ultimate objective of this book is to describe new ways of working together to deliver business value to software customers. The heart of ASDEs is a core belief in people--their individuality and their interactions. It's impossible to discuss people and their ways of working together (ecosystem) without discussing values and principles. It's impossible to discuss values and principles without also discussing assumptions about how organizations do, or should, work. It's impossible to compare Agile approaches with non-Agile approaches using "methodology" as the only mechanism for comparison.

In closing, I need to state that I don't speak for anyone in the Agile community other than myself. I may interpret what Ken Schwaber says about Scrum, what Jeff De Luca says about FDD, or what Alistair Cockburn says about Crystal Methods, but they are my interpretations. In making generalizations about ASDEs, I'm sure I've made statements that would generate disagreement from 1 or more of the other 16 authors of the Agile Manifesto. However, I have talked to, corresponded with, or worked with all these authors, so although they are my own interpretations, they also reflect a deep sense of being part of, and contributor to, this community.

Although 17 individuals authored the Agile Manifesto, thousands support this effort. Many, many individuals have signed the Manifesto Web page, and an array of Web sites prominently display the statement "We support the Agile Manifesto." Agilists have stirred a healthy debate within the software development and project management communities. I hope this book will contribute to that debate and encourage others to join in it.

In a highly volatile software development environment, developers must be nimble, responsive, and able to hit a moving target--in short, they must be agile. Agile software development is designed to address this need for speed and flexibility. Agility describes a holistic, collaborative environment in which you can both create and respond to change by focusing on adaptability over predictability, people over process. Agile software development incorporates proven software engineering techniques, but without the overhead and restrictions of traditional development methodologies. Above all, it fulfills its promise of delivering software that serves the client's business needs.

Written by one of the leaders of the Agile movement, and including interviews with Agile gurus Kent Beck, Robert Charette, Alistair Cockburn, Martin Fowler, Ken Schwaber, and Ward Cunningham, Agile Software Development Ecosystems crystallizes the current understanding of this flexible and highly successful approach to software development. It presents the key practices of all Agile development approaches, offers overviews of specific techniques, and shows how you can choose the approach that best suits your organization.

This book describes--in depth--the most important principles of Agile development: delivering value to the customer, focusing on individual developers and their skills, collaboration, an emphasis on producing working software, the critical contribution of technical excellence, and a willingness to change course when demands shift. All major Agile methods are presented:

• Scrum
• Dynamic Systems Development Method
• Crystal Methods
• Feature-Driven Development
• Lean Development
• Extreme Programming
• Adaptive Software Development

Throughout the book, case stories are used to illustrate how Agile practices empower success around the world in today's chaotic software development industry. Agile Software Development Ecosystems also examines how to determine your organization's Agile readiness, how to design a custom Agile methodology, and how to transform your company into a truly Agile organization.