Crystal Clear: A Human-Powered Methodology for Small Teams [Paperback]

Crystal Clear: A few key rules to get a small project into its safety zone.

You have barely enough resources to get the system out. You don't want the team to write long documents, but they are forgetting things they are supposed to know about. You dislike heavy software development processes, but you want your team to work better than just randomly. You particularly want the software to come out the door successfully.

You considered sitting down and writing out the basic discussions the team should have and the work products they must be careful to attend to. You asked yourself:

What have other small, successful project teams done?

What practices do they use?

This book answers those questions. It is the result of ten years of debriefing successful small teams. Most of them repeated the same message:

• Seat people close together, communicating frequently and with goodwill.

• Get most of the bureaucracy out of their way and let them design.

• Get a real user directly involved.

• Have a good automated regression test suite available.

• Produce shippable functionality early and often.

Do all that, and most of the process details will take care of themselves.

This book sets out one of the most efficient and habitable methodologies you might hope to find: Crystal Clear. It is a human-powered methodology, most simply described as follows:

The lead designer and two to seven other developers in a large room or adjacent rooms, with information radiators such as whiteboards and flip charts on the wall, having access to key users, distractions kept away, delivering running, tested, usable code every month or two (okay, three at the most), periodically reflecting and adjusting on their working style.

This simple recommendation rests on both experience and theory. Software development can be characterized as an economically constrained cooperative game of invention and communication.1 The way the team plays each game has everything to do with the project's outcome and the resulting software. Crystal Clear tackles the economic-cooperative game directly, addressing where to pay attention, where to simplify, and how to vary the rules. A number of teams have shared with me—and now with you, through this book—examples of their rules, work products, and even office layouts.

Many so-called "best" methodologies get rejected by a team as being too constraining, too invasive, or too difficult. Crystal Clear does not aspire to be a "best" methodology; it aspires to be "sufficient," in order that your team will shape it to itself and then actually use it.

The IBM Consulting Group asked me in 1991 to write a methodology for object--technology projects. Not knowing enough about methodologies at that time to make the crucial decisions, and at the suggestion of my boss, Kathy Ulisse,2 I started interviewing project teams. What they told me was very different from what I had been reading in the books. In particular, they stressed aspects not covered in the methodology texts: close communication, morale, access to end users, and so on. It was not long before these issues separated in stark contrast the successful projects I visited from the failing ones. I came to see these issues, and not the design techniques, as the key to reaching a successful project outcome.

I got to try these ideas out as lead consultant on a $15 million, fixed-price, fixed-scope project of forty-five people. The ideas worked as advertised (coupled with a lot of creativity along the way), and showed themselves as core success factors. I wrote up the lessons learned from the project interviews and that project in Surviving Object-Oriented Projects (Cockburn 1998).3

One particular triplet showed up repeatedly: colocation of the team, frequent delivery, and access to an expert user. The differences in results between projects that did and didn't do these far exceeded any other short list of practices. This book builds from that triplet.

The projects in my career have generally been of the fixed-price, fixed-scope variety. People usually underbid on these projects, which means that the only way the team can deliver on time is by being very creative with their development process. Unlike most of the other authors of the Agile Development Manifesto, I came to the agile principles through the need for efficiency, not the need to handle rapidly changing requirements.

As a result, Crystal Clear is well suited to the fixed-price context. If you are in such a situation, use the planning, communication, and reporting mechanisms I describe to meet your (probably unrealistic) deadline, and just be careful not to change the requirements at the start of every iteration. If you have an exploratory project in which the requirements are unknown or fluid, that is fine. Then allow the requirements to move at the start of each iteration.

Crystal is a family of methodologies with a common genetic code, one that emphasizes frequent delivery, close communication and reflective improvement. There is no one Crystal methodology. There are different Crystal methodologies for different types of projects. Each project or organization uses the genetic code to generate new family members.

The name "Crystal" comes from my characterization of projects along two dimensions, size and criticality, matching that of minerals, color and hardness (see Figure 7-1).

Larger projects, which require more coordination and communication, map to darker colors (clear, yellow, orange, red, and so on). Projects for systems that can cause more damage need added hardness in the methodology, as well as more validation and verification rules. A quartz methodology is suited to a few developers creating an invoicing system. The same team controlling the movement of boron rods in a nuclear reactor needs a diamond methodology—one calling for repeated checks on both the design and the implementation of their algorithms.

I characterize Crystal methodologies by color, according to the number of people being coordinated: Clear is for collocated teams of 8 or fewer people, Yellow is for teams of 10–20 people, Orange is for 20–50 people, Red is for 50–100 people, and so on, through Maroon, Blue, and Violet. Crystal Orange is described in Surviving Object-Oriented Projects (Cockburn 1998), and its variant Crystal Orange/Web is described in Agile Software Development (Cockburn 2002). I find that, except on life-critical projects, people can add in the verification activities through the methodology shaping and tuning workshops.4

Crystal's genetic code is made up of:

• The economic-cooperative game model

• Selected priorities

• Selected properties

• Selected principles

• Selected sample techniques

• Project examples

The economic-cooperative game model says that software development is a series of "games" whose moves consist of nothing else besides inventing and communicating, which is typically resource limited. Each game in the series has two goals that compete for resources: to deliver the software in this game and to set up for the next game in the series. The game never repeats, so each project calls for strategies slightly different from all previous games. The economic-cooperative game model leads people on a project to think about their work in a very specific, focused, and effective way.

The priorities common to the Crystal family are

• Safety in the project outcome

• Efficiency in development

• Habitability of the conventions (the developers can live with them)

Crystal has the project team steer toward seven safety properties, the first three properties of which are core to Crystal. The others can be added in any order to increase safety margin. The properties are

• Frequent delivery

• Reflective improvement

• Close communication

• Personal safety (the first step in trust)

• Focus

• Easy access to expert users

• Technical environment with automated testing, configuration management, and frequent integration

Crystal's principles are described in detail in Agile Software Development (Cockburn 2002). Among them are a few central ideas:

• The amount of detail needed in the requirements, design, and planning documents varies with the project circumstances, specifically the extent of damage that might be caused by undetected defects and the frequency of personal collaboration enjoyed by the team.

• It might not be possible to eliminate all intermediate work products and promissory notes such as requirements, design documents, and project plans; but they can be reduced to the extent that short, rich, informal communication paths are available to the team and working, tested software is delivered early and frequently.

• The team continually adjusts its working conventions to fit the particular personalities on the team, the current local working environment, and the peculiarities of the specific assignment.

Among the rest are trade-off curves that highlight the cost implications of different communication mechanisms, different project situations, and different strategies for concurrent development. I used the principles to derive Crystal Clear, but don't discuss them separately in this book.

The Crystal package includes selected sample techniques, including ones for methodology shaping, planning, and reflective improvement. Crystal does not require any specific technique to be used by any of the people on the project, so these techniques are included only as a starter set.

Each member of the Crystal family is generated at the start of a project by shaping a base methodology according to the genetic code. Since the situation changes over time, the methodology is retuned during the course of the project. Both shaping and tuning are performed fast enough that the time spent gets repaid within the project time frame.

Crystal Clear is an optimization of Crystal that can be applied when the team consists of two to eight people sitting in the same room or in adjacent offices. The property of close communication is strengthened to "osmotic" communication, meaning that the people overhear each other discussing project priorities, status, requirements, and design on a daily basis. This enhanced communication allows the team to work more from tacit communication and small notes than otherwise would be possible.

Because every company and project is slightly different, even Crystal Clear is not fully specified. The first steps in adopting Crystal Clear are to uncover your organization's strong points and weaknesses and to fit the recommendations of Crystal Clear around them to capitalize on the strong points and cover the weaknesses.

For some organizations, this is too much work. Crystal Clear is not for those organizations. It is for groups that want to build their own, personal, strong, and effective way to deliver software repeatedly.

Crystal Clear shares some characteristics with XP but is generally less demanding. You might think of it as a more laid-back alternative, a place to fall back to if XP isn't working for the group, or a springboard to get some agile practices in place before jumping into XP.

This book is part of the Agile Software Development series edited by Jim Highsmith and myself. The series describes both theory and practice for software development.

The theoretical underpinnings are discussed in four books: Agile Software Development (Cockburn 2002), Adaptive Software Development (Highsmith 2000), Agile Project Management (Highsmith 2004), and Lean Software Development (Poppendieck 2003).

The other books in the series pick up the thread either by describing a technique for a particular individual or role, a technique set for the entire team, or a methodology sample.

• We find attention to the role of project manager in Surviving Object-Oriented Projects (Cockburn 1998) and Agile Project Management (Highsmith 2004), and attention to the role of requirements writer in Writing Effective Use Cases (Cockburn 2001b) and Patterns for Effective Use Cases (Adolph 2002).

• We find attention to the entire team in Improving Software Organizations (Mathiassen 2001) OK and Configuration Management (Haas 2003).

• Specific agile methodologies are described in DSDM (Stapleton 2003), Agile Software Development Ecosystems (Highsmith 2003), Agile and Iterative Development: A Manager's Guide (Larman 2003), and this book.

Future books will follow the theme, adding techniques for collaboration and team health.

Some people will read this book as an introduction to agile development techniques, and be relatively new to pair programming, test-driven development, osmotic communication, economic-cooperative game, continuous integration, and information radiators.

If that is you, then read this book pretty much straight through, because you are the person for whom I designed the chapter ordering.

• I wrote the Chapter 1, Explained (View from the Outside), as an e-mail exchange between Crystal and myself to expose how these teams look to an outsider (remember, it was once all new to me, too).

• Chapter 2, Applied (The Seven Properties), is the most important chapter. It describes what the team is aiming for, not the procedure it uses to get there.

• Chapter 3, In Practice (Strategies and Techniques), should give you a handle on how some of this is done.

• Chapter 4, Explored (The Process), describes the cyclical development processes core to all agile (indeed all modern) methodologies. It is something in which everyone should be fluent.

• Just scan Chapter 5, Examined (The Work Products), on the first pass, since it is very detailed. Use it as an encyclopedia of work product samples when you get that far.

• Chapter 6, Misunderstood (Common Mistakes), and Chapter 7, Questioned (Frequently Asked), should answer questions about what counts as okay and not-okay variations.

• Chapter 8, Tested (A Case Study), gives you another chance to see the methodology from the outside, since it was written for people not familiar with Crystal Clear. It also contains an ISO 9001 auditor's analysis and recommendations, which shines light from a different angle.

• Read Chapter 9, Distilled (The Short Version), to see if it makes sense at that point. If it doesn't, you may have to go back to Chapter 7 to see why such a simple recommendation works.

Some people are fully versed in modern agile development, including test-driven design and continuous integration. If you are one such person, I suggest you go directly to Chapter 9. After that (when you are done snickering), read Chapter 2, probably the most important chapter in the book. My guess is that you will find some new ideas to try out in Chapter 3. After looking through those chapters, return to Chapter 9 to see if it all hangs together for you.

If you are giving this to your boss or manager, my hope is that the first chapter, with its e-mail format, is the kind of thing that can be read on the airplane or in the bathtub. A manager or executive should also read the Chapter 4, because learning how to fit a cyclical process into the organization is important.

Process or methodology designers are quite likely to turn straight to Chapter 5, because this is a standard way of evaluating methodologies (although in the case of Crystal Clear, quite insufficient). To complete the evaluation, though, you need also to read Chapter 7 to see the comparison with other methodology systems.

Finally, you will be ready to start. At this point, read the answer to the last question in Chapter 7, "How do I get started?", and work from there. That will take you back to the methodology shaping and reflection techniques, and Chapter 2.

I wrote each chapter in its own unique style and tone. There is a reason for this. People learning a methodology are in much the same situation as blind men trying to guess the shape of an elephant, each feeling a different part and coming up with a different answer. Each person's unique background causes that person to notice and look for different things. Therefore, the nine different chapters are written in quite different ways. I don't expect everyone to be happy with every chapter, but I do hope that everyone finds some chapter that addresses his and her individual background and interests.

I am indebted to many more people for this book than for any of my previous ones: people who told me their stories, people who tried out the ideas, people who contributed samples, people who reviewed the text, and people who supported me emotionally.

Most of the people who told me their project stories during the last ten years don't know how much value they provided as they explained—even apologized for!—their ways of working. They often said, "We don't use a methodology here, we just . . .," or "I'm sorry, we don't bother to do xyz, or maintain our documents, but we have found. . . ." It was only by comparing results across a number of projects that I discovered that in many of these instances no apology was ever needed and that there was a strength in the way they worked.

Certain people were pioneers in trying out these ideas out on their projects. Jens Coldewey was the first, back in 1998, Robert Volker in 1999, Géry Derbier in 2002, Stephen Sykes in 2003. Dr. Christopher Jones tried out an early version on his unsuspecting senior software engineering students (who used every imaginable implementation technology). His students showed me where my writing was ambiguous or poorly described. Stephen got permission for me to include both his field report and the auditor's analysis in this book, a huge contribution.

Pete McBreen kept reminding me that people are also valuable carryovers from one project to the next (something I knew, but which kept slipping out of my descriptions until Pete reminded me again). Jeff Patton and Andy Pols were continual discussion partners on the topics.

Quite a number of people contributed office photos and work samples. Their names are listed separately in the permissions section and next to their contributions in the book, but I should like to highlight their importance here. For me as a writer intending to create something useful to you as a reader, it was important to have real work samples and not dummied up toy examples. These people recognized that importance and contributed what they had. I expect that all readers will appreciate their contributions.

Luke Hohmann and Tom Poppendieck read everything I wrote with astonishing thoroughness and caught errors of omission, commission, and even intention. Tom came up with the idea of dating the e-mails instead of numbering them. (Duh, why didn't I think of that? Thanks, Tom.)

Rich Turner is one of the people who could help me with the CMM(I) discussion in the context of Crystal Clear, and I'm thankful for his advice and review.

The Silicon Valley Patterns Group, and especially Russ Rufer and Chris Lopez, who read the manuscript carefully not just once, but twice, giving their usual thoughtful comments and pushing back on me when they felt I had gone off track. Russ argued me down relentlessly on some of sections until I finally got his point.

A few people read it from the Web and wrote in with corrections and improvements. Thank you, Marco Cova, Todd Little, Alan Griffiths, Howard Fear, Victoria Einarsson, Paul Chisholm, Pierce McMartin, Phillipe Back, Todd Jonker, Chris Matts, Gain Wong, Jeremy Brown, John Rusk, Johannes Brodwall, and Toby Kraft for your eagle eyesight.

The people in the Salt Lake Agile Group round table even ran through a "design the box" exercise for Crystal one day. The top quote from that day was,

"I've used Crystal all my life, and I've never been the same!"

Thanks to Jim Highsmith, discussion partner and series coeditor for the many illuminating discussions that shaped our ideas, our book series, and the wonderful Agile Development Conference in 2003.

Thanks to my new favorite coffee shop, the Salt Lake Coffee Break, which stays open until 2:00 A.M., has interesting clientele, power outlets that work, and baklava and Turkish coffee for those rare moments. Thanks to my family—to Deanna for checking in with me at the coffee shop at 1:00 A.M. to see how the typing was going (and then letting me sleep in in the morning), and to Kieran, Sean, and Cameron for their positive regard of my writing habit.

Footnotes

1Described at length in Agile Software Development (Cockburn 2002) and recapped in the first answer of Chapter 7.

2Thanks for the brilliant advice, Kathy!

3The project debriefings ended up as the basis for my doctoral dissertation, "People and Methodologies in Software Development" (Cockburn 2003a).

4If your company develops FDA- or other life-critical "validated" systems, you may want to set up three base methodologies; one for a Clear (quartz) projects, one for Yellow or Orange, and a third for all of the validated systems. Those three should provide adequate basis for shaping to any of the projects in your company.