Testing Object-Oriented Systems: Models, Patterns, and Tools [Hardcover]

This book is about systems engineering and software engineering as much as it is about testing object-oriented software. Models are necessary for test design--this book shows you how to develop testable models focused on preventing and removing bugs. Patterns are used throughout to express best practices for designing test suites. Tools implement test designs--this book shows you how to design effective test automation frameworks.

This book is intended for anyone who wants to improve the dependability of object-oriented systems. The approaches presented range from basic to advanced. I've tried to make this book like a well-designed kitchen. If all you want is a sandwich and a cold drink, the high-output range, large work surfaces, and complete inventory of ingredients won't get in your way. But the capacity is there for efficient preparation of a seven-course dinner for 20 guests, when you need it.

I assume you have at least a working understanding of object-oriented programming and object-oriented analysis/design. If you're like most OO developers, you've probably specialized in one language (most likely C++ or Java) and you may have produced or used an object model. I don't assume that you know much about testing. You will need some background in computer science and software engineering to appreciate the advanced material in this book, but you can apply test design patterns without specialized theoretical training.

You'll find this book useful if you must answer any of the following questions.

The material here is not limited to any particular OO programming language, OOA/D methodology, kind of application, or target environment. However, I use the Unified Modeling Language (UML) throughout. Code examples are given in Ada 95, C++, Java, Eiffel, Objective-C, and Smalltalk.

My seven-year-old son David asked, "Dad, why is your book so big?" I'd just told David that I'd have to leave his baseball game early to get back to work on my book. I wanted to explain my choice, so I tried to be clear and truthful in answering. This is what I told David at the kitchen table on that bright summer afternoon:

As I was speaking, I realized this was the point of view I'd been struggling to articulate. It explains why I wrote this book and the way I look at the problem of testing object-oriented software. Testing is an integral part of software engineering. Object-oriented technology does not diminish the role of testing. It does alter some important technical details, compared with other programing paradigms. So, this is a large book about how testing, viewed as software engineering, should be applied to object-oriented systems development. It is large because testing and object-oriented development are both large subjects, with a large intersection. By the way--David hit two home runs later that afternoon while I was torturing the truth out of some obscure notions.

No one who helped me with this book is responsible for its failings.1 Dave Bulman, Jim Hanlon, Pat Loy, Meilir Page-Jones, and Mark Wallace reviewed the first technical report about the FREE methodology Binder 94.

In 1993, Diane Crawford, editor of Communications of the ACM, accepted my proposal for a special issue on object-oriented testing, which was published in September 1994. The contributors helped to shape my views on the relationship between the development process and testing. Bill Sasso (then with Andersen Consulting and now answering a higher calling) sponsored a presentation where questions were asked that led to development of the Mode Machine Test pattern (see Chapter 12). Bob Ashenhurst of the University of Chicago, James Weber, and the rest of the Regis Study Group raised more fundamental questions: What is a state? Why should we care about pictures?

The following year, Marie Lenzie, as editor of Object Magazine, accepted my proposal for a bimonthly testing column. Since 1995, writing this column has forced me to transform often hazy notions into focused, pragmatic guidance six times each year. Lee White of CASE Western Reserve University and Martin Woodward of the University of Liverpool, editors of the journal Software Testing, Verification, and Reliability, encouraged my work in developing a comprehensive survey, patiently waited, and then allocated an entire issue to its publication. Writing the survey helped to sort which questions were important, why they were asked, and what the best available thinking did and did not answer.

My publications, conference tutorials, and professional development seminars on object-oriented testing served as a conceptual repository and proving ground. Many of these materials, with the necessary changes, have been reused here. The cooperation of RBSC Corporation, SIGS Publications, the ACM, the IEEE, and Wiley (U.K.) is appreciated in this regard (see preceding Sources and Credits for details). The real-world problems and questions posed by my consulting clients and thousands of seminar participants have been humbling and constant spurs to refinement.

The patient support of Carter Shanklin and his predecessors at Addison-Wesley kept this project alive. Boris Beizer's steady encouragement, suggestions, and acerbic critiques have been invaluable.

Several adept programmers suggested code examples or helped to improve my own: Brad Appleton (C++ in the Percolation pattern and elsewhere), Steve Donelow (Objective-C built-in test), Dave Hoag (Java inner class drivers), Paul Stachour (Ada 95 assertions and drivers), and Peter Vandenberk (Objective-C assertions).

Drafts of patterns, chapters, and the entire book have been reviewed by many people. I am very grateful for the reviewers thoughtful and detailed feedback. Elaine Weyuker helped to debug my interpretation of her Variable Negation strategy presented in Chapter 6. Brad Appleton and the Chicago Patterns Study Group held two pattern writer's workshops that focused on the test design pattern template and early versions of the Invariant Boundary and Percolation patterns. Ward Cunningham commented on an early draft of the test pattern template. Several people reviewed test patterns based on their work: Tom Ostrand (Category-Partition), John Musa (Allocate Tests by Profile), and Michael Feathers (Incremental Testing Framework). Derek Hatley reviewed an early version of Combinational Logic (Chapter 6); Lee White, Regression Testing (Chapter 15); Doug Hoffman, Oracles (Chapter 18); and Dave Hoag, Test Harness Design (Chapter 19). Anonymous reviewers of an early version of the manuscript pointed out many opportunities for improvement. Brad Appleton, Boris Beizer, Camille Bell, Jim Hanlon, and Paul Stachour reviewed the entire final manuscript and provided highly useful commentary.

Finally, thanks to Judith, David, and Emily for years of support, patience, and encouragement.

Some of the author's previous publications have been reused or adapted under the terms of the copyright agreements with original publishers of Object Magazine, Component Strategies, Communications of the ACM, and the Journal of Software Testing, Verification and Reliability. See the Bibliographic Notes section in each chapter for specific citations.

The other sources, citations, and applicable permissions for the materials quoted on this book's epigraph page and chapter opener pages follow.

Epigraph Page From Geoffrey James, The Zen of Programming (Santa Monica: Info Books, 1988), Koan Two. Reprinted by permission of Info Books.
Chapter 2 From Lewis Carroll, Alice in Wonderland (Project Gutenberg etext Edition, 1994). In the public domain.
Chapter 3 From Michael A. Friedman and Jeffery M. Voas, Software Assessment: Reliability, Safety, Testability (New York: John Wiley & Sons, Inc, 1995), page 26. Reprinted by permission of John Wiley & Sons, Inc.
Chapter 4 Attributed to Edward A. Murphy, Jr., an engineer working on U.S. Air Force rocket-sled experiments. Sixteen accelerometers were attached to a test subject as part of the instrumentation. Each could be attached in two ways, but only one was correct. Murphy made this observation after discovering that all 16 connections were wrong. The statement was repeated by Major John Stapp at a subsequent 1949 news conference. In the public domain.
Chapter 7 From Lewis Carroll, Through the Looking Glass (Project Gutenberg etext Edition, 1994). In the public domain.
Chapter 8 A "ha-ha, only serious" slogan often repeated by Professor Robert Ashenhurst, University of Chicago Graduate School of Business. Printed here by permission of Robert Ashenhurst. Ashenhurst notes that, "My quote is in fact parallel to a saying by philosopher W.V.O. Quine, 'No entity without identity.' Although he was speaking in the context of ontology (part of the preoccupation of the branch called analytic philosophy), it is actually also apropos for object modeling without a change in wording, using the concepts 'entity' (= object) and 'identity' (= system id) as they are understood on the OO context."
Chapter 9 As quoted in Daniel A. Yergin and Joseph Stanislaw, The Commanding Heights (New York: Simon & Schuster, 1998), page 195. Reprinted by permission of Simon & Schuster.
Chapter 11 From Brian Marick, The Craft of Software Testing: Subsystem Testing Including Object-based and Object-oriented Testing (Englewood Cliffs, NJ: Prentice Hall, 1995), page 342. Reprinted by permission of Pearson Education.
Chapter 14 From H. Tredennick (trans.), Aristotle's Metaphysics (Cambridge, MA: Loeb Classical Library, Harvard University Press, 1933). Reprinted with no objection from Harvard University Press.
Chapter 15 From Eric Raymond, The New Hacker's Dictionary (Cambridge, MA: The MIT Press, 1991), page 205. Reprinted by permission of The MIT Press. Chapter 17 At a White House Press Conference, December 1987, President Ronald Regan said: "Though my pronunciation may give you difficulty, the maxim is, 'doveryai, no proveryai'--Trust, but verify." See George Schultz, Turmoil and Triumph: My Years as Secretary of State (New York: Charles Scribner's Sons, 1993). The Russian proverb translates as the imperative "trust, but verify," which rhymes in spoken Russian. My thanks to Nadya Moiseeva, Oksana Deutsch, and Igor Chudov who verified the spelling and translation in response to a query in soc.culture.russian.moderated.newsgroup. In the public domain.
Chapter 18 From The Histories (ISBN: 0460871706, J. M. Dent) by Herodotus, translated by George Rawlinson, edited by Hugh Bowden. Copyright (c) 1992, J. M. Dent. Reprinted by permission of Everyman Publishers PLC.

Use of a term in this book should not be regarded as affecting the validity of any trademark or service mark.