How to Break Software: A Practical Guide to Testing [Paperback]

* Apply inputs that force all the error messages to occur.
* Apply inputs that force the software to establish default values.
* Explore allowable character sets and data types.
* Overflow input buffers.
* Find inputs that may interact and test combinations of their values.
* Repeat the same input or series of inputs numerous times.
* Force different outputs to be generated for each input.
* Force invalid outputs to be generated.
* Force properties of an output to change.
* Force the screen to refresh.
* Apply inputs using a variety of initial conditions.
* Force a data structure to store too many or too few values.
* Investigate alternate ways to modify internal data constraints.
* Experiment with invalid operand and operator combinations.
* Force a function to call itself recursively.
* Force computation results to be too large or too small.
* Find features that share data or interact poorly.

The attacks for system interface are:

* Fill the file system to capacity.
* Force the media to be busy or unavailable.
* Damage the media.
* Assign an invalid file name.
* Vary file access permissions.
* Vary or corrupt file contents.

Each of the attacks is presented using the subsections:

* When to apply this attack.
* What software faults make this attack successful?
* How to determine if this attack exposes failures.
* How to conduct this attack.

This approach leads to a very thorough demonstration of how to perform rigorous software testing in a limited amount of time. If I ever teach a course in software testing, this is what I will use as a text.
The book includes a CD containing two software testing tools, one of which I wish was available when I was developing software. While it is running, you can move a slider to have it bind memory resources and learn the point of memory use where your software performance begins to suffer. This is very useful, and is much easier than trying to load up many other applications.
Software testing is a critical area of development that is still in the process of being codified into patterns for reuse. This book demonstrates many of the currently available strategies and should be read by all members of testing teams.

As a test designer, myself (and a competitor of Whittaker's) I can certainly find things to nitpick about this book. But I won't do that here, because the big picture is far more important. That picture is simply this: if you are confused about what to do to uncover problems in software before it ships, EVEN IF you have no specifications to test from and EVEN IF no one listens when you rant about "quality assurance processes" they should follow, then there are only a few testing books yet published that will help you. This is one of them.

Most books that purport to be about testing are really about something else. They're generally about planning, or process, or mathematics, or graph theory. Often, they're about making models of software so that you can demonstrate that there are indeed jillions of paths through a given piece of software--hardly news to anyone who's bothered to think about it for a while. Sometimes they're about the underlying theory of the thing you're supposed to be testing, such as "Web applications" or "security". All of these are useful things to think about, to be sure. Many of these books are large, and this one is small. I would venture to say, though, that few books talk about actual bugs as much as this one does, and provide such entertaining, cringeworthy examples.

This book is about testing, and it's about thinking about testing. It provides a set of theories of error, and follows these with worked-out examples of using those theories of error to find bugs in real software. What a concept.

In some reviews of this book, you'll find pious pronouncements about process; you'll see one that complains that this book doesn't have anything about testing J2EE applications; or that this book somehow applies only to Microsoft software. Those reviews all represent valid points of view, equivalent to the valid point of view that Moby Dick is a book about a big fish.

Some of the information presented is quite basic. Mind, as a tester, testing trainer, and user of software, I've seen a lot of software--a LOT of software--not Microsoft products, some written in Java, built with well-defined process... but some pretty basic bugs. Mission to Mars, anyone?

Some reviews also seem to believe that there is One True Way to develop and test software. That may be true, though I doubt it. But either way, it's unquestionably true that the followers of The One True Way are in the extreme minority, and the rest of us testers have to live by our wits, work under pressure in chaotic organizations, and find important bugs quickly.

The book inspired me to think about the way that I approach a piece of software that I haven't seen before. I know some things about the underlying operating system (whatever it may be); I know something about the way data is represented in binary coding systems (whichever one might be in use at the time); I know something about the construction of programs (irrespective of the programming language); I know something about the way the program interacts with humans and other software. I also know something about the way programs and programmers can screw up--that is, I know something about certain risks. As a real tester in the real world, sometimes that and the program are all I have to work with. Nonetheless, I can use those things to find bugs effectively. Besides, even if I do have a specification, it's invariably incomplete, or wrong, or out of date, or so thick as to be unreadable in the time I have to test.

The book is fun to read, too--some of the fun is in the Microsoft-inspired schadenfreude, and some is the relaxed, conversational style of the writing. One nice notion expressed in the book is getting together with other testers and talking about bugs for fun. Good point--I believe that people learn more easily when they're talking to each other and having fun.

So this book helped me by providing an example of a taxonomy of software interfaces and theories of error, and ways of attacking the software based on those interfaces and theories. I have my own theories of error, and my own models, too; this book helped me to think about them and refine them. It's not the One True Way of Software Testing. Good thing, too: there isn't one.

Don't get me wrong: I would love to have a perfectly written specification to which the software completely conformed. If I were confident that such a thing were possible, I would never have to test; by that fantastic definition, the software would work perfectly.

A good book should help and inspire you to think for yourself. If your mind is closed to extending the ideas in this book (or any other), you probably won't like it much--but then you probably won't be able to function very well when you move to a different development culture. That is (sad to say) you won't be a very good tester when you leave your cocoon. In fact, if your mind is closed, you're probably not a very good tester now.

On the other hand, I believe that this book is very useful if you keep your mind open, accept its lessons and examples, and apply them to your own projects, your own environment, and your own thinking. We need more testing books like it.