Customer Oriented Software Quality Assurance [Paperback]

This is a comprehensive, practical "How-to" guide to customer-focused software quality assurance, for organizations of all sizes and types.The premise of this book is simple: your customers are the best judge of software quality. Therefore, your customers must be an integral part of your QA program. Learn how to design a QA program that builds on your customers' expectations. Understand how to select the right metrics, test methods, types and tools. Finally, walk through QA program development, and consider the appropriate role of formal evaluation programs such as ISO 9000 and SEI CMM.For anyone faced with building or improving a software quality assurance organization, or building quality into software products. Titles include: QA managers, test managers, software development managers and senior IT executives.

In the Eyes of the Beholder


“The customer is the ultimate judge of product quality.”


Ask a software developer to justify a claim that their company produces quality products, they will likely tell you, “We follow coding standards and hold code inspections, our organization is registered to ISO 9000, we eliminate all known defects before shipping a product, and each product is thoroughly tested. Our tests cover 95 percent of the code!” Quality is defined in terms of the processes used to develop and test their products, the test methodologies followed, test procedures employed, and empirical test data. Statements such as, “We pushed the system to its limits for 24 hours and encountered no failures,” “we ran the regression test suite and discovered no defects,” and “we went through four beta test cycles that lasted over two years,” are often made to support the claim. Now ask that same software developer to think of a quality software product, or any quality product for that matter, that they've recently purchased. Then ask them to define quality as it applies to that product. Most likely, their reply will include statements that have little or nothing to do with processes, procedures, and test results. As a customer, they are more concerned with such things as the product's reliability, availability, performance, and the responsiveness of the company's service and support organizations. Statements such as, “All of the defects have been eliminated” and “it was thoroughly tested” are replaced by, “It is one of the easiest products that I have ever used,” “I really like the way they've organized the toolbars,” or, “I called their help hotline and received a quick answer to my installation question!”


Customers tend to define quality in terms of attributes that the product and the company producing the product possess, such as, it's easy to use, it's well packaged, or service and support are impressive. Another way to look at this is to think of each attribute as one element of a set that represents their definition of quality. Unfortunately, these quality attributes are often completely ignored by the company developing the product. Instead, the company's quality assurance (QA) goals and objectives are set based on an internal definition of quality that is often mandated in a corporate standards document that dictates criteria, such as, 85 percent code coverage and no severity one defects (a severity one defect is often defined to be a defect where a product feature or function is nonoperable).


If you accept the premise that your customers are the ultimate judge of your product's quality and you don't know their definition of quality, then how can you ever hope to produce a quality product? In this book, you will learn how to build a QA program that is based first on their definition of quality. I refer to this methodology as “customer-oriented software quality assurance.”


This book addresses and provides a means by which an organization can solve several common problems faced when producing software products:

• Complex software systems having hundreds of millions of possible test cases and test scenarios are rarely if ever completely tested due to the practical constraints of time and resources. How does one select a sufficient subset of tests and test scenarios so that the quality of the end product satisfies the customer?
  
  
  
• Metrics are used throughout the software industry to gauge product quality. A metric is a measurement with an associated desirable value. For example, the defects per thousand lines of code measurement has an associated desirable value of less than one defect per million lines of code. If a product has one defect per ten thousand lines of code, is its quality poor? Also, does this measurement ultimately mean anything? Or, is it more important to measure the impact those defects have on the customer as they use the product? How does one select the right metrics to gauge product quality?
  
  
  
• Tests are the primary means by which product quality is assessed. What does it mean when a test fails? Is the quality of the product poor? Perhaps, but ultimately, the value of a test lies in its ability to determine whether or not the product satisfies customer quality requirements. For example, is a test that places a load on the system five times greater than any customer will ever place on the system a good determinant of quality? How does one select the right tests to use?
  
  
  
• There are a number of popular quality assurance appraisal programs advocated today. Charged with selecting one, which is the best to follow?
  
  
  The chapters of this book are presented in chronological order. Chapter 1 presents an example of a quality attributes set. The quality attributes set is the single most important element of the customer-oriented software quality assurance methodology. It embodies customer quality requirements for a particular product or family of related products. The book begins with an example of such a set that is used throughout the book to clarify the methodology. Chapter 2 defines the process of building the quality attributes set through the construction of the example set presented in the preceding chapter. Chapter 3 is all about quality metrics. The two goals for this chapter are to develop an understanding of and appreciation for metrics and other methods of analysis and to define a process that can be followed to build a set of specific metrics from a quality attributes set. Chapter 4 discusses test methods. The two goals of this chapter are to review various test methods and test types and to learn how to select the ones needed to develop a test suite that will determine whether or not the product being tested satisfies or exceeds customer quality requirements. Chapter 5 brings it all together by defining the QA program. The goal of this chapter is to examine various activities that an can be engaged in and tools that can be used to assist with the production of a product that satisfies customer quality requirements. Finally, Chapter 6 reviews two important process appraisal programs: one from the Software Engineering Institute (SEI) and the other from the International Organization for Standardization (ISO).
  
  
  The primary audience for this book is anyone faced with the task of building products that must meet or exceed their customers' quality requirements. In terms of job titles, functions, and responsibilities, those who I suspect will benefit the most from reading this book include quality assurance managers, software test managers, software development managers, and their respective teams, and perhaps senior staff members such as directors or vice-presidents responsible for such functions.
  
  
  Throughout my career, I have studied and applied a long list of methodologies, techniques, processes, and procedures, attempting to produce software products of the highest quality. Along the way, I began to ask myself: Which one is best? I wanted to settle on a particular one that I could apply over and over again to achieve consistent desirable results. As I gave this particular problem serious thought, it occurred to me that to achieve that goal I would first have to define “desirable results.” Isn't the primary desired result to produce a product of the highest quality? Yes! But, what does that mean? I realized that I would have to find the elusive definition of quality before I could solve this mystery. This book is the end result of that effort. It's not based on extensive research conducted in libraries and research labs. It's based on years of ad-hoc research, trial and error, and my personal experiences.
  
  
  It is time now to venture forth to design a customer-oriented software quality assurance program, one that will enable you to produce products that satisfy customer quality requirements. As you read, keep in mind that the effort put into producing such products is only one piece of a larger effort that brings a product from marketing to design to production to sales to distribution to the customer and then to support and finally back to marketing. These functions can be thought of as nodes connected along a circle. The success of a product and ultimately the company producing it is directly linked to the quality of the nodes that form this circle and the quality of the communications channels that bind them. If you neglect any one node or the associated communications channels, you run the risk of collapsing the circle, ostracizing your customers, and losing the game.