Requirements Document
The answer to “What is a Defect?” may seem obvious. Team members usually think of ‘defect’ as meaning something is wrong. But, how do testers really identify what’s wrong when testing a program? Are defects always incorrect behavior or can they be something unusual or something that’s not great? Recognizing defects isn’t always as straightforward as it seems, but studying how testers identify defects can help define what they are. This article will examine ways to define ‘defects’ and techniques we can use to identify them.
The Origins of an Agile Workflow
One way to define ‘defects’ is to think about how testers identify them. Testers often start looking for defects by reading a requirements document, then examining the software to determine whether it meets the requirements. Requirements documents are usually written by some combination of the development team and the business stakeholders to explicitly state the business value that the software must present to the users. Any team member can refer to the requirements document to determine whether the software is working correctly. Sometimes the specification is called an oracle because it acts as one possible source of truth.
The requirements documentation is a useful way to identify defects. Any place where the software is different from the specification might be a bug. And, it may be tempting to define ‘defect’ as ‘any software behavior that is not described by the requirements document.’ However, most experienced software testers recognize that there will be defects that fall outside of the expected behavior defined by the documentation. These are generally recognized using other types of information or rules of thumb.
Checking for Consistency
One common way to identify bugs used during testing is consistency. A requirements document may indicate that an OK button will be used to confirm a change, but it may not indicate the color of the button. If all the other buttons used in the program are blue, but in one window the button color used is green, most testers would recognize that inconsistency as a defect. Additionally, testers might look for consistency with claims. If the sales materials makes a claim about the software that falls outside of the requirements document, making sure the program meets that claim helps the software meet customer expectations.
Consistency with other software products can be valuable to users. Since many available programs use an ‘X’ button in the upper right corner to close windows, including an exit comment in a menu instead of the typical ‘X’ could confuse users. Consistency makes the program easier to use, and since testers recognize the value of usability, they find consistency to be a good way to identify defects.
The "Goldilocks" Heuristic
Another common heuristic identified by Elisabeth Hendrickson is the ‘Goldilocks’ heuristic. Goldilocks reminds testers to try entering data into data entry fields that are ‘too big’, ‘too small’ and ‘just right’. When using this heuristic, testers need to discover what is meant by ‘too big’, ‘too small’ and ‘just right’ for the particular field they are working with. Data that is too big, either numbers that are too big or strings with too many characters, can cause problems from miscalculation up to program crashes. Entering data that is ‘too small’, like null values or 0, can cause the same kinds of problems. Poor handling of unexpected input, like data that is ‘too big’ or ‘too small’, makes the program less useful to anyone who might enter that data, whether they need the data purposefully or erroneously.
Defect Definitions
Jerry Weinberg states that “quality is value to some person”. Testing is one way to discover what that value is, and how it might be threatened by bugs. James Bach defines the term software bug as “anything that threatens quality”. Putting these two definitions together extends the definition of a defect to “anything that threatens the value of the software to some person.”
It is important to think about who is referred to by ‘some person’. Often testers think of the person as the customer or the user. But often, the customer and the user are not the same person. Sometimes people affected by a piece of software never even see it. Consider a company whose structure includes an IT department that purchases, installs, and tests new software, field users who actually use the software, executives who review results from the software, and customers who depend of the services of the field users. A single piece of software might influence the workflow and experience of all these groups of people, even though the customers of the company may never see the software.
A more general way of thinking about defects is “anything that threatens the value of the software to some person who matters.” This definition allows testers to clearly state why what they have found is a bug based on what value is being reduced, and for whom. It further encourages testers to look beyond requirements documents in their analysis to find the problems that really matter.