Is it a Defect? No, but it is a Flaw.

Over the last few weeks I've had cause to spend some time thinking about the nature of software problems and the language that we use to define them.

The driving reason for this is that I've been looking over some old customer support service level agreements (SLAs) as part of our transition of ownership. Occasionally reviewing something familiar for a different purpose can provide new insights into the content of it. As I read the SLAs I was struck by the absolute nature of the terminology used in relation to issues encountered. To paraphrase one such agreement with a former customer:

• Defect: a deviation from the documented product behaviour
• Bug: See Defect.

Such statements are commonplace in contractual agreements , both within and between organisations. The text has clearly been structured for the purposes of avoiding ambiguity, and on the face of it this has been achieved. While the terms appear straightforward and unambiguous, the actual wording contains some interesting and possibly concerning implications for software delivery.

Over the next couple of posts I'm going to examine the concept of defects and their definition in contractual type agreeements in light of experiences in testing, customer support and documentation, to see how our behaviour in the field of software has evolved some interesting and adverse characteristics in response to these definitions.

What's in a Name?

The term 'defect' bothers me. The concept of a defect in terms of service level agreements such as the one quoted above have an incredibly narrow scope of applicability. As we can see in the above example, the definition relates specifically to a deviation from the documented behaviour. When compared with the range of ways that I have encountered that negative relationships can arise between software and customer the definition of a defect is restrictive. It simply does not cover the range of problem types that can occur in software use that I encounter on a daily basis and can impact on our contractual relationships.

The concept of a problem being defined as a deviation from the documented design leaves a wealth of potential problems undefined within the scope of such agreements:

• the design fails to consider some valid scenarios of use
• a performance issue whereby the software does not meet the customer expectation
• behaviour that is not explicitly referred to in the documentation
• an unintuitive interface
• an inconsistency with expectations based on related products or technologies

All of these things can potentially impact customer perception, yet sit outside of the definition of a defect in the terms of many SLAs.

From my experience running the technical support team of a data storage system I can honestly say that a significant proportion, if not a majority, of tickets that are raised by customers do not relate to defects per-se but rather areas of behaviour, such as the above, that are far more ambiguously defined. My policy historically in these matters has been to strive towards delivering excellent customer service over quibbling around the definition of defects. Even so there are cases where we have to manage the customer expectation due to their specific query shape not having been optimised over tables containing hundreds of billions of rows of data, which can sometimes be difficult.

Even when adopting a more flexible stance, the expectation and even preference from many of our customers can be on a rigid approach due to the fact that that is what their processes are set up for. The rigidity provides the impression of predictability - a desirable characteristic of the support service.

It is this exact rigidity, however, that gives the greatest scope for conflict when problems such as those I mention above arise. At that point it is entirely dependent on the supplier whether to take a flexible stance and provide the assistance the customer needs, or whether to state that the software is behaving as designed, quote the SLA agreement and close their ears.

An internal problem

A rigid definition of what constitutes a defect can also affect internal relationships such as that between developer and tester. There was a time when a significant portion of my working day was spent raising bugs during a waterfall testing phase and seeing them fired back in the bug tracker with the comment 'as designed' added by the programmer.

This "bug tennis over a waterfall" situation can be a soul destroying one for testers, when their efforts in raising genuine concerns over the product are met with the brick wall of a blank refusal to question the design. I'm fortunate, as are many folks working in collaborative teams, to not encounter this situation, but I've encountered it first hand in other roles. Based on prior experience I can understand the stereotypical tester complaint of not having complete specifications up front. When the concept of a defect is defined in relation to adherence to specification then without it, it can be difficult to justify the existence of a problem, as the tester simply can't prove the presence of a 'defect' in absence of a conflicting specification.

When I first introduced group exploratory testing on a middle tier data framework in 2004 the outcome was a collection of new defects but also a series of concerns over the implemented design. We fed these back to the product management team but with no mechanism to get the design changed these concerns went no further. These included issues such as a major inconsistency between the technical knowledge required to configure the user connectivity and the knowledge of the majority of the users.

Thankfully more of us are finding ourselves working in more collaborative teams, Agile or otherwise, where within the team at least there is less need for rigid Defect definitions. In my team the term 'bug' has historically been sufficient to cover all manner of potential problems from design through implemented behaviour and documentation. A bug is as much a placeholder for work as it is a defect in the system. How that approach will need to change as our bug processing comes under the scrutiny of a more traditional organisation structure remains to be seen.

Wherefore art thou defect?

Given the range of situations in which I communicate, even with a fairly informal intra-team definition of bug, I still find the need to be careful with terminology when communicating in other forums. As we can see in the SLA definitions above, 'bug' is synonymous with 'defect' in many contexts, including SLA agreements, and also in the ISQTB glossary of terms. Some, however, define a bug as a superset of defects, while for others it relates to a different entity entirely or an entity in a different stage in the software lifecycle.

Even if we limit ourselves to examination of just defects, what I have observed is that the definitions even within the software community, are inconsistent, sometimes dangerously so.

We've already seen that, in technical support context, a defect can be defined as a difference between behaviour and documentation. This also translates into software testing context, where a defect can be simplistically defined as a difference between actual and expected of a test - for example in testingmentor - a software testing website

While testing when a tester executes the test cases he might observe that the actual test results do not match from the expected results. The variation in the expected and actual results is known as defects.

ISQTB has the following :

defect: A flaw in a component or system that can cause the component or system to fail to perform its required function, e.g. an incorrect statement or data definition. A defect, if encountered during execution, may cause a failure of the component or system.

The IT Law Wiki has an interesting definition - not least for its brevity considering the unquestionable importance of defects to software law.

[a] problem or “bug” that if not removed, could cause a program to either produce erroneous results or otherwise fail.[1]

(As an aside - a positive element for me both of these definitions is that they reference the potential for the problem to occur - a key characteristic of software problems that I wrote about in this post.)

This following definition, a popular one in many websites, does consider the possibility that a failure to meet user expectation constitutes a defect, however it does so in the limited context of incorrect or unexpected results.

A Software Defect / Bug is a condition in a software product which does not meet a software requirement (as stated in the requirement specifications) or end-user expectations (which may not be specified but are reasonable). In other words, a defect is an error in coding or logic that causes a program to malfunction or to produce incorrect/unexpected results.

With the variation in these, and many other, definitions it is not suprising that there is some confusion. Witness this conversation on stack exchange, originally asking about the difference between a defect and a bug, for examples of the confusion that supposedly rigid definitions of terms an cause.

Not a Defect, but a Flaw

In response to these definition problems I've found the word 'flaw' to be an extremely useful one. I find myself increasingly using it in situations where defect is overloaded and may have undesirable connotations.

Although the terms defect and flaw have similar definitions they carry very different meanings when applied. The beauty of the word flaw for me is that it has layers and nuances of meaning. It is this very flexibility and lack of formal definition which provides value when tip-toeing over the eggshells of behavioural conversation. Even the Oxford English definition provides three applications:

• a physical flaw
• a personality flaw
• a flaw in a plan or design.

Picture what comes to mind if I described a car as having a defect, as compared to a car having a flaw. To me the former is a fault in a specific vehicle, a badly moulded or cracked component perhaps. The latter could as easily conjure up images of a general problem in the design which affects all cars of the model, from the major design flaw such as the fuel tank problems of the Ford Pinto through to the minor niggles that fill the pages of this discussion - Minor flaws in the cars you love - on piston heads.

The key for me here is that both flaws and defects can be defined in relation to an ideal, a model of perfection. With flaws, however, this ideal does not have to be achievable, and in some cases it is expected that it will not be so. I particularly like, for example, the parallels between the application of the term 'flaw' to software and to people. To say someone had a defect brings to mind physical defects such as heart defects and other conditions. To say someone is flawed carries a very different connotation. Even the most accomplished individuals in all fields, whether politicians, athletes, business people or any walks of life, have flaws. This doesn't mean that they need medical treatment or surgery - it means that they are not perfect, however this is in the context of perfection being neither expected nor attainable. Everyone has flaws, everyone's are different. People may work over time to improve on their flaws, but not doing so doesn't stop them from being successful, wonderful people.

I like to think of software in the same way. The principle of the defect has at its heart the unrealistic notion that perfection is achievable if we could simply find and remove all the defects. There is no better deconstruction of this notion than Jerry Weinberg's excellent - "Perfect Software and Other Illusions About Testing". Even though most acknowledge that it is not possible, the implication in the majority of defect definitions is still there that the only thing between us and software perfection is our inability to find and fix all of the defects.

A Flawed Idea

This principle that the defects are all that separates a piece of software from perfection is one I think we need to move away from. All software is flawed just as people are. Software evolves through human endeavour. Throughout its creation it will have been subject to innumerable human decisions over workarounds, limitations and compromises. Even applications targeted at the same job by different teams will have areas of relative strength and weakness irrespective of their defects. Some designs that work well in some instances may even constitute flaws in others - the phrase 'one of their greatest strengths was also their greatest flaw' is sometimes used to describe people but could equally apply to software, or products of any kind. The simplicity in running one piece of software, for example, may be a flaw in one situation where greater configuration flexibility is required, whereas the rich feature set of a competitor may constitute unnecessary complexity in another.

I tend to use flaw particularly when discussing product behaviours that work as originally designed, but when the design has some limitations of disadvantages in applicability in certain scenarios. This may be a known limitation in capability of running certain shapes of query, a relative weakness in support of one data type over another, or simply a process that is more complicated than it needs to be. The phrase 'one of the flaws of the system is that it doesn't handle storage of large binary objects as well as other data types' carries a whole different meaning if I swap the word defect in there.

If I examine our bug database, what is clear to me is the behaviours captured within it are not just defects, they are flaws. This is a good sign as it means that the testers are looking beyond simple programmatical errors to identify a range of potential problems. For example recently a bug was fixed to return a useful output in our diagnostic capture process. It wasn't a requirement of the original command, it was simply that on using the feature it didn't make a lot of sense not to have it there. Now you could argue for days over whether this was actually a new user story, or a defect through a missed requirement, however this simply reinforces my point. If we concern ourselves too rigidly with sticking to work that conforms to narrow definitions, then some useful and important might be ignored simply through the inability to define them as defects or justify them as new requirements, when in reality they are neither, they are flaws. I've been involved in feature discussions with customers where obvious and useful changes are ignored at the expense of trivial cosmetic changes, as the latter can more easily be categorised as defects and therefore fixes justified. It is saddening to see chances to improve products being overlooked simply because our categorisation of what constitutes a problem in the system is simply too narrow to include them.

It's in the Documentation - We're Covered!

In my next post I'll go on to discuss another worrying implication of defects being a deviation from documentation ...

Images

Many thanks to Shilpa J at cartoonistshilpa.com for the excellent cartoon - feel free to get in touch with me if you have any ideas on testing cartoons that you'd like - Shilpa is looking for inspiration!

Wikipedia - David - for many Michaelangelo's Daid was flawed - see http://www.globusjourneys.com/travel-stories/italy/michelangelos-david/ for some interesting criticisms of the 'flaws' in the statue from its hands to its nose. Few that have seen it (which I am fortunate to have done) would argue, however, that it is a masterpiece.

References

• Cem Kaner provides a very thorough breakddown on the definition of defects in this paper - "What is a Software Defect? - Cem Kaner, JD, Ph.D."
• ISQTB Definitions including defect and bug - "ISQTB Glossary of Terms"