It's In the Documentation - We're Covered

In my last post "Is it a defect? No but it is a flaw" I examined some of the terminology used in defining defects across our contractual interfaces. I explored in that post some interesting behaviours have arisen as a consequence of this kind of definition. In this post I'll examine what, for me, are some of the most interesting consequences which relate to treatment of the technical documentation of our products.

Let's review a typical statement from a customer Service Level Agreement (SLA).

Defect : a deviation from the documented behaviour

A logical conclusion from this statement is that the presence of a defect in the system is defined on the basis of inconsistencies between product behaviour and product documentation. The relationship between documentation and product therefore moves from one of support and instruction to one of contractual importance. In essence what the SLA wording is doing here is elevating the technical documentation to be the pivotal point on which discussions of customer problems will hinge. The consequences of this are profound, if sometimes amusing.

• I have had customer support tickets raised with customers asking specifically whether a behaviour is documented. Not whether it was intended, or desirable, but documented. When we replied that it was they agreed to close the ticket - we asked but never found out what their issue was with it.

• In an extreme case , one company I worked for actually removed all of the documentation, including help, from the application because of a customer contract based on adherence to the docs. They didn't want to run the risk of the out of date documentation being used as leverage against them so they simply removed it all.

It's not in the documentation - we're stuffed

Just as it is impossible to test every possible combination of behaviours in a piece of software, it is equally impossible to document all of the behaviours of a software system of any complexity. This inevitably leads to a set of characteristics of our software that are not documented. This disparity between documented and actual product behaviour constitutes something of an awkward 'grey area' when the contractual SLAs are considered and can make support discussions difficult. How do we define whether or not the system is adhering to the documentation when the behaviour in question is not covered in the documentation?

On the face of it I can understand the desire to tie the software contractually to do what we claim it will, however in practice the way that software is documented, sold and delivered means that this approach is far from ideal.

• The documentation is delivered with the software.

  For most software the provision of technical documentation is part of the software release process. I've rarely encountered a customer who insists on rigorously reviewing the documentation to check for inconsistencies with their needs before the software is actually delivered.

• Documentation is referred to on a task basis

  Technical guides are rarely read from cover to cover before people start using the product. Far more often the software will be taken first and the documentation referred to at the point that the user needs help with achieving a specific task. One of the biggest challenges in documenting a big data system, where it is crucial to consider data design up front, is to ensure that the relevant documents are read in time to influence decisions on tasks that affect the data structure, and not when the problems have been encountered.

• Documentation is not designed to legally capture the system behaviour.

  Good documentation is designed to help the user to achieve their goals. As I've learned from my team and from reading some of the excellent technical documentation blogs out there, such as 'Every page is page one', there is a huge amount of thought and skill in designing documentation to help users achieve their goals by providing them with the relevant information needed to help with each task. The primary purpose of technical documentation is not as a legally worded contract exhaustively detailing every nuance of our software to use as reference over service contract negotiations. It is there, as it should be, to improve the experience of users and stakeholders in the software.

Its in - we're covered

Whilst it is not possible to document every aspect of system behaviour, the importance of the documentation in defining defects can all lead to some adverse practices in our attempts to try.

When a customer raises questions over behaviour this can often result in a hurried search to establish whether it is covered in the product guides. The ensuing sigh of relief is tangible when we discover that it is included and we are able to refer the questioner to the appropriate guide covering the behaviour in question. It is much harder to argue the case that a feature is incorrect if it is clearly stated in the product guides that it works in that way. A system may be riddled with flaws, but if they are documented then we can argue that they are not defects according to our service agreements.

I know from personal experience how easy it is to fall into a pattern of gamification of documentation because of this. It is hard not to strive for a false goal of getting all of our adverse behaviours documented somewhere as release deadlines approach. Whilst there may be intentions behind this - helping the users to avoid pitfalls - we'd be kidding ourselves if we pretended that there wasn't also some element of self preservation involved. The release note in particular can end up as the documentation equivalent of the safety net, where last minute limitations are slotted in to cover ourselves for problems that are too tricky to address in the time remaining.

What not to do

It is a common mistake to introduce negative behaviours into a process by measuring something on the basis of it being easy. Whether measuring testers based on bug counts, or developers on lines of code, it is incredibly easy to encourage activities that focus on achieving measurable quotas at the expense of achieving overall goals. In the same way, placing contractual emphasis on the documentation is in my opinion inappropriate and detracts from its primary purpose.

Whilst I don't claim to have a clearly defined alternative approach, it seems clear to me that this is not the way to specify defects. I'm sure that technical authors everywhere, would breathe a collective sigh of relief if we find another way of defining our defects and free up the documentation to what it should be for - guiding users through understanding their options and achieving their tasks when using our software.

References

There are some great technical documentation blogs out there. The ones that I read most regularly are:

• http://everypageispageone.com/
• https://ffeathers.wordpress.com/
• http://idratherbewriting.com/