A Problem Halved - using a context specific cheat sheet to share product test heuristics

A term that is often used when highlighting the skilled nature of the job of software testing, particularly exploratory testing, is the use of testing heuristics. Heuristics are experience based rules of thumb that we use selectively to guide our approach to a problem, such as testing software features. Over the course of a testers career, (which, as I discuss here can be viewed as its own minor evolution) they will evolve a set of personal heuristics. We can apply these selectively in the appropriate contexts to increase our chances of quickly exposing the potential or actual problems with a software solution. The posession of an excellent set of heuristics can make a huge difference in the effectiveness of testing performed within a given timeframe.

Every solution is different

Over the course of time working on testing a product a tester will develop not only their general testing knowledge but also an excellent set of testing heuristics that relate solely to the context of that product. This knowledge is non-portable and valuable only within its present testing context, however within this context these heuristics can be the most valuable resources for targeted testing of the product in question. Some examples from my current context

• There is an internal boundary within our NUMERIC libraries between NUMERICs of precision 18 and 19 which adds extra boundary cases to any NUMERIC type testing
• Some validation on data imports is client side and some is server side, error handling and server logging testing needs to consider both types
• The query system follows a different execution path when operating on few (<10) data partitions so testing needs to consider both paths
• For our record level expiry, different paths are followed depending on whether some records in a partition, all records in a partition or all records in an import are expired

The knowledge of these facts adds a richness to any testing beyond that which might be obvious from the external, documented behaviour of product.

(Shared) knowledge is power

For some, their heuristics may be closely guarded secrets, the knowledge that elevates them above the competition or renders them invaluable within the organisation. My opinion, however, is that our goal is to produce the best software that we can as a team, and the sharing of knowledge that has arisen through our combined experience is essential.

Having read Elisabeth Hendrickson, James Lyndsey and Dale Emery's excellent Test Heuristics Cheat Sheet, I found this to be a great method of communicating simple high level heuristics. I felt that a context specific heuristics cheat sheet would be an excellent tool for sharing our internal testing knowledge and for use as a guide/checklist in exploratory testing of our product features. Using a similar layout to the testobsessed cheat sheet, I created a simple framework on our Wiki for our own context specific version for my team to use to document our internal heuristics. We break down by product area and then individual functions or operations. These are then annotated with key-word points that might need to be considered when testing that aspect of the product. If a concept merits further explanation then we link to a separate "glossary" where terms can be expanded upon to avoid cluttering up the sheet. This helps to keep the sheet as a lightweight reference tool rather than anything more involved. The page is maintained by the team, with each member updating the list if they encounter any behaviour that might be useful to others if testing that feature in the future. For example, an entry might look like

Record Level Operations

• Record Expiry - Expire Subset of Records from Tree ; Expire All in Tree ; Expire All in Import ; Record Level Delete ; Records On Legal Hold ; Purged Records

We've been working with this tool for a few months now and have found it to be very useful. We reference the heuristics in exploratory testing, in elaboration meetings and also charter reviews of story coverage as a mental checklist and to prompt new ideas. If new problems are discussed in reviews then we suggest adding them to the sheet so that it is maintained by the team and for the team. (We have not had to maintain the sheet for long enough to drop features or consider versioning which may cause minor headaches). Of course no checklist or cheat sheet should be considered exhaustive and we have to apply our own personal intelligence to every testing challenge, however, as a simple tool for sharing testing experiences and driving new ones, I can recommend it.
image : http://de.wikipedia.org/wiki/Benutzer:KMJ