Fractal Exploratory Testing

 

Last summer I was lucky enough to have an undergraduate, Tom, on an internship plement working on a testing project for my team, I wrote up a summary of the experience here. Tom contacted me last week sending me an interesting article on pair testing that he'd encountered through his studies and thought I might find interesting. This contact jogged my memory on something that came up during Tom's time with us that I'd meant to write about at the time and forgotten, so here it is, hopefully better late than never.

A simple graphic

During Tom's internship I spent some time introducing him to the approaches to testing that I have adopted in the company and the schools of thought behind them. In one session I found myself trying to describe the process of exploratory testing, and specifically explain the difference between ET and a more traditional scripted approach that Tom was familiar with.

I drew a horizontal line across a whiteboard to represent a products feature set. I explained that, with a given set of functionality, a scripted approach would traditionally be driven by a traceability matrix that ensured test cases covered each of the functional requirements. In the simplest representation of the approach the result would be a uniform distribution of test cases across the feature set, which I represented as another horizontal line above the first. We then discussed the fact that specific areas of that system may have greater complexity associated with the underlying modules, a weaker design or simply poorer code which would result in a higher risk of problems in that area. I shaded these on the first line and explained how these details might only become apparent at the point of testing and so test cases designed in advance may not provide any greater focus on these areas than any others.

I went on to explain the concept of exploratory charters and the idea of executing tests, learning from the results and applying further tests based on the information obtained. As I talked I drew a second line which undulated across the board, with sharp peaks of activity as I we encountered the shaded areas of higher risk. Tom quickly appreciated from this simple description how this approach allowed the tester to focus their attention around problem areas as they became apparent.

The beauty of recursion

Having used this simple graphic explanation of some of the benefits of ET, I then went on to discuss charters of testing. We discussed how each charter encapsulated an exploration of a function area or failure type in itself, and how each could recursively generate further, more focussed testing activity. As we talked I was reminded of a great talk by Elizabeth Keogh at an Agile Testing Exchange that I'd attended. Liz presented the idea of 'fractal bdd' where she described each story and epic as a smaller copy of a larger development activity, with bdd essentially comprising the same shaped process applied at many hierarchical levels. I felt that ET could be described in a similar way, and as Tom and I discussed the idea we elaborated our simple model.

If instead of a line we consider a circle of functionality. Much as with the line description, a scripted approach describes a uniform coverage of the circle with each test case a point on the circumference. With an ET approach, however, as each flaw in the circle is discovered we expand a new set of tests on discovery of the presence of an issue. This mini exploration will result in a more targeted testing exploration around this feature area, which we can represent as a circle off the original. Now imagine finding a more subtle flaw in that feature area, again you may expand your testing around that area to assess the problem and associated risks. In this way our uniform circle evolves into a fractal pattern, with branches of testing activity blossoming out from the original level as issues are identified and inconsistencies investigated. The recursive explorations may be smaller or larger than the first, just as a discovery in a simple retest can expose the need for a much larger investigation.

What I particularly like about this idea is that, in addition to an elegant way to contrast the natures of exploratory and scripted testing, it helps to provide some understanding of the potential issues people have with an exploratory testing approach. A uniform circle of test case 'points' is predictable. It is measurable in terms of circumference. As we complete the circle we can easily describe our progress and predict when we might finish. A fractal is more complex. We cannot measure the circumference, and it is harder to predict the remaining effort as we cannot know the scale of recursion activity that could be created at any point. All we can do is describe what we have done so far and some starting points of future activity in our remaining charters. This is understandably less appealing from a planning perspective. What we do provide, however, is a much richer and more elaborate level of detail through the process.

A usable model

I really liked this idea of fractal exploratory testing as compared to the uniform circle of prescripted testing, however I wasn't sure how well my enthusiastic ramblings on the subject had translated to Tom. I was, therefore, pleasantly surprised when Tom used this exact model to describe exploratory testing during his 'Agile Testing' presentation to my team. He presented the idea in his own words in a manner that both made sense, and reassured me that he had obtained a good understanding of the key benefits. To this end, the concept has served its purpose to me (although I shall certainly use the idea again) so here it is for general consumption.

References

• Liz Keogh, Fractal BDD, Agile Testing and BDD Exchange 2009
• What are fractals, Fractal Foundation
• What is ET?, James Bach Satisfice
• Resources on Exploratory Testing, Michael Bolton, DevelopSense

Main Image: http://www.goodfon.com/wallpaper/312410.html
Other images are my own - can you tell?