Monday, 12 October 2015

Fractal Exploratory Testing Revisited

0 comment

In my recent post 'Blog Post Room 101' I discussed the situation where the ideas that we present perhaps don't hit the mark with others or have the staying power that we first hoped. In contrast there are ideas that can be reinforced through our reading and subsequent experience or adoption by others, and we find them developing over time. One such idea for me is the concept of Fractal Exploratory Testing that I first wrote about in 2013.

This is an idea that I've had good cause to review recently as Janet Gregory and Lisa Crispin included the idea in the Exploratory Testing chapter of their "More Agile Testing" book. I was both flattered and somewhat unnerved by this as I felt that the idea as presented in my original post was quite raw and not as fully formed as I'd have liked. What I find, however, is that the more that I review the ideas that I originally outlined in that post in light of my subsequent experience and other material that I've read on the subject, the neater the idea fits and I'm very pleased that it was included.

An exploration within an exploration

In her excellent book on the subject of Exploratory Testing, 'Explore IT' , Elizabeth Hendrickson uses the example of Jeffersons letter prompting the Lewis & Clark expedition to find a navigable route across the U.S.A as an example of a good charter. James Bach also references this expedition in his earlier work, including this paper explaining ET. I think this analogy of exploratory testing and exploration of territory is a good one and the Lewis and Clark expedition provides a particularly good demonstration of a successfully executed exploration. Whilst comparing their mission directly as equivalent to an individual test charter provides a useful analogy, an element that isn't necessarily apparent through such a comparison is the many layered nature of exploration demonstrated through the expedition.

Lewis and Clark's overall charter was well defined, however the decisions over how to explore and what resources to use during each stage of the expedition were developed through a series of smaller explorations that formed that greater process. Lewis and Clarke's charter was a high level one with a very open ended remit on hiring people and buying equipment to carry out the mission. Within that charter they undertook a variety of smaller exploratory activities, some planned, many prompted by the events and discoveries that they made during their mission. Decisions were made and refined on the basis of information gathered at each smaller stage of exploration and Lewis and Clark pursued new avenues of exploration and changed the resources used on the basis of the discoveries made.

  • On discovering that their boat was too large to navigate further up the river they used local resources and skills to create wooden canoes
  • On encountering a fork in the river with two branches of nearly equal size they spent days exploring both branches to decide on which was the Missouri
  • They experimented with a wire frame canoe covered in hide, experimenting with different hides to see which, if any, was most suitable
  • On discovering no suitable hides to line the canoe they abandoned this approach and explored locally to find more wood to fashion traditional canoes

The success of Lewis and Clark's mission relied heavily upon their willingness to alter their approach as they went along. Imagine instead if Lewis and Clark had decided up front that the larger boat was the only means by which they were going to explore the river and they would go to any lengths to achieve that goal, or if they'd decided that the wire framed canoe was the only means by which they would navigate the smaller sections of the river, would they have been as successful? I don't believe so. The success of their mission came about in a large part due to their ability to experiment and learn at each stage of their activities and redefine their approach as a result of the discoveries they were making, even if it meant completely abandoning an approach in light of evidence that it was ineffective. What Lewis and Clark were doing, in the process of undertaking one large exploration, was tackling many smaller exploratory activities. Each of these activities possessed characteristics that were apparent in the overall mission, but on a smaller level, and each targeted towards a common goal defined by the larger mission, but each in themselves distinct.

Fractal Recursivity

I believe that the exploratory activities that are undertaken as part of a larger mission exhibit the characteristics which can be viewed as "Fractal Recursivity". The term fractal recursivity originated from the study of ideologies of language, and occurs when groups which share a common language differentiate themselves from 'others' based on nuances of accent. The fractal element occurs because the phenomenon can be observed at the local and regional levels just as effectively as at the national. In this paper Mary Antonia Andronis explains the core principle

Integral to the idea of fractal recursivity is that the same oppositions that distinguish given groups from one another on larger scales can also be found within those groups. Operating on various levels, fractal recursivity can both create an identity for a given group and further divide it.

So the core idea is that characteristics that can be used to differentiate items at one level can also be applied in the same way to differentiate sub-elements of those items at a lower level.

In "Explore IT" Hendrickson outlines a simple template for an exploratory charter, and in that template identifies three primary characteristics of an exploration:-

  • An area that is targeted for exploration
  • Resources that will be used
  • The information hoped to be gained

The ability to define and differentiate activities at various levels from the overall mission to smaller explorations within it through referral to these characteristics is what, for me, characterises exploration as demonstrating Fractal Recursivity. Definitions on the terms of these properties can allow an individual or team to understand the scope of an exploratory operation and how it differs from others, at whatever level that operation is defined. This allows the coordination of effort ensuring that the relevant tasks are undertaken, but without predefining the actions to be undertaken to complete them.

A hypothetical example

The Lewis and Clark expedition has been well covered by greater minds that I, so let's instead look at a hypothetical example as a means to illustrate the idea further. Imagine that you are leading a mission to sail to an island which you believe to be uninhabited. Your ship has crew, equipment and rations suitable for your mission, which is:

"Explore the island with the vessel, equipment and crew to establish the suitability of the island for establishing a permanent settlement"

This is quite a broad mission, so in planning your approach you would probably break the mission down into some activities that you intend to focus on at the start:

"Explore the coast of the island with the ship and rowing boats to establish locations for a harbour"

"Explore the forest behind the beach with axes and saws to locate suitable building materials"

"Explore the valleys inland with a small team of people and digging equipment to identify sources of fresh water"

"Explore the sea around the island with fishing nets to establish whether there are sufficient fish to constitute a useful food source"

These activities all form part of the wider mission, and all have the properties that Hendrickson defines as making up a good exploratory charter - resources, a subject area and information to obtain. Whilst some may share some of these properties, no two activities will share all of them, thus these smaller explorations can be differentiated from each other through these, just as the wider mission may be differentiated from other such missions on the same basis.

There will likely be a set of initial planned activities, however as these are being carried out, the teams will be gathering further information about the landscape and environment which may lead to further tasks.

"Find out the best route from the sheltered harbour we discovered to our preferred base camp location 2 miles up the coast using a compass and a machete"

Or they might possibly identify new risks which merit new exploration:

"Explore the inland swamp with spears to check whether those small crocodiles we saw have any potentially man eating cousins"

"Explore those ominous drumming sounds that we heard when exploring the forest using keen hearing and tip-toes to establish whether the island really was uninhabited"

The need for these activities can't necessarily be predicted before the exploration has started. The discoveries that are made through the initially planned explorations give rise to further activities which target the discovery of different information, and may require different resources to complete. The expedition is characterised by a series of explorations, some planned, some triggered by discoveries through the process, but each with it's own independent set of goal, information and resources, that contribute to the overall charter of the expedition.

The ability to define and differentiate at each level is key for coordinating the overall expedition. Imagine if you were leading this expedition and asked the team heading to look for water what they are doing and they replied: "We're exploring the island with a ship and crew to establish if it's suitable for a settlement". If the same question to those leaving to look for building materials in the woods, or those about to fish in the bay, elicited the same response then would be difficult and confusing to understand the various activities being tackled. The nature of fractal recursivity in exploration supports the ability to define and plan activities at each level at a sufficient level for the person coordinating at that level, whether that involves coordinating only their own activities or others as well.

Enough with Analogies Let's Talk About Testing

As with an expedition, in terms of testing software testing our testing activities will also be defined by an overall mission. This might be summarised at a very high level, such as for a certain software product:

"Explore this product using the skills of these development and testing teams, the budget available for tools and hardware, and the time available, to establish how well the software delivers our target value to the relevant beneficiaries"

It's a bit vague and high level, however a statement at this level could validly be used to identify the responsibilities of a testing group within an organisation. At a lower level, within the software development process on this product, a tester may be working on a user story. They will probably have agreed some target acceptance criteria to guide their testing of the story with the aim of obtaining information on how well the software meets those criteria. Again the testing at this level can be expressed in the form of a statement defining the testing mission :-

"Explore the new feature and areas potentially impacted by it using my test environment, tools and knowledge and about two weeks of the sprint to assess the new behaviour in relation to the acceptance criteria, risks and assumptions identified during elaboration"

Within the testing work on that story the tester might define a series of charters covering the areas that they intend to explore whilst working on that story, such as this one:

"Explore the inputs of the configuration screen using a range of valid and invalid inputs to identify functional gaps in the validation"

This is the level that exploratory testers might expect to define a set of 'test charters'. This level is appropriate for a tester to define and breakdown a piece of testing sufficiently to differentiate between their testing tasks, plan how they will cover the testing at hand and manage their testing in a structured way.

In her book Elizabeth Hendrickson provides some excellent examples of test charters that are too narrow in focus to be useful to a tester in planning their activities. Statements at a lower level, such as this

"Explore the date of birth field with the value 29/2/2013 to test how it handles invalid dates"

Would be more appropriately considered tests or test actions than charters. Similarly Hendrickson rightly points out that charters at a higher level are too broad to be a useful testing charter. I agree, however as I've shown above I think that it is possible to define testing activities at many levels that we might not necessarily perceive as 'charters' yet can be expressed in a similar way. Each level is made up of a series of activities at a lower level that can in themselves be expressed by an exploratory mission statement appropriate to that level.

Each activity also has the potential to generate further activities at any level as a result of decisions arising from the information obtained. The outcome of an exploratory test could be some more tests, or possibly the creation of a new charter. In some cases I've seen exploratory tests result in entirely new user stories or testing activities at the equivalent level due to the serious ramifications of the information discovered.

So is it...?

Generally with any model there are two important questions to assess its value:-

  • Is it valid?
  • Is it useful?

In terms of validity I hope that I've presented a reasonable case here for the validity of a fractal model of exploration. All models are flawed and this is no exception, there are limits to applicability, however as I stated at the outset, I've had good cause to reflect on the original idea and think that it stands up.

In terms of whether it is useful, that is less clear. It's certainly not a model that I refer to on a daily basis, however I do refer to the idea when introducing exploratory ideas to new testers in my organisation. I think that the value for me is in demonstrating that an exploratory approach can be applied equally at many levels. Exploration in testing is not limited to executing tests through charters, with all of the other rigid structures of boilerplate test strategies and rigid definitions around test stages and non-functional testing applied. Defining test strategy can equally be an exploratory activity whereby new testing approaches and methods are introduced as a result of the discoveries made. Rather than prescribing the exact approaches that will be taken in our high level test planning, instead I favour an approach of considering the high level testing activities as a set of overriding test missions which are the responsibility of teams or individuals to deliver. It is not up to the test manager to dictate how these missions are to be completed, as long as there is a clear understanding of the area, resources and information targeted. As long as we have sufficient coverage in the scope and responsibility of the defined activities at each level then the focus of test strategy and planning moves from predefining each testing activity to helping the team to obtain the skills and resources that they need to carry out their missions.

References

You Might Also Like
No comments

Post a Comment

Whatsapp Button works on Mobile Device only

Start typing and press Enter to search