Automated Testing Fax Machine

A couple of events recently have demonstrated to me that different organisations approach the subject of automated testing^** with very different mindsets to the one that I adopt on the subject.

Firstly I interviewed a test candidate a couple of weeks ago who explained to me the automation approach at his employer. They were using an third party tool which drove the GUI screens through recording click locations on specific points of the screen, and validated results by comparing bitmaps of the resulting screens.

I also recently visited a test automation team in another company to share testing ideas. While I was there we compared the test automation approaches in our organisations. Being a large organisation they had standardised their approach around the former Mercury tools , now part of the HP testing suite. The approach was split along the traditional lines of performance testing and functional automation. As we discussed the latter, the team described a clearly defined process around the development of their automated tests. Testers working on scripted test cases in other teams would, on completion of the manual testing, submit specific test cases to the automation team to be converted into automated tests. These tests would involve driving the system through the GUI following the same steps as the 'manual' test case.
Whilst both of these processes were well defined and obviously achieving some success in the eyes of their adopters in both cases the development of tests would start with the recording of a human interaction with the system under test and then modify this. It was clear that both operations were constrained within the idea that the goal for functional automation was to automatically execute scripted, formerly manual, test cases. 

More than a fax machine

Being a sucker for metaphors (much to the annoyance of Phil Kirkham I'm sure), the resounding image conjured by this idea is one of a fax machine. This is a shining example of an application of technology being constrained to the recreation of the related pre-existing human activities. If we look, conversely, at the great strides that have been made in electronic communication over the years, each has arisen through taking a step further away from any association with physical activity from which they originated. The speed of Twitter communication, the richness of Facebook and the scale of email are a far cry from what could be achieved if we limited our imagination to the sending of physical paper documents electronically. All these different communication tools we now take for granted, yet for many organisations their approach to functional automation is still rooted in replicating manual activities.
I'm sure that there are many valid explanations why automated test approaches are rooted in this mindset:

• It looks easier
In the sales pitch and the conference stand, GUI automation is an easier sell as you can apply automation easily against the existing product. Not the case. In the case of the bitmap based tool described by my interview candidate, any changes to the GUI , even changes to button colour , would result in the need to re-record the test case and re-save a new set of expected bitmaps. As the tests accumulate so does the level of rework that is required when changes are implemented that affect the test results. With a GUI based tool the first test may be quicker to set up, but subsequently the overheads of adding further tests will not reduce significantly afterwards as each new test must be recorded in the same way and can take days to create. A test harness that drives business logic through an API is likely to cost more for the initial set of tests but sensible design should allow very easy development of further scenarios through variation and manipulation of the controlling data, which can be easier than having to record and edit each new test.

• It looks cheaper
The big sell here is that you don't need programming knowledge so you can maintain separation of test from 'expensive' programming disciplines. Again this is a false economy. Any saving in not needing programming input in testing operations I think is more than offset by the knowledge needed in the specific tool itself and the level of input could well be lower long term if the test code is sufficiently isolated from the test data that drives it. Additionally you have the cost of  reworking the tests more regularly with changes in the application. A more involved approach programmatically may involve a heavier starting investment but, designed well , the long term maintenance costs in both test application and test data should be lower. 

• It is easier to budget
In the team that I visited, the automation engineer would complete a ROI calculation based on the time taken to develop the test against how many times it would be executed and accept or reject the development of the case. If manual test cases are perceived as having the same value as the automated execution of the same cases it is easier to budget and calculate the ROI of automation. This is, however, a flawed idea. When a person executes a test then they are observing a much wider range of criteria than an automated check, whether this is part of the explicit expected result or not. Attributing the same value to a human executed test as an automated comparison against a restricted set of check criteria is massively under-selling the value of human observation. Nevertheless it is done, as it affords an easy calculation of the return on investment in test automation. Calculating the value of a fundamentally different approach which acknowledges the relative strengths of automated and human driven interactive functional tests is much harder and therefore the approach is harder to sell.

Flexibility of purpose

When implementing automation into a testing operation it pays to maintain awareness that you have flexibility of purpose in the approaches that you choose. Replication of a human interactive execution of scripted tests through the GUI is just one of many available strategies. I'd suggest that there are usually other more economical and robust approaches by which the same, and a great deal more, information can be obtained. Therein lies the key point for me. We have the power and flexibility to design test automation around the information that we wish to obtain, irrespective of what can be achieved and observed by a human interacting with a user interface. If that information relates to the behaviour of the GUI when presented with those exact inputs, then this approach may well be appropriate. If the information that is being targeted is the behaviour of the business logic when presented with specific inputs then it is unlikely to be the best way to achieve this.

In the case of my visit I observed that the GUI application was one of three applications that interacted with the business logic in question, which indicates to me the presence of a business API against which they could probably execute a lot of their testing. This would provide a much more stable and controllable interface than a GUI and the creation and maintenance of tests would almost certainly be a lot easier given the appropriate approach such as a keyword or script driven structure. As it was, the creation of a new test was a multi-day operation of recording and modification, with the result that a very small percentage of tests passed the ROI review. I provided some feedback to the team manager from what I had observed and the possibilities of extending their automation away from the GUI. He was very receptive and the team were enthusiastic to look at different ideas, and I'm looking forward to discussing this with them in the future as they look at their options.

Questioning the approach of automating through the GUI is not a new approach and I know there are many testers out there promoting the idea of automating below the GUI against the business logic. I'm not advocating this approach particularly, as that investment may be equally inappropriate depending on the context. What I am saying is, when looking at test automation the options available, both in the tools chosen and in the testability of the application itself, extend well beyond what is possible as a human through the existing user interfaces. What I would advocate is using your imagination as to what is possible and drive testing activities, and the testability of the product, towards the information that you want to obtain. 
 
** whilst I know that some folks regard the term 'automation' and 'automated testing' as ambigous or misleading here I am using these terms based on a generally accepted understanding of their meaning. At some point I may try to tackle the ambiguity of these terms and the grey area of manual vs automated, but not in this post.

Copyright(c) Adam Knight 2009-2010