A lot has been written recently about the concept of Testing vs Checking (the first reference I have seen on the subject was Michael Bolton's developsense blog) The argument makes an important distinction between the 'Checking' that is performed by either a machine or human performing scripted testing against a set of narrow pass/fail criteria, and 'Testing' which is a sapient process of investigating quality, constantly reviewing and amending the approach based on discoveries made and the expertise of the Tester.
While I agree wholeheartedly that the distinction is relevant and necessary, I am not completely comfortable with the meme chosen to represent the concept, and the re-definition of terminology involved. A recent discussion on this point in the Agile Testing discussion group got me thinking on this point and trying to produce a suitable metaphor to explain my thoughts.
What I came up with was the concept of a doctor examining a patient. The doctor will perform a series of tests in order to make a diagnosis of the patient's condition. Initially these are likely to be general tests, or related to specific symptoms, however as information comes out of these tests, the doctor will use their knowledge and expertise to identify further tests to gather more detailed information and allow them more accurately assess the patient's condition.
Once a diagnosis is made, and a course of treatment prescribed, often the patient is required to be checked regularly to ensure that their condition does not worsen, and that the initial assessment and course of treatment remains valid. These checks will most likely take the form of performing some kind of test (e.g. a blood sugar test for a diabetic) and having some simple criteria to check against the result of the test to give confidence that things are still OK. The patient does not have the medical knowledge to reassess their condition or change their medication, they simply have the ability to perform a test and check the results against a set of check criteria. Should the results differ from expectation then the expertise of the doctor will again be required to re-assess the patient's condition and recommend the appropriate action.
The point that I am trying to make with this example is that, both the doctor and the patient are testing. One tests as part of and assessment process based on their expertise and knowledge, the other as part of a basic checking process to give confidence in the persistence of the previously performed assessment. To label one of these actions "Testing" and one "Checking" could potentially be confusing, and, to my mind is not necessary given the wealth of words we have that may be more appropriate. I'd personally find it easier to present the concept in the sense of testing being made up of two separate important strands, rather than attempt to redefine "Testing" away not only from its literal meaning but also the context specific meaning that it already carries in the development community.
about me
Adam Knight
A Sisyphean Task is one that involves solving the same problem repeatedly - which is a familiar feeling to anyone involved in creating software. I write about Software Product Management, Innovation, User and Customer Experience, Software Quality and generally tackling the Sisyphean Tasks we all face in trying to create great software products.
Blog Archive
Popular Posts
-
Software Product management isn’t about creating products... For any product folks reading this who have just spat out their coffee, allo... -
I’ve worked in a couple of organisations that were coming from a background of a services based delivery model and trying to move to being... -
I have recently found myself with an interesting insight into the software product talent market, both from the position of an employer loo...
Creative Commons
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.
The patient does not have the medical knowledge to reassess their condition or change their medication, they simply have the ability to perform a test and check the results against a set of check criteria. Should the results differ from expectation then the expertise of the doctor will again be required to re-assess the patient's condition and recommend the appropriate action.
That's one case. The problem with an overemphasis on checking is the assumption that as long as the patient's blood sugar is okay, then everything is okay. That is, the problemmatic case is the one is which the blood sugar reading is okay, but something else is out of whack, undetected by the checks. In (Agile) projects, that translates to the Narcotic Comfort of the Green Bar; "when all the acceptance tests pass, we're done".
Another worry that I have is with the idea that there is one and only one meaning for
"testing" (the context specific meaning, as you say) in the development community. I think there are several such meanings, hence my desire to distinguish between them.
I'd be happy to know whether this post (http://www.developsense.com/blog/2009/09/elements-of-testing-and-checking/) and this one (http://www.developsense.com/blog/2009/11/merely-checking-or-merely-testing/) address your concerns in the first part of your post, and if this one (http://www.developsense.com/blog/2009/09/testing-checking-and-changing-language/) addresses your concerns in the latter part.
Cheers,
---Michael B.
Michael,
Thanks for taking the time to comment. If Testing vs. Checking does gain general acceptance it will be thanks to the passion and enthusiasm you and James are showing in improving the language of the Testing Community.
The third link that you provide extensively covers the issue of the need to constantly redefine language. I agree with the majority of points in here. I accept that making the definition between human examination of a system and scripted "checking" of the same in terms of the level of information that each will provide is a necessary and important concept to raise within the Testing community. The concerns that I have relate to your choice of terminology to describe this distinction, specifically the labels of "testing" vs. "checking":-
1. When attempting to identify two distinct fields within an area of knowledge and expertise, why attempt to use the name that is generally accepted as applying to that field of expertise to label one of those distinctions?
As alluded to in your blog post, "I think people who are genuine students of their crafts would be able to cope by making an instant mental translation, and those who aren’t are unlikely to read books anyway". This is true, however, why introduce a conflict in understanding between those two groups of individuals when one is not necessary. I work with competent software testers who may not necessarily be following the latest trends in terminology. I find it much easier to introduce concepts to this wider audience when I am not trying to redefine terminology that already carries meaning in this wider community, particularly when there are plenty of other terms which would suffice. I have actually discussed the distinction between "testing" and "checking" within my organisation, however I chose to avoid using these labels for this exact reason.
Also, if you are going to use the term used to describe an area of expertise to describe a subset of that area, does this not introduce an issue in describing the wider more generic field of knowledge? If the term "Testing" is to be used specifically to relate to the process of manual exploration and assessment of software, how then does one describe the field that covers both this and the definition and execution of scripted checks? Yes, at present the term “Testing” may cover “several such meanings”, but do you not think that it would be clearer to provide clearer definitions for these within the wider scope of the existing label rather than exclude certain practices to narrow the scope of the existing field?
2. If you are going to make a distinction between two areas within a field of knowledge, why use terminology that does not imply the aspects of the subset which form the basis of your distinction
- You make an excellent argument as to why an automated check is not a sapient activity, yet the label chosen to distinguish the alternative does not imply any level of sapience. As you state yourself, the process of "checking" involves the execution of tests, so could validly also be described as testing. Why not use a term which carries the implication of sapient activity (e.g. Assessment, Appraisal) to clarify the distinction.
3. By using the terms "testing" and "checking" I personally feel that you are confusing the subject of the distinction. For me "testing" is something that is performed against a system in order to obtain information about that system, this may be automated, manual, scheduled or the result of an exploratory decision. A check is an activity performed against the result of a test to compare against an expected outcome and produce a boolean result. I feel that the distinction to be made is in the actions that are performed on the results of tests, and the decisions that arise out of these actions, rather than the actions performed against the system itself.
I'm sure that you have considered and have answers to these issues. If this is the case I would be more than happy to read further.
Many thanks, Adam.
My answer to (1) is that testing subsumes checking. Checking is surrounded by testing; checking is dominated by testing.
I have actually discussed the distinction between "testing" and "checking" within my organisation, however I chose to avoid using these labels for this exact reason.
That's cool. How do you describe them?
If the term "Testing" is to be used specifically to relate to the process of manual exploration and assessment of software, how then does one describe the field that covers both this and the definition and execution of scripted checks?
That would be a problem, except that, in my description, the term "testing" is not to be used specifically to relate to the process of manual exploration and assessment of software. The term "testing" is to be used for exactly what you're describing as the second thing: "the field that covers both this and the definition and execution of scripted checks". Testing is the wider stuff.
Where did I say "manual"? Perhaps the problem is that some people believe that exploration of a product must be manual. That's not the case. You can certainly use tools in exploratory testing. What makes testing exploratory is the mindset and the mandate of the tester. http://www.developsense.com/blog/2008/09/evolving-understanding-about/
I'm not redefining testing, as I see it. In fact, I don't have a definition of testing that I've authored. I use three:
1) Testing is questioning a product in order to evaluate it. (James Bach)
2) Testing is gathering information with the intention of informing a decision (Jerry Weinberg; paraphrased from his book "Perfect Software and Other Ilusions about Testing)
3) Testing is a technical, empirical investigation of a product, done on behalf of stakeholders, with the intention of revealing quality-related information of the kind that they seek.
When I say that testing is a process of exploration, discovery, investigation, and learning, that's an explication, rather than a definition. I'm emphasizing the interactive and cyclic and open-endedness of the practice.
When I say that checking is a process of confirmation, validation, verification, that is something that could be included within the larger process. My beef is with people who say that checking (confirmation, verification, validation) is all that there is to testing.
You may need (alas!) to read farther in the series for this to be more clear. These were blog posts, so the idea has been refined since I wrote it. Try this:
http://www.developsense.com/blog/2009/11/merely-checking-or-merely-testing/
So when I read your (3) above, I perceive that we agree entirely. Where I'm puzzled is where you get the impression that testing is a manual-only activity. The original post even incorporates this: "You might use automation to assist your exploration; perhaps you would use automation to generate data, to track coverage, to parse log files, to probe the registry or the file system for unanticipted effects. Even if you used automation to punch the keys for you, you’d use the automation in an exploratory way; you’d be prepared to change your line of investigation and your tactics when a test reveals surprising information."
Perhaps you interpreted "distinction" as meaning "polarity"; I don't know. But as I say, with respect to your (3) above, I agree. If you can help me to understand where I've said something that suggests otherwise, I'd be happy to address it.
Cheers,
---Michael B.
Michael,
When discussing the distinction with my team I used the terms assessment vs checking and they accepted this disctinction without issue, and appreciated the relevance of the subject.
As a practitioner under permanent employment I do not have the opportunity to attend as many conferences and therefore rely heavily on blogs and other reporting to keep in touch with current developments. I think that the idea of mutual exclusivity between testing and checking arise through general discussions and reading blogs on the issue within the testing community. For example Gojko Adzic's blog post Testing is not checking, checking is not testing, which reported on James Bach's presentation on this subject at the Oredev conference, certainly carried this implication for me.
As you suggest I cannot find anything on your posts that implies polarity, and thanks to your replies here I will have an excellent reference to provide to anyone who requires clarification on this point.
Regards,
Adam.
When discussing the distinction with my team I used the terms assessment vs checking...
Cool. The labels are far less important to me than the distinction.
I think that the idea of mutual exclusivity between testing and checking arise through general discussions and reading blogs on the issue within the testing community.
I suspect and regret that I haven't been clear enough on this myself. But your blog has afforded me an opportunity to do so. Thanks.
---Michael B.
It's probably worth summarising my position on this as a result of Michael kindly taking the time to clarify his case. I am much happier with the distinction of checking as a subset rather than as a mutually exclusive activity to testing. As mentioned I have already started to introduce this concept in my organisation and will continue to do so.
Am I going to start calling my automated regression/acceptance tests automated checks? Probably not, for the following reason. The checks that are in place in the harness do not check all of the information that the tests collect. The checks are in place as indicators of change in this information, however for each check the tests also gather and store additional information such as logs, test timings, memory usage information amongst other things which all support a more detailed manual assessment in the event that the check flags up an unexpected result, and also to monitor patterns of behaviour over time through analysing results stored in the regression results data archive. Just as I appreciate that checking is a subset of testing, it also applies in my organisation that the automated checks are a subset of the automated tests.
What I will aim to do is continue to make the distinction where it is relevant, for example in discussions on approaches to testing and the relative risks involved.
Post a Comment