(re)Defining Testing

It's Funny Meme

It’s really hard to explain to people why testing is a valuable activity if you can’t describe what it is you are doing and why you are doing it. Intuition is a big part of testing, but intuition is fed by understanding. A good detective or doctor might have a hunch, but that hunch is supported by years of experience and education. If it’s not then it’s really just a guess.

So being able to speak clearly and effectively about testing is a big deal. A stepping stone in clear communication is a shared vocabulary. It makes sense then that an organization like the International Software Testing Qualifications Board (ISTQB) would make part of its certification understanding common terms in testing. To be completely transparent I am not certified by the ISTQB or all that familiar with their materials. I noticed their definitions being shared on twitter by the ISTQB Tester, and a few caused me to raise an eyebrow so I have selected a few to dig into from the ISTQB Glossary.

Accuracy Testing

Testing to determine the accuracy of a software product.

I question my understanding of a subject if I have to use the term I am defining within its definition, so it set my spidey-senses a tingling when I first read this definition. There are mixed feelings within the testing community about certifications, so my first thought was this might be someone trolling on twitter. This is not the case. Accuracy Testing as defined above is part of the Advanced Test Analyst body of knowledge.

At its core, isn’t any or all testing in some way determining the accuracy of a software product? In what context is the accuracy being measured? Is it the accuracy between the developers and end users expectations? Is it about precision, like the testing the department of weights and measures does on scales and other devices?

Compliance Testing

Testing to determine the compliance of the component or system.

Seeing a second example of the using the word in its own definition made me wonder if all the definitions would be like this. I understand the intent of this definition, it’s referring to verifying if software is conforming to the rules prescribed by a regulated standard or governing body. The definition doesn’t state that though, it’s relying on an implied context, which probably means that it’s not worth defining.

Would testing client integration with an external api really be referred to as compliance testing? I think I could make a case where it would fit that definition. However, I would seriously question a tester’s competency if they used compliance testing to describe that activity. The difference here is that I can have that conversation with a tester, and we might both come out of it learning from each other. Definitions are put into place to avoid needing these types of conversations.

Dynamic Testing

Testing that involves the execution of the software of a component or system.

This one marked is as part of the ISTQB Foundation level certification, and I have to say it was quite a let down. How exciting does Dynamic Testing sound, seriously the name sells itself. Looking at the definition, doesn’t testing itself require the software to be running? Seems like a non starter if you are trying to test without the software.

Static Testing

Testing of a software development artifact, e.g., requirements, design or code, without execution of these artifacts, e.g., reviews or static analysis.

I looked this one up in the glossary after seeing the dynamic testing definition, and thinking is there really static testing? No, that would be analysis, like the code analyzing tools JSLint or NDepend. But ISTQB beat me to the punch, and even included static analysis as an example.

I draw the line here, there’s no such thing as static testing. Testing is an interactive process, it’s cause and effect. Analysis of code, or requirements or other artifacts can help inform your testing, White Box Testing is fairly standard example. Saying looking at an artifact is testing is kind of like calling spell and grammar check static editing and sending a your writing to a book editor dynamic editing. These are just not the same class of things, both may catch errors or potential issues but just because they may find the same issue does not mean they are the same thing.

Test

All these very specific definitions for different types of testing made me wonder… How are they defining a test?

A set of one or more test cases.

Seems fairly obtuse, a test is essentially a term for quantification or grouping. That seems troubling, lets see what a test case is:

A set of input values, execution preconditions, expected results and execution postconditions, developed for a particular objective or test condition, such as to exercise a particular program path or to verify compliance with a specific requirement.

I’m not sure all aspects of a program are specifically required, software usage is highly subjective. I don’t have a problem with the test case definition, that is generally how I understand them. They are explicit and prescriptive, and based on that I find their value dubious. The big issue here is that a test is just a series of these test cases. Therefore testing is entirely scriptable right? It’s just a matter of documenting all the conditions…

Well, I see they also have a definition for testing, maybe I’ll save that one for another post…

  • Greg Gauthier

    Here are some terms to define:

    What is “accuracy”? According to the dictionary, it is “a quality or state of being precise”. But what is implied in that definition is what someone is *saying* about something. In otherwords, *software* cannot be “accurate” or “inaccurate”. But our *assertions* about it can be.

    And I would suggest that most of the assertions in the ISTQB glossary are highly inaccurate.

  • Amit

    Ha,
    I really enjoyed reading this post. Even though it felt a bit too easy to point at the various points of failure in the ISTQB approach (at least for the foundation level, the concept is “just memorize a bunch of definitions”. When I got my certification, I noticed that one can extend this approach and say “just memorize a bunch of definitions, even the wrong or senseless ones”).

    However, On one point I don’t think I agree with you – why wouldn’t there be static testing? If we agree that the division between “static” and “dynamic” is whether or not the software is running, than I would claim that there is a significant part of my testing that is static in that manner – when I test requirements, I interact with them – I pose them in various relevant scenarios, I look for places that can be misunderstood, for mistakes and incomplete requirements, I look for inconsistencies and contradictions, and generally speaking, I am being quite dynamic as a tester. Still – I don’t run the software. The same is true for code reviews – I trace the various flows, build an understanding of the program structure, construct test ideas, look for hidden assumptions (“sure, this remote call never fails to execute…”).
    So If I take Bach & Bolton’s definition for testing (http://www.satisfice.com/blog/archives/856), I am definitely evaluating the product and learning through exploration and observation (though without the experimentation part), so I’m testing. And since the code isn’t running, this is static testing (again, if we agree that “static testing” is defined by testing done when the product is not running.
    Obviously, this isn’t an airtight definition (for instance – how would one categorize background processing that happens during the weekendafter a session of interacting with running software?), but as a rule of thumb I find this division useful: There is a different skill-set required to interact with a running software than there is to interact with some written, unresponsive artifacts. Separating the two can help in developing the skills relevant for each of them.

    • Thanks for reading and commenting! I agree about ISTQB being easy to pick on, I just didn’t know at the time of writing how serious it was taken.
      I wrote a follow post on static testing (http://wp.me/p6vwxg-44) but in general, I don’t see it as software testing. It’s different, and while its not used in the same context I like Michael Boltons’ quote “there’s a world of difference between sheet music and a musical performance”