(re)Defining Testing: Static and Otherwise

testing you gotta be kitten me meme

In my last post I took a look at some software testing definitions provided by the ISTQB. In that post I stated I did not think there was such a thing as static testing, which has sparked some debate so let’s dive a little deeper.

Static Testing

It’s defined by the ISTQB as:

Testing of a software development artifact, e.g., requirements, design or code, without execution of these artifacts, e.g., reviews or static analysis.

The Wikipedia entry for static testing redirects to Static Program Analysis which falls more in line with how I would describe it.

In Cem Kaners’ response to the Stack Overflow question on Why do some consider static analysis testing and others do not? explains that it depends on your definition of testing and provides two examples.

Execution of the program with the intent of finding bugs -Glen Myers

Which Kaner concludes under this definition, static testing is a misnomer since the program must be running in order for testing to occur. This narrows the scope of software testing to investigations only occurring while the program is running. Ok, that may seem obvious but it also exposes an implied context that the static testing definition ignores. When talking about testing in the context of software, the words software testing and testing can and often are used interchangeably. Following that logic static testing and static software testing are synonymous. Just off the cuff, does that sound right to you?

Consider other descriptors here that follow the same pattern: Performance Testing, Security Testing, Acceptance Testing, etc. None of these terms change the context of the system under test the way that static testing does. Static testing shifts the focus to artifacts related to the running software.

Kaner then gives his definition of testing at the time:

Empirical investigation of a software product or service in order to learn quality-related information about it

Even under this definition, he says it depends, but to quote him:

It doesn’t much feel like testing.

That sentiment pretty much sums up my feelings on static testing, but since it depends on the definition of testing lets take a look…

Testing

ISTQB

The process consisting of all lifecycle activities, both static and dynamic, concerned with planning, preparation and evaluation of software products and related work products to determine that they satisfy specified requirements, to demonstrate that they are fit for purpose and to detect defects.

It’s been pointed out to me that the ISTQB has some definite credibility issues in testing community, so I don’t want to beat a dead horse but that definition reads like an after thought to wrap other concepts instead of being the primary concern.

So let’s take a look at the context driven testing worlds testing definition from James Bachs’ post Testing and Checking Refined

Testing is the process of evaluating a product by learning about it through exploration and experimentation, which includes to some degree: questioning, study, modeling, observation, inference, etc.

Let’s contrast that with checking:

The process of making evaluations by applying algorithmic decision rules to specific observations of a product.

Adding more detail:

Checking is a process that can, in principle be performed by a tool instead of a human, whereas testing can only be supported by tools.

and

Testing is an open-ended investigation– think “Sherlock Holmes”– whereas checking is short for “fact checking” and focuses on specific facts and rules related to those facts.

Michael Boltons’ earlier post Testing Vs. Checking offers some further insight.

Testing is, in part, the process of finding out whether our checks have been good enough.

Bolton is also quoted in the Bach posts which describes my reaction to static testing:

there’s a world of difference between sheet music and a musical performance

These definitions do a much better job of describing testing, and as tester they resonate with my experience. But consider this you could change the word evaluate to write or codify in testing definition and you end up with an apt description of a software developer. After years of tension between testers and developers is the line that thin?

Static testing frequently occurs as a developer writes their code. Developers are evaluating requirements, specifications, reference material and investigating until the code is checked in that process isn’t completed and is open-ended. Along the way they may create unit tests to help confirm and define their findings, this then adds dynamic testing into the mix. The process may even repeat as code maybe formally or programmatically reviewed, and/or integration tests are written to test component interactions. This isn’t a new revelation, Michael Bolton describes this (better) in the Testing vs. Checking Is A Leaky Abstraction segment in Testing vs. Checking.

By definition developers are both static and dynamic testers, which essentially covers the fully defined test spectrum. Let that fry your synapses as you think about it. Using those definitions, I can see how managers and developers could infer they do not need to hire a person with the dedicated role of testing.

Maybe the real problem is that the terms testing and tester are both too general and too confining. I think I’ll start referring to myself as a developer of quality, and if questioned explain testing is just a part of my process…

  • Amit

    I read this a couple of times, and I couldn’t escape the feeling that static analysis is equated here with static testing (at least until the part where you started talking about development process as static testing, in which point I probably lost you).
    I want to touch a point that you mentioned and made me squint a bit.
    When you mentioned security performance testing, you said “None of these terms change the context of the system under test the way that static testing does” and my first reaction was “sure they do!”.
    Then I thought about it a bit more, and I think that the main difference between “modes of testing” is the questions that are being asked – When testing for performance I ask “How does the system handle load?, how much resources does it consume? how fast is it?”, etc. When testing for security I ask questions like “can I find information leaks? can an attacker cause damage? etc.” During that latter activity I might use tools that are being traditionally used for load, when I try to uncover insecure behaviors under stress situation (such as race conditions).
    In the same manner, static testing changes the questions I ask – I can ask “Is the feature designed properly”, or “are there any bad coding patterns stupid mistakes?”
    I can’t see any difference that can separate “static testing” from “testing”.
    Static analysis – yes, that’s checking, but the fact that Wikipedia redirects static testing to static analysis is a bug in wikipedia, not an indication that these activities are the same.

    • Static Testing is about testing artifacts, including requirements documents. A developer will interact with these requirements in a testing fashion. They have to develop an understanding before they can create anything. If they find issues in the requirements it’s no different than a tester raising concerns. Which depending on your definition equates to testing.

      I think static testing and static analysis are either synonymous or very closely related, not simply because wikipedia redirects. It is also a concrete example provided by the ISTQB in their definition and many others on the web. This and reviews basically encompass the process.

      Static testing is fundamentally different than software testing, by definition you are testing artifacts. Performance and Security testing actually involve the software, static testing just covers artifacts related to it. I think the term is misleading, and those actions related to it are better described explicitly using terms like review or analysis. It’s like we are mixing what testers do and what testing is.

      • Amit

        First, let’s check if I understand you correctly.
        I identified the following statements:
        1) A developer is testing requirements as part of the development process.
        2) Static testing and static analysis are very close.
        3) Other types of testing are involving running software, and this difference puts static testing as “not testing”

        If I understood these properly, my response is “no, no and no”.
        1) While it is true that developers can test the requirements, and sometimes they even do that (intentionally or unintentionally), I don’t think many of them do, since even if the actions look the same, the purpose and thought processes are very different. A developer reading the requirements is usually doing so as part of the design process, trying to figure out what should be built.
        I realized the difference when I heard this talk (https://www.youtube.com/watch?v=MriidhWFDHA) of a developer (Claudia Rosu) using tests as part of her development process. I remember that one of my impressions (besides “hey, it’s cool to see a developer that cares so much about testing”) was “I haven’t seen yet an approach to tests in such a functional manner – each phase of tests had a visible, tangible artifact that is used during the development process – BDD scenarios are used to uncover missing requirements and get customer-buy in and approval, front-end unit tests are producing a list of services that should be developed. The only part I felt familiar with when I heard this story, was a little “by the end, I do some exploratory testing to fill in for the gaps” that was presented in the beginning and not really discussed.
        Worse I get when some of the developers I work with are writing unit tests – they don’t test (or test in a very shallow manner) when they design those unit tests. Instead, they produce some artifacts that enforce the current state of the code
        In some manner, it is similar to the difference between listening and waiting for my turn to speak (random google result: http://www.barrett.com.au/blogs/SalesBlog/2012/2019/support-services/are-you-really-listening-or-just-waiting-to-speak/ ), a developer reading requirements isn’t usually looking for contradictions or missing parts, but rather for instructions that lead to “Ok, i can fulfill this requirement by doing that”, and will normally notice problems is they interfere with implementation (or are not to this developer’s taste). A tester reading the same requirement looks for problems and therefore is more likely to find some. That’s the difference between reading a document and testing it. In both cases you read, and probably take notes – but when testing, you are actively looking for issues.

        2. Static testing and static analysis are about as far apart from each other as testing and checking are (that is – the latter is a subset of the former). Even if i go to the ISTQB foundation syllabus (http://www.itcb.org.il/images/ISTQB_CTFL_Syll%202011-t%20-%20protected.pdf) , they mention that static testing (Chapter 3) is divided to reviews( 3.2) and to static analysis (3.3). And, let’s look a bit on the following scenarios:
        – A new GUI is planned implemented. Someone prints a mock of the new intended UI and starts looking for things that should be changed. Maybe even ask a potential user to use this mock – the software is not running. I say this is testing.
        – in a video you’ve probably seen (https://www.youtube.com/watch?v=ILkT_HV9DVU&t=11m35s) James Bach is standing before a classroom showing them a flowchart. They then continue to discuss the system and design some test cases – I say this is testing.
        – The requirements are being reviewed inside the team prior to getting developed. I say that in some cases this is testing.
        – As part of the periodic security activities we do, I created a threat model of our software. Most of the activity was creating a data-flow diagram, and then using it to find threats on our system. I say this is testing.
        – someone proof-reads the release documents and looks for places where the documentation is incorrect with respect to what has been developed. I think this too is testing (less sure about this one, though).

        The common thing for all of those is that in neither case there is a software running (and in the first three cases, there isn’t even a software that can run) but there is testing this can be (or is being) done. Therefore, they are all instances of static testing, In my eyes, it’s a bit more that just static analysis.

        3. I think that most of the examples I gave above demonstrate a testing that can fall into the categories you mentioned (we had usability and then security testing), and not long ago I paired with a developer who wrote unit tests. During that session of creating the unit tests we uncovered that simply wasn’t implemented. No static analysis tool can find missing requirements (unless it’s a tool that you can code your entire requirements system in UML form. In which case, guess where would be your bugs now?). Therefore, I don’t think we can fairly say that those types of testing requires the system to be in any state (maybe the one exception for that is performance testing, but I’n not sure about it)

        Also, Where do you categorize the dynamic analysis?There’s software running, but the processes and concepts are very similar to static analysis runs – is this part of testing since the software runs? If so, is it different enough from static analysis?

        • I’m wary of boxing in developers the way you describe purely from the stand point that they do this same characterization for testers and say good testers are unusual. I don’t think its outlandish to consider that they are testing things as they develop, Bolton describes this better in the Testing Vs Checking post I linked to in the post.

          Assuming you believe static testing is testing, there is no reason this testing cannot occur during the design process. In fact it’s probably most likely to occur here. If they are reviewing and getting clarification on a requirements document that completely qualifies as static testing. My point here is that is not software testing, it may be artifact testing or some other term but that just feels like word games. Its review and analysis over static content.

          I don’t believe static analysis and static testing are far apart, I feel reasonably comfortable for your GUI example also. There is a good chance the GUI is being reviewed against a common standard or style guide, internal or industry standard. However you refer to using a mock, if it is an interactive example I don’t think that qualifies under the static testing definition.

          For your third point, I’m not sure I follow. Uncovering unimplemented functionality through examining the code I would consider code review rather than testing. I don’t mean all static analysis occurs through tooling. A person can analyze a static artifact, I still consider this a form of review rather than testing, the meeting where people do this are usually called code reviews.

          As far as Dynamic Analysis, that seems like it would be part of something like system monitoring which could be a part of performance testing. Honestly though I think I’d need a clear context and definition for dynamic analysis.

          Thanks for Reading!

          • Amit

            You are correct in your caution of boxing developers in that point – since they can, and do test. Probably a better way of stating what I mean is to say that I see design and testing as two activities that are hard to keep in mind at the same time. Software design is an activity of learning what is needed and providing a solution, Preferably, minimizing the scope of the change to a minimum. Testing is the opposite – it’s about broadening the scope, looking for more context and learning about the system as a whole. Adopting the testing approach to development leads often to some sort of analysis-paralysis. When I see testers code, we tend to suffer from the same problems as developers. So, it will be more accurate to say that while developers do test (and they test quite a lot), I don’t think that someone reading requirements is necessarily testing them in any manner more than shallow, since reading requirements as part of the design process is not about learning the product, but rather about applying one’s knowledge about it to see how can one change it to fit the requirements.

            I don’t understand your separation of reviewing from testing. If we accept Bach’s definition that you quote, reviews meant to evaluate a certain aspect of the product, and in order to do that we are learning – we are questioning, we are modeling and we are inferring. I would say that this is definitely an exploration. All aspects of a software that are not the code itself can be tested only in a static manner – if we agree that requirements can be “tested”, I think we can’t escape on agreeing that testing the requirements is equivalent of reviewing them. If reviewing is not testing, I think the conclusion is that requirements cannot be tested, only reviewed.

            I don’t think I follow you in the GUI example – If I’m sketching something on the board during a design review, there’s no code running – so it’s static. Wouldn’t you agree that we can test this sketch? Or, to borrow an example from this usable security course (https://www.coursera.org/learn/usable-security) – If I want to design a face recognition camera in order to allow people entry to a parking lot (of a secured location), and I am conducting the following experiment: I take a piece of cardboard, place it in the exact location of my intended camera and ask the drivers to treat it as they would a face recognition camera. Is this testing? Is this dynamic? Since the system I want to test isn’t developed, I consider this to be a static form of testing.

            As for dynamic analysis, I was thinking about dynamic code analysis, which involves running the software (often an instrumented version of it) to determine some properties – code coverage, data validation etc. However, thinking a bit more – I would include also automated tools such as scanners (nmap, sqlmap), monitors (such as JVisualVm to check resource consumption) fuzzers, and some of the functionalities of attack proxies (burpsuite Owasp ZAP), namely the crawlers and automatic security alerts.