Part 4 of a series of Core values for Regression Testing
What is Conformity
to act in accordance with prevailing standards or customs
A second definition brings us closer to showing the holistic value testers can help bring to software
bring into harmony or accord
As software is released we naturally develop both explicit and implicit contracts with customers. These contracts include the functional and non functional requirements of the system. As testers part of our responsibilities lie in helping to ensure release to release that these contracts are not inadvertently altered or violated.
What is Complexity
The state or quality of being intricate or complicated.
or
a whole made up of complicated or interrelated parts
Testing is a complex activity involving arranging software into different states in attempts to discover potential issues. However there is a constant push and pull between what is considered a reasonable test and what is not. At any point in development life cycle, an issues complexity can cause it to be dismissed as outlandish attempts by testers to draw out bugs.
There is an inverse relationship between the number of steps, or complexity of input required to reproduce an issue and the likelihood that it will be prioritized for fixing.
During regression it may not feel like you are using all your testing powers, like you are barely scratching the surface of functions. In exploratory testing, when you encounter an issue how much time do you spend unwinding your actions? Attempting to distill the steps you took into the purest form so you can report your findings.
Regression isn't the time to dive extra deep, any value added is weighed against the impact of slowing release. Instead be clinical, surgical and precise when regression testing.
Uphold the Social Contract
Updates and changes to software can be functioning as designed but still violate the expectations customers have developed over time. Regression testing is a time for verifying that the norms established in the softwares existing functionality remain unchanged.
Skill in regression is related to knowing your users and their needs, not upping test case counts, and not searching for novel new ways to draw out bugs. Quality contributed to by ensuring stability. Maintaining users experience in spite of change, ensuring updates always add value and never detract.
It's like Scheduled Maintenance
Consider your motivations when you take your car into the shop for maintenance. You probably want it done in the most reasonable amount of time and the technicians to only do whats required. Outside of the required work, you only really care to be sure that the rest of the car is operating within acceptable tolerances.
Beyond that don't you get concerned about what the mechanic is telling you? Things seemed to be working fine before, why suddenly are these things wrong? It simply raises a slew of undesirable considerations. Is the work necessary? Is the car still safe? How Long will it take? What if they find something else? How much will all this cost?
Even if the information is valid you aren't really open to hearing it. In fact, it may cause you to question the quality of both previous work, and the current information.
When changes are requested that are outside of the context of expectations, uncertainty rises. Since one set of expectations has been violated, it's easy to set off a chain reaction of fear. In the face of fear, people turn to blame to provide a sense comfort.
No one would take their car to mechanic if they knew with certainty the car would function perfectly without it?
Similarly management sees regression testing as a just in case activity. Bugs are not expected but its not worth the risk of skipping it entirely.
Any issues raised during regression testing will trigger the same fears you get with bad news from the mechanic about your car. There is now risk and uncertainty where we expected none. How did this happen? Should we fix this? Why didn't we catch this sooner?How does this impact schedule? Quality overall comes into question.
How do you fight this reaction?
Multi-Point Quality Inspection
Bundled in with most cars scheduled maintenance is some form of multi-point vehicle inspection. Opportunity for selling additional services aside,this inspection provides a context for potential issues that are found. Its not some mechanic looking for opportunity, it's part of a standardized information driven review process.
Inspection checks are not about new tests. They aren't evaluating the design of parts. They aren't introducing additional or new stressors on components. Inspections are measurements of how the subject being evaluated conforms to existing standards.
Modeling regression testing as a quality inspection helps set expectations to avoid the blame cycle that can arise when bugs are found at unexpected times. Regression activities are then based upon agreed upon standards, part of a larger testing strategy. The testing does not have to be scripted but the expectations are clear and measurable.
Previous posts in the Regression Testing Manifesto Series
Part 1: Intent over Implementation
Part 2: Common over Complete
Part 3: Behavior over Bugs