Test Intensity Table

Rationale

In traditional projects, it used to be common to write a Master Test Plan that contained the Test Strategy which described what test intensity should be applied to the various parts of the test object. (high risk ► high test intensity, low risk ► low test intensity).

In today’s digital age with short cycled development and maintenance iterations such a Test Strategy document is not opportune. Still, the testing efforts should be aligned with the expected business value and the potential quality risks of the parts of the system such as user stories or other functional and non-functional test units. But development and operations teams often struggle with establishing a suited variety of test approaches and test methods. Or they simply decide to test everything with the same intensity regardless of the risk level, which often results in running out of time and randomly selecting areas where limited or no testing will be done.

To enable development and operations (“DevOps”) teams to easily vary their testing efforts they can use a test intensity table.

A test intensity table guides the team in deciding how to achieve the desired test intensity by defining what test approaches and test design techniques should be used for which level of quality risk.

The test intensity is directly related to business value and quality risk. A test object (also called a system under test) is often quite large, meaning that the test intensity will not be the same for the whole test object. Therefore, the test object is divided into smaller pieces, the so-called test units. A test unit (for example a user story) that has a high value and/or high risk should be tested more intensely than a low value/low risk test unit.

But how can we differentiate the test intensity?

This can be done by applying different test approaches and test design techniques. TMap defines two groups of test approaches, the experience-based approaches and the coverage-based approaches. Within the experience-based approaches there are checklists and exploratory testing. Within coverage-based testing we see test design techniques such as process cycle test (for testing business processes), decision table testing (for combinations of conditions) and data combination testing (for detailed testing of data values). Usually experience-based testing and coverage-based testing are combined to best work towards confidence that the test object is fit for purpose.

An example of the differentiation of test intensity, in a situation where the test unit is about testing a business process:

• High value/risk:
  • Approach – Coverage based – Path coverage – Process Cycle Test – Test depth level 2 AND Experience based – exploratory testing.
• Medium value/risk:
  • Approach – Coverage based – Path coverage – Process Cycle Test – Test depth level 1.
• Low value/risk:
  • Approach – Coverage based – Path coverage – Process Cycle Test – Happy Path OR Experience based – exploratory testing.

TMap groups the coverage-based test design techniques in four coverage groups: process, condition, data and appearance. This principle of describing which test approach and/or test design technique to use can be applied for all different coverage groups, so that teams will have a quick reference to determine how to achieve the desired test intensity based on the assessment of business value and quality risk that has to be done during the refinement of new features or change requests. This assessment often is done using risk poker prior to establishing the number of story points (or any other measure of size of functionality).

An example of a test intensity table is shown below:

^{Test intensity table}

Teams that want to create a test intensity table may wonder:

• How can a team define their test intensity table?
• Do all teams in the organization use the same test intensity table or should there be several?
• What reasons are used to decide between various test design techniques?
• When should experience-based techniques be applied?

The starting point is that every team has their own test intensity table.

Because in general, teams have different tasks, different test objects, different technologies etcetera. Therefore, different teams have different needs for testing, which is reflected in a different test intensity table.

On average a team will have a need for approximately 8 different test approaches and test design techniques. Why eight? There are dozens of test design techniques, but normally a team works in a stable situation with always the same technology and the same people. Therefore, a subset of about 8 (plus or minus 2) test approaches and test design techniques is sufficient to cope with every situation.

This naturally implies that two different teams might require quite a different set of test design techniques and test approaches.

The coverage groups are a great help for teams to make sure that they have a good range of test design techniques. As a rule of thumb, a team must make sure that in their test intensity table they have at least one test design technique for each coverage group, so that they will be able to address every possible test challenge.

Still, teams must keep in mind that sometimes using a test design technique of one group will result in such a set of test cases that a technique of another group wouldn’t add any coverage. For example, if a Decision Table is used to create test cases for a decision in a process flow, then the process cycle test would never result in additional test cases for that decision.

Coverage-based testing and experience-based testing should always be combined. Experience-based testing will supplement coverage-based testing with examples that are important because they are special situations that wouldn’t come up using a technique. Also experience-based testing is mostly done in pairs or even larger teams which helps building confidence because the participants actually see the system at work.

Read more about test design technique and test approaches.
 

People Involved

The members of a cross-functional team together define their test intensity table. They may consult people form outside the team such as a test consultant, an agile coach, a product specialist and other relevant stakeholders.

Artifacts

The test intensity table describes the relation between quality risk (and/or business value) and the intensity of testing. Also the table describes which test approach and/or test design technique must be used to test at this specific intensity level.

Success Factors

A cross-functional team must make a test intensity table that can used for a longer span of time, at least for multiple sprints. Normally, however, short after the test intensity table was created, based on outcome of retrospectives the team, the table can (and must) be adapted to the experiences of the team and the stakeholders.

References

This building block was first introduced in the book “Testing in the digital age; AI makes the difference”.