Introduction Generic test agreements, Test strategy and Test plan
Quality engineering is about teams taking responsibility to deliver IT systems that have business value. Regarding the objectives of the IT delivery process, various stakeholders want information about quality, risks, time and cost. This information is gathered by measuring indicators which is done mostly by testing and other quality measuring activities.
The Quality & Test policy forms the basis for quality engineering and testing. But more details will be needed for organizing and performing the actual activities.
When organizing testing, the team(s) must define the information needed and the indicators that are relevant to measure data that is the basis for this information. After these indicators are known, the testing will be organized in test varieties, approaches, techniques et cetera. The indicators are used on a level of team or at value stream level. So the indicators are overarching, and not directly linked to, user stories or test varieties or other artifacts discussed in this section.
In this section we use the starting point that the people involved have already defined the relevant indicators that need to be measured by testing.
When organizing quality engineering and especially testing three important and related artifacts are:
- the generic test agreements,
- the test strategy,
- the test plan.
In this section we give an overview of the coherence of these artifacts and how to work on these. We start with the definitions:
|
Generic test agreements (GTA) |
Definition: Generic test agreements describe the overall approach for the setup and organization of test processes that applies to more than one project or release. General agreements on e.g. the test process, standard strategy, estimating method, procedures, organization, communication, documentation, etc. |
|---|---|
| Test strategy | Definition: A test strategy is the allocation of quality measures to balance the investment in testing and to make an optimal distribution of effort over test varieties and test approaches to give insight in test coverage and test intensity. Often this is based on the quality risk levels and the pursued business value. |
| Test plan | Definition: A test plan is the description of the general structure and the choices with respect to the tests to be executed and the way to supply information. The test plan forms the reference during organizing and performing of the tests and also serves as an instrument to communicate with the client. The test plan is a description of the test project, including a description of the activities and the planning. So, a test plan is NOT a description of the tests, e.g. test cases, themselves. |
Generic Test Agreements (GTA)
The generic test agreements describe the way of working (WoW) for a group of teams. This way a shared understanding and shared terminology amongst teams is promoted for easy collaboration.
The generic test agreements are meant to remain stable for a longer period of time and therefore should be described and stored in a way that people involved can easily access them (for example on an intranet, in the repository or in an easily accessible document).
Some organizations, projects or teams make use of a so-called generic test agreements (GTA) document. Often, in a GTA large parts of a (master) test plan are copied with reuse in mind. In many cases, such as in DevOps, outsourcing, offshoring, or in maintenance, the execution of test activities goes beyond a single release. To prevent new agreements on the what and how of testing having to be made for every release, these are stipulated in the GTA document. This contains the generic agreements on organizing and performing topics. This makes it the overall approach for the setup of quality engineering activities that apply to (all) future releases. Based on the GTA, it is then decided whether to adapt, for each release, the quality engineering activities.
In DevOps, a quality & test policy or GTA are not usually drafted as separate documents. But these are often part of the WoW, whereby – if necessary – a more detailed interpretation is given in a DoD.
Note: Although for historic reference we maintain the name “generic TEST agreements”, of course the scope is broader, to cover everything related to quality engineering.
Test strategy and Test plan
The test strategy and test plan may be described in a combined document or in separate documents, but other ways of communicating the test strategy and test plan are also viable (for example using a team board). Below we will describe examples of both ways and give hints & tips about this.
Note: Although the test strategy and test plan largely focus on testing activities, of course other subjects related to quality engineering in general can also be integrated using the Quality Engineering Strategy.
Why do we need these artifacts?
The main goal with the artifacts (Generic Test Agreements, Test Strategy and Test Plan) is to provide structure in the quality engineering activities. This way the people involved know what needs to be done, why, when and how. The main reason for this is to make the work more efficient by reducing the need for communication. Also it will reduce the number of faults (and thus rework).
Another important goal is to communicate which indicators were selected to measure whether the objectives of IT delivery are achieved, this information is used by the stakeholders to establish their confidence level that the pursued business value will be feasible.
Test strategy
The test strategy aims to align the quality engineering efforts with the pursued business value and the potential quality risks of the IT system. This is valuable because teams and/or the people in the teams, may struggle to establish a suited variety of test approaches and quality measures. When they struggle, sometimes they even simply decide to test everything with the same intensity regardless of the quality risk level.
A high-performance cross-functional team will usually define their test strategy for multiple sprints and make adjustments whenever that is needed.
Basic steps in creating a test strategy are:
- Quality risk analysis
- Create (or adjust) the Test strategy table
- Create (or adjust) the Test intensity table
- Fill in the selected test design technique(s) and/or quality measure(s)
(Note: an extensive description of this process is to be found here)
Step 1 – Quality risk analysis
Agile teams commonly use planning poker to determine the size of a user story. To determine the quality risk level of a user story, they extend planning poker with risk poker.
First, they determine the relevant quality characteristics for that user story (e.g. functionality, usability and security). Then they use the poker cards to determine the risk for each of the quality characteristics of the user story. Usually they use the planning poker cards 1, 2 and 3, to determine the chance of failure and the impact separately, and the result is used to determine the risk class.
The result is that for each user story the risk levels (usually expressed as a risk class) are known (and registered on the story card). Commonly used risk classes are A (high risk), B (medium risk) and C (low risk). And keep in mind that if the result of the risk analysis is that a specific part (e.g. user story) has NO risk, this means that not only it doesn’t need to be tested, but also it doesn’t need to be developed (because if it fails nobody has a problem, so nobody really needs it: no risk – no test – no dev).
Step 2 – Create or adjust the Test strategy table
Using the user story – quality characteristic combinations the test strategy table is filled in with the risk class and then per test variety the test intensity. The test intensity is identified with a number of bullets, • for low intensity, •• for medium intensity, ••• for high intensity.
With a high risk class (A) at least one test variety must have high intensity, with a low risk class (C) no test variety should have more than low intensity.
Of course, the test variety that gets high intensity in case of a high risk class, should correspond with the quality characteristic. So in case the high risk is for usability, there should be a high intensity for usability testing which may be a separate test variety or may for example be part of acceptance testing.
This will result in a table as shown below.
Step 3 – Create or adjust the Test intensity table
A team should have their own test intensity table that gives their initial choices for implementing low, medium and high test intensity. This table contains for each group of test design techniques how the low, medium and high intensity will be achieved. Of course, other quality measures can also be used to achieve the test intensity, especially in case of some specific quality characteristics.
The contents of such table normally will be stable over a longer period of time but of course when new situations arise the table must be adjusted, therefore every time the team must consider whether or not the table needs to be changed.
The table below is the example from the book “Testing in the digital age”.
Step 4 - Fill in the selected design technique(s) and/or quality measure(s)
For each combination of user story and quality characteristic, select from the test intensity table the relevant test design technique or quality measure for every test variety.
This will result in a table as shown below:
Also an excel-template for a test strategy can be downloaded.
This way a test strategy is available that describes how the test effort is divided over the various components of the IT system and the different test varieties.
The Test plan
The fundamental reason for having a test plan is to make clear who does what at what time.
In a high-performance IT delivery situation generally the test plan doesn’t need to be an actual document. Often the test plan is just a number of (sub-)tasks on the team’s board (such as Scrum board or Kanban board).
However, when multiple teams work together on creating IT systems to support an end-to-end business process, there will be a need for an overarching plan (for example as a result from the PI planning ceremony in SAFe®).
While planning the testing activities, use the QA & testing topics as a checklist to make sure that no relevant activities have been forgotten. Some topics will have been completely covered in the generic test agreements and/or in the test strategy, others need to be detailed in the test plan (and in case the test plan is not an actual document the activities need to be registered and monitored as tasks).
Mutual agreement
The people involved need to agree with the generic test agreements, test strategy and test plan. In a small-scale situation (one team) this will be relatively easy. In a large-scale situation (multiple teams for one value stream) this may require specific attention. In SAFe® the agreement should be part of the outcome of the PI planning ceremony.
Quality Engineering Strategy and -Plan
The test strategy and test plan largely focus on testing activities, but in high-performance IT delivery other subjects related to quality engineering in general can also be integrated. Based on this a team can decide to extend it to a Quality Engineering strategy and a Quality Engineering plan.
Sources:
- Enterprise DevOps Report 2020-2021, by Microsoft & Sogeti
- Book: Quality for DevOps teams
Related building block:
Quality risk analysis & Test strategy
Creating a Test strategy and Test plan in high-performance IT delivery