The many aspects of creating tests and exploring a test object
Quality engineering consists of many different tasks for a DevOps team. A part of quality engineering focuses on building quality into the IT system. Another part of quality engineering assesses the quality of the delivered IT system.
The building block "Test design", focuses on the creation and execution of tests, such as specifying test cases and performing exploratory tests.
Creating some tests and executing them may sound easy. But structured testing requires careful consideration of what to test and how to test it. We use the term "test design" for the complex whole of these activities, even though in some approaches to testing there is no actual up-front design involved.
Test design can be as simple as hitting some keys unprepared and see if a system works at all (ad hoc testing). And it can be as sophisticated as identifying test situations, assuring coverage by designing test cases, automating these in a test script and executing the script to automatically create a test report.
Both unstructured testing and structured testing will contribute to establishing confidence that the IT-solution will be able to deliver the pursued value. The preferred approach very much depends on the situation, especially with the quality risks involved. We do however already want to emphasize that a structured approach to testing is generally better than unstructured testing.
Two distinct ways of preparing and performing tests
The test strategy defines the intensity of (specific parts of) the testing of the test object. This test intensity is achieved by applying test approaches and test design techniques.
A test approach is a way of working for designing and executing tests. There are two groups of test approaches: experience-based testing and coverage-based testing.
The test approach is the approach that someone takes when creating test cases. To prepare and perform tests, we distinguish two overall approaches:
Experience-based testing is a group of test approaches that are based on the skills, intuition and experience of the tester. These approaches leave the tester free to design test cases in advance or to create them on the spot during the test execution, mostly testers will do both.
Experience-based testing is testing based on the experience, skill and intuition of the tester(s).
Within experience-based testing we identify various approaches. In the past we used to call them techniques but in our opinion a technique follows strict rules to design one or more test cases. The experience-based approaches by definition do not follow strict rules, instead they rely on skill, intuition and experience. Therefore, we call them approaches.
Every experience-based approach basically follows the flow as depicted the figure. The experience-based approaches are described in section Experience-based testing.
Coverage-based testing aims to demonstrate a specific type of coverage of one or another aspect of an IT system. This can be done by designing test situations and test cases with test design techniques.
Coverage-based testing is a structured approach to testing that aims to demonstrate a specific type of coverage by applying one or more test design techniques.
Below figure describes the coherence of the relevant terms in structured test design.
In coverage-based testing we always have a structured way of detailing information from one level to another. The starting point of testing is the value that is pursued by the stakeholders. In the development process this value is translated into requirements and specifications. Based on both value and requirement specifications, the quality risks are identified. Using the requirements specifications and the quality risks, the test situations are identified. Using test design techniques, these test situations can be covered at the right level by creating test cases and accompanying test data. Finally, the test cases are combined in test scenarios and (optionally) automated in test scripts. When these test scenarios and test scripts are executed, the actual outcomes are compared with the expected outcomes. There are two possibilities:
- If the outcomes match, this means the test has passed, the quality risk is covered, the requirement implemented and the stakeholders can be informed through the reports that this part of the pursued value seems to be achievable.
- If the expected and actual outcomes do not match, this means the test failed and a quality risk has materialized, and the requirement is not yet implemented. Therefore, the stakeholders must be informed through the reports that this part of the pursued value is not yet achievable.
More detailed explanation can be found in the Building Block "Coverage-based testing". Here we start with the test design entities, followed by the coverage groups and an overview of test design techniques.
Reporting based on test results
Based on the results of both experience-based and coverage-based testing, information is provided (orally, through documents or in a real-time dashboard) to the stakeholders.
The debriefing information from experience-based tests is integrated with the information from the coverage-based tests and this together is the basis for reporting to various stakeholders.
There are of course multiple stakeholders in any given situation; the reporting needs to be tailored to their needs. In general, we propose three levels of reporting:
- Detailed reporting for the members of the DevOps team and their direct contacts.
- Overview reporting for people that look upon the test object from a business perspective such as the product owner and users.
- High-level reporting for the stakeholders that are more distant to the team's activities such as high-level managers.
These levels of reporting are shown in the below figure, which also shows the relation with the test design entities.
More details on reporting in a DevOps situation can be found in "Reporting & alerting".
Always combine experience- and coverage-based testing
The use of experience-based and coverage-based approaches is not a matter of one or the other. It is always wise to combine both test approaches, sometimes more experience-based, sometimes more coverage-based, but never exclusively one or the other.
When the quality risks are relatively low, testers may decide to mainly apply experience-based testing. Still, if the tester for example encounters a boundary value during exploratory testing, it is very wise to apply boundary value analysis on the spot and make a test for each boundary value.
On the other hand, when the quality risk level (or any other good reason) is such that coverage-based testing is the main approach, then some experience-based testing is still a good thing to do. For example, investigating unexpected outcomes, or giving an important stakeholder a chance to see for themselves that the system actually works (because nothing adds so much to confidence as seeing it with their own eyes).
The choice for the division of testing over experience-based and coverage-based may also depend on the type of IT system that is tested. With a system that contains Artificial Intelligence, it is difficult to make an exact expectation of the outcome. Therefore, more experience-based testing may be a good choice. An IT system that relies heavily on calculations (for example, a financial application), may need very precise test cases and therefore will use more coverage-based testing.
Selecting and combining approaches and techniques
There are a great number of approaches and techniques, which one(s) should you use? You can choose static testing or dynamic testing or (often) both. But what should you do in your specific situation?
Of course, it starts with knowing your test approaches and test techniques. At first, they may look difficult to apply, but in general people will master a technique after using it for three or four times. It is also very important to realize that no one uses all approaches or techniques. You always use a selection. But how do you select the appropriate approaches and techniques?
Let's have a look at the figure above. The test basis is crucial when deciding which approaches & techniques are possible. Some techniques require very specific input, when that information is not available in the test basis, it is not possible to apply that technique; you will need to select another, or you need to find additional information for example by applying a static testing technique such as a walkthrough. Although an interview with a relevant stakeholder may also be sufficient.
You cannot test everything, so you need to select what to test and what not, and you also need to select the intensity of the testing, which is very important. This is mostly determined based on the quality risks. If there is a high risk you test with high intensity, if there is a low risk you test with low intensity. The intensity defines the required coverage which influences the number of test cases.
In any situation you need to consider the relevant quality characteristics (see the appendix for more information). These also impact the choice of approach(es) and technique(s). Often, there's an emphasis on functional testing, but functionality is just one of the many quality characteristics. The other (so-called non-functionals) will require other approaches and techniques.
An important factor to take into consideration are the skills available in your team. If no one knows how to apply a specific test design technique, it may be wise not to select that technique. Or you may want to arrange training and coaching to learn how to use the technique.
Finally, high-performing teams will define their own specific test intensity table, which is an easy tool to support the selection of test approaches and test design techniques in relation to the required test intensity.