Test Varieties

What are test varieties?

Definition

The term Test Variety aims at making all stakeholders aware that there will always be different needs for testing and thus different test varieties will have to be organized. Whether these are organized separately or combined depends on the situation. Test varieties are defined based on the relevant quality characteristics and other relevant perspectives.

Quality engineering is a broad area. Many different activities contribute to the end result. But how can a team determine what has to be done? Quality engineering, for example, involves the approach to requirements specification, to software development, to operations and also to testing. Sometimes team members have the misconception that testing is just one straightforward activity. But it is not. 
People are different. Projects are different. Systems are different. Environments are different. So, it would be an illusion to think that one-size-fits-all exists for testing. You need variety in your testing. But how? What should be your focus when organizing your testing?

All testing activities should collectively cover all important areas and aspects of the system under test: that's the main objective. Different people involved have different needs and expectations. Arranging your test varieties means having an overview of these needs and expectations and the relevant testing activities to fulfill those. 
This chapter will help define the necessary variety from different perspectives. There are many different possible reasons for bringing variety to testing. The sections below are by no means a complete overview of all possible considerations, you will have to determine the necessary variety in your specific situation.

When organizing testing, in the traditional view on testing the test manager had to structure the testing activities in a hierarchical way and based on quality characteristics. But also a test manager often distinguished various stages. Defined terms that were often used are Test Level, Test Type, Test Phase and Test Stage.

In today’s view on testing people involved in testing are hesitant to use the word Test Level since it seems to imply that various groups, based on various hierarchical responsibilities, will perform various testing tasks without interaction between these test levels.

Also many testers have often struggled to distinguish between Test Levels and Test Types. And a Test Stage, is that identical to a Test Level or not?

What should be our focus when organizing testing?

All testing activities together must cover all important areas and aspects of the system under test, that’s the main objective.

To cope with the confusion around how to distinguish testing tasks we introduce the term Test Variety. 

How to define your test varieties?

So your team needs to define the test varieties that suit your needs. How will you do that? And how will you call your test varieties? 
We start with a disclaimer: everything in the remainder of this section is meant as an example. Your situation may be totally different, other terms may be used and other activities may be grouped.

Spheres of testing

Some testing activities can be grouped together, others will need to be organized separately. The larger the scale of your testing, the more likely the need for separate testing. The figure shows the spheres of testing.

Spheres of testing

This gives an idea of the difference in scope of the test goals. And keep in mind: the broader the scope the more complex the tests will be. When testing an individual unit of code, the scope of the tests will be very limited. It may be testing that a numeric value can be put in a numeric field and an alphabetic value cannot be put in a numeric field. On the other hand, when testing an end-to-end business process, the scope will be very broad. For example, test whether an order can be placed and paid in an online-shopping system.

The scope chosen, however, says nothing about the number of test cases, as we will see in the following section. Also, please keep in mind that the spheres of testing only relate to the scope of what is tested. Within different scopes there still may be a wide variety of goals of testing and reasons for testing. In this chapter you will find more about what varieties of testing you might define for your situation.

Test strategy

In the test strategy the team describes how the test efforts are distributed over the test varieties. In a small team the test strategy may consist of some agreements only that were discussed among the team members and not necessarily documented. 
In a large and complex end-to-end process the test strategy may have to be documented extensively. Still, keep in mind that the test varieties are just meant to explicitly define what kind of testing is organized in what way.

Subjects to consider:

Testing pyramid levels

Next, look at the levels in the testing pyramid for deciding on what kind of testing to automate. If you have a few small modules, you will define automated unit tests that cover the functionality. And you will create some automated API tests to test the integration of the modules. Finally, you will prepare some exploratory testing charters so that users can validate whether the IT system delivers the pursued value. 
In the case of the entire customer journey, you must make sure that all individual parts of the systems involved, have been properly tested. Often, terms such as system test and acceptance test will be used. The system test and system integration test will be automated on an API level. The acceptance test is performed from a user perspective on a GUI level. And some exploratory testing charters are prepared for various stakeholders to explore whether the system will live up to the expectations of quality and pursued business value. 

Testing quadrants

Finally, see if the testing quadrants add extra variety to your testing. When testing a few small modules, you will have covered most of the testing quadrants already. Maybe you will need to add some security testing to make sure that the IT system complies with security regulations. 
In the case of the entire customer journey, you may need to check the ISO25010 quality characteristics for non-functional test varieties needed, such as performance and maintainability testing. In an extensive situation such as this customer journey, normally the various DevOps teams involved, together with support teams, will organize and perform a number of test varieties that together cover all spheres of testing, all levels of the testing pyramid and all testing quadrants.