Coverage types

Coverage, coverage type and coverage ratio

There are a lot of definitions relating to coverage. The definition below is a simple one, but it reflects the essence perfectly well:

Definition Coverage is the ratio between that which can be tested and that which is tested with the test set.

In other words, coverage says something about how many of 'all possibilities that can be tested' are actually tested. This means that there are 2 phenomena that together determine the coverage (see 14.4):

• Which possibilities are distinguished?
  This is termed coverage type. It indicates what type of possibilities are involved. These are subsequently elaborated into the required test situations. For instance, it is possible to look at the possible combinations of paths in an algorithm. But in that same algorithm, you can also look at the possibilities to cover one outcome or the other within each decision point. These are 2 different forms of coverage (i.e. 'coverage types').

Definition Coverage type[Register: coverage type] is the form in which the covering of test situations deducible from the test basis, is expressed.

• How many of these possibilities are tested?
• This is called the coverage ratio and is generally expressed as a percentage. It shows which part of all of the alternatives have actually been tested. The number (percentage) does not have meaning until it is clear which possibilities are involved, i.e. when it is associated with the coverage type.

Definition Coverage ratio is the percentage of test situations, as defined by the coverage type, that is covered by the test.

 

Figure 14.4: Components of the concept of 'coverage'.

This chapter discusses the how and why of coverage, coverage type and coverage ratio in detail. It first discusses why coverage is a useful and practical concept for testers. It then explains, using a comprehensive example, why it is necessary to distinguish several types of coverage. Finally, it describes how test design techniques constitute the link between test cases and the coverage they achieve.

Why coverage?

'Coverage' is a useful concept that helps testers with complex questions, such as:

• Since it is impossible to test everything and we are therefore forced to test only a subset of all of the possibilities, what is the best subset?
• What is the difference between 'elementary testing' and 'thorough testing'? What does it mean concretely for the test cases we need for this?

Coverage has everything to do with the wish to find the most possible defects with the fewest possible test cases. Instead of simply testing 'just any' subset of options, we aim to compile a set of test cases that create the greatest possible change of finding the defects that are there.

We can never be certain that all defects have been found or even that 60% of all defects has been found. After all, we do not know how many defects there actually are. What we can demonstrate, is the coverage realised by the test. And this confers a certain level of confidence that the chance of any defects remaining in the tested system is small. In summary: the higher the coverage realised, the smaller the chance that any unknown defects remain in the system.

The decision to 'test more thoroughly' concretely translates to a decision to achieve more extensive coverage. In principle, there are 3 options in this respect:

• A more thorough coverage type
• Multiple coverage types
• A higher coverage ratio of a specific coverage type.

One type of coverage or more?

It should be clear by now that a statement like 'I want to test with a 75% coverage ratio' does not say much, for the following reasons:

• What type of coverage are we talking about?
• Why does 25% (and which) not require coverage? (In other words: Why not simply cover 100%?)

This section will discuss these issues in greater detail and explain

• that we cannot talk about the coverage, but talk of many forms of coverage (i.e. 'coverage types')
• that the choice for more thorough testing must be worked out primarily with the choice for another coverage type
• that the use of a coverage type implies that one aims for a coverage ratio of 100%
• that generally speaking, the various coverage types CANNOT be compared in terms of 'X is better than Y'. Different coverage types tend to supplement rather than replace each other.

To explain this, we will use the following example about a system to order books via the Internet. It describes only a fraction of what such a system could do in reality, but it will suffice for our purposes.

Example The chart below describes the process to order books in our example system. We have used an intuitive diagram technique with decision points (A, B, and C) and paths (1 through 7). In reality a lot is happening on the paths of the process, but this is not relevant to the example. It is mainly about the fact that the process includes different possibilities and the question which different aspects should or can be tested. The ordering process looks like this. The numbers and letters in parentheses refer to the decision points and paths in the diagram: After the general details of the ordering party are entered (1), it is decided (A) whether to continue with an existing order (2) or initiate a new order (3). An order may contain more than one book. After the required books are ordered, it is decided (B) whether to continue to payment (4) or change the order by cancelling ordered books (5), after which the order can continue. When determining the final price, it is checked if the customer is a 'good customer' (C) and is entitled to extra discount (6) or not (7).

 

 

Figure 14.5: Chart of the process 'ordering books'.

Discussion on test cases and coverage ratio

Say that the system component described above requires thorough testing. What does that mean exactly? Which test cases are required?

A test case runs through the process to be tested from start to end, covering various decision points and going through different paths. For a thorough test, we must look for a set of test cases that responds to the objective in the best possible way: 'If there is a defect in the process, then this set of test cases will find it.'

Now look at the following set of 2 test cases, which are described by a series of numbers that shows which paths are followed consecutively:

TC-1 = 1 2 5 3 4 6

TC-2 = 1 3 4 7

Thanks to the loop in the chart, test case TC-1 covers virtually all of the paths. Only path 7 is not covered. This is why TC-2 has been added.

So do these 2 test cases achieve a coverage ratio of 100%? If a coverage tool were used to measure the coverage, it would indeed report that 100% coverage is achieved. Still, many testers would say that these 2 test cases test far from everything... that many defects may exist in the system that would slip through these test cases. Are they right? Absolutely! For instance, look at this:
Say that there is a defect in the system in path 5. When an ordered book is cancelled, a related discount must be cancelled as well – but the system incorrectly retains the discount.

Is this defect found with the set of 2 test cases described above? No! While test case TC-1 does cover path 5 and will therefore receive an excessive discount, the latter then disappears when the test case continues through path 3 and starts a new order from scratch with discount=0. This defect would be found with a test case that continues with path 2 after completing path 5. In this case, the same order would be continued and the excessive discount would be visible on payment.

In other words, this is a defect type that does not 'occur in a path' randomly, but occurs only in case of a special combination of 2 successive paths. So the 2 test cases TC-1 and TC-2 do have a coverage ratio of 100% in the sense of 'covering all paths', but they do not cover 'all combinations of 2 successive paths'. For a set of test cases that does cover this, we speak of achieving 'test depth level 2'.

Test depth level 2 = the certainty that all combinations of 2 successive paths are covered.

For '100% of test depth level 2', path combinations 2-4 and 3-5 must also be covered. This can be achieved by adding the following test case:

TC-3 = 1 3 5 2 4 7

(Please note that this eliminates the need for test case TC-2.)

It is a very common phenomenon that paths affect each other, so that defects do not occur until paths are combined in certain ways. This influence may be functional (such as in the above example of excessive discount), but it may also be technical. An example of the latter is:
When cancelling an earlier discount in path 5, a buffer overflow occurs, overwriting the memory location of the variable 'extra discount'. When a test case going through path 5 then runs through path 7 (where no extra discount is given), this defect will not be noticeable. However, if the test case would continue with path 6, a bizarre extra discount would be allocated or the application might even crash.

Naturally there may also be defects that do not occur unless a specific combination of 3 successive paths is followed. This defect type cannot be found for certain unless the set of test cases achieves test depth level 3. And so on... The higher the test level achieved, the greater the certainty that the more exceptional defects are found.

The concept of 'test depth level' and its application is explained in greater detail in the section on 'Path coverage'.

So is this it: The higher the test depth level, the more thorough the test? The maximum test depth level is 'infinite', but this is clearly not practically feasible. But could one (theoretically) conclude that with test depth level 'infinite' one will find all defects? No! Not even that! The example below shows a defect that cannot be found with certainty even with a test set of test depth level 'infinite'. And the defect described is not even farfetched:
A 'good customer' is given an extra discount. This is established in decision point C, where the system checks whether the customer has ordered a lot of books or for a high amount. More precisely:
IF (number of books > 8 OR amount ≥ €250) THEN extra discount

There are therefore 4 possible situations in which extra discount will or will not be given. These are shown below.

number of books	>	8 ; amount	≥	250 →	YES extra discount	(1)
	>		<		YES	(2)
	≤		≥		YES	(3)
	≤		<		NO	(4)

Say there is a defect in the system, so that you do not get the extra discount in situation (3). So, if you order few books but they are expensive, you will not get the discount – incorrectly.

The problem described above (situation 3) occurs in path 6, where you ARE supposed to get the extra discount. But there are 2 other situations also associated with path 6. Our test set of test depth level 'infinite' may have passed path 6 many times, but that does not guarantee that at least 1 test case has actually tried situation 3. It might accidentally be true, but we cannot be certain that situation 3 will be covered. Why not? Because the test set of test depth level 'infinite' is not designed for this purpose. The test set was designed to cover path combinations, but this defect has nothing to do with that. The defect would have been found with certainty if a test set was used with a different coverage type, i.e. a coverage that focuses on how a complex decision point handles the various options. E.g. by covering all possible combinations of all conditions. In the example above, this amounts to all 4 specified situations. Such a coverage type is called 'multiple condition coverage'. Another coverage type is 'decision coverage', which covers only the 2 possible outcomes of the decision point (YES or NO extra discount). This is less thorough coverage than multiple condition coverage, which would not detect the above defect with certainty.

So what is best? Do we choose covering paths or covering decision points? Clearly doing both achieves the highest level of thoroughness. But say that this is not allowed (e.g. because it is too costly). Say that we have to choose, then what would be best? Or more concretely:
Which coverage type is better: 'test depth level 2' or 'multiple condition coverage'?

This question may seem simple, but it isn't. One coverage type could only be called 'better' with any practical use if it would find at least all of the defects found by the other coverage type plus some additional defects. But the example above demonstrated that multiple condition coverage finds a specific defect that is not found with test depth level 2 (i.e. the extra discount for few but expensive books). But the reverse is true as well (i.e. the discount that is given incorrectly in case of cancellation). So the best answer is:
We cannot unequivocally talk about 'better'!
The various coverage types find different defects.

Tip If multiple coverage types can be used from which a choice has to be made, consult with the client. Analyse where the highest risks are located and select the coverage type that matches this area most adequately. Then make sure that the client is aware of the fact that there are risks in the area relating to the coverage type that was not selected.

For instance, the tester and client must typically ask themselves the following for the earlier question whether to choose test depth level 2 or multiple condition coverage: Is it a process in which many different types of actions are executed whose consequences might affect each other? In this case, path coverage test depth level 2 would be a good choice. But in case of a process where a specific discount is given or not after every decision point, path combinations are not that interesting because the discounts do not have a mutual impact. However, the client might consider the possible situations in which the discount is given or not given extremely important. In this case, coverage type multiple condition coverage might be the wiser choice.