Code coverage in many different intensities

Example of pseudo code with coverage  

Example - The magic boatride

The example case is about an amusement park, called "TestLand". There are many different attractions. One of them is the "magic boat ride". For a group of people to be allowed on this magic boat ride, there are a few rules that the group must comply with:  

Everyone in the group must be at least 120 cm tall and the total weight of the people in the group must be at least 100 kg and no more than 300 kg.  

In our magic boat ride example part of the program code that determines the length of the shortest person in the group, could be as shown in below figure in so-called pseudo-code. 

code

How many test cases do we need to achieve full coverage of this code? Would you believe that just one test case could do that? And if so, would you call that "thoroughly tested" (which is the term the Scrum guide uses) [Scrum 2017]?  

Let's take a look at code coverage and the different possible coverage levels in below figure.

Overview of code coverage types and preference based on intensity.

Code coverage types 

Code coverage can be measured by specific tools during the execution of tests (in other words during dynamic testing). It could also be calculated by walking through the test cases and the code manually, but this is very labor-intensive and therefore seldomly done.  

Many people suggest that achieving 100% code coverage is the desired goal. But few people specify the coverage type for which they want to achieve 100%.  

We distinguish the next coverage types (and the coverage group they belong to) that are relevant for code coverage: 

  • Line coverage (process-oriented)
  • Statement coverage (process-oriented)
  • Decision coverage (condition-oriented)
  • Branch coverage (condition-oriented)
  • Path coverage (process-oriented)

Other coverage types will go into detail of the conditions that make up the decision point, for example condition decision coverage (CDC), modified condition decision coverage (MCDC) and multiple condition coverage (MCC).

Let's briefly define and try each coverage type from the bullet list for the example in pseudo-code.

Line coverage 

100% line coverage is achieved if every line of code has been exercised by a test case. In the example we use the following values for test data:   

nop = 1  
lop = 250  

We achieve 100% line coverage. Because at least something in every line is exercised, even though for the next line of code:

IF lop < lsp THEN MOVE lop TO lsp ENDIF 

Only the "IF lop < lsp" is exercised. The statement "MOVE lop TO lsp" is not exercised, but still the line counts as covered.

Statement coverage 

100% statement coverage is achieved if every statement in the code has been exercised. The previous example with line coverage did not have 100% statement coverage. How many test cases do we need to achieve that? Well, it could still be done with one test case. In the example  we use the following values for test data:  

nop = 1  
lop = 110  

We achieve 100% statement coverage because all statements have been executed at least once. But the decision in the IF statement has two possible outcomes (true and false). And only one (true) was executed.  

Decision coverage and branch coverage 

100% Decision coverage ensures that all possible outcomes of the decision are tested at least once. In the pseudo code example , we use the following values for test data:  

nop = 2  
lop = 110 (test case 1)  
lop = 150 (test case 2)    

We achieve 100% decision coverage, with two test cases (a person of 110 and a person of 150 centimeters). In the first test case lop < lsp, in the second test case lop > lsp, so both outcomes of the decision are tested.  

Branch coverage is a variant of decision coverage. Branch coverage relates to the way the control is transferred internally, this is mainly observed in low-level programming languages such as assembler that are not often used, we advise therefore to just use decision coverage and ignore branch coverage. Also since branch coverage and decision coverage only have different outcomes when there is no 100% coverage.  

Path coverage 

In the building block about path coverage, we describe how to guarantee that every path or path combination in a flow is tested. This gives the same coverage as decision coverage. That is because decision coverage tests every outcome, and every outcome is a path. However, when 100% path coverage with test depth level-2 is achieved, this often means that much more than 100% decision coverage has been reached because the decision is tested multiple times.  

To choose between path coverage and decision coverage the team members will think about whether they just want to demonstrate that every outcome is possible, in which case they would choose decision coverage, or they want to demonstrate that combinations of business logic are covered, in which case the would choose the algorithm test technique using path coverage with TDL-2.  

Other coverage measures 

If we look at more complex decisions that consist of multiple conditions, even 100% decision coverage may not be enough to establish confidence that the IT system will be able to deliver the pursued value. If more confidence is needed then coverage types such as condition decision coverage (CDC), modified condition decision coverage (MCDC) and multiple condition coverage (MCC) are available.

More about these coverage types can be found in the section "Coverage-based testing".

Final important remark: expected results!  

Many tools that count code coverage will just count if a line of code was invoked (in case we do line coverage), but the tools seldomly check if the test case was complete with a check if the actual result was equal to the expected result. Without such a check it is almost useless to invoke the code, because the only thing that is tested is if the program doesn't crash. Only a check of actual with expected outcome can be used to decide if the test actually passed.  

So be aware that people may claim 100% code coverage but still didn't do any real test.