In information technology, we come across processes at various levels. Business users will have an end-to-end business process that consists of various steps and decisions to influence the flow of the process. Systems developers will define the flow of a program in their code by writing statements, of which the decision statements will influence the flow of the program. For testing all these kinds of flows, an important coverage measure is path coverage. It can be applied in different test design techniques, especially process cycle test (for testing business processes in e.g. an acceptance test) and algorithm test (for testing code structure in e.g. a unit test).
In this figure the core elements of a process or program flow are shown: these are the decision points, the start and end point(s) and the paths between them. The aim of path coverage is to cover the variations in the flow that are possible according to the chart. The test cases are described to indicate which paths in the flow should be followed consecutively, from start to end. The test cases together will achieve the desired coverage.
Keep in mind that such flowcharts with decision points and paths do not necessarily have to be about the functionality of the system. Security processes or work procedures in business processes can also be described with such charts, which makes the basic technique described here applicable for many different test varieties. And note that this type of coverage does not consider the conditions that are in the decisions, it only considers the possible resulting paths. When covering conditions is essential, apply a technique from the coverage group condition-oriented testing.
One coverage type, two test design techniques
If path coverage is applied on business process level, we call the test design technique process cycle test. When applying the same path coverage to program code we call the technique algorithm test. The way of working for both techniques is exactly the same.
Steps to create test situations and test cases using path coverage
Using path coverage, there is a three-step way of working: first determine the paths, then derive the test situations and after that combine test situations into test cases:
- A test situation is a path or a combination of paths.
- A test case is a combination of test situations which begins at (one of the) start point(s) of the flowchart and ends at (one of the) end point(s) of the flowchart.
Step 1: Determine the paths
Simplify the chart by removing everything else than decision points, start, end and lines. Determine the paths: a path is a line between one decision point and another decision point, or a line from the start to the first decision point, or a line from the last decision point to the end. Number the paths and letter the decision points.
Step 2: Determine the test situations using test depth level
In the coverage of paths, various levels are possible. These levels are called "Test Depth Level" (abbreviated to TDL). The first level of coverage, test depth level-1 (TDL-1), provides the guarantee that each path has been travelled at least once. Therefore, for TDL-1 every path will occur in at least one test case, thus ensuring each path is tested. In our example this means there are three test situations: path 1, path 2 and path 3.
Higher levels of coverage create test situations from combinations of paths. Test depth level-2 (TDL-2) means that for each decision point every combination of each incoming path with each outgoing path is a test situation. To easily create the test situations for TDL-2 we create a table with columns for the decision points, the incoming paths, the outgoing paths and the test situations, which are the combinations of incoming and outgoing paths.
|Decision points||IN||OUT||Test situations|
1-2 1-3 2-2 2-3
Higher levels are also possible; for test depth level 3 for each combination of two decision points, every combination of three consecutive paths is a test situation. In our example, there is only one decision point; so TDL-3 wouldn't make sense. In practice, most teams will find that TDL-2 will give them enough confidence: TDL-3 and higher are seldomly used. Basically, all following steps are the same for every test depth level. We will therefore only explain TDL-1 and TDL-2 in detail.
Test depth level-N is a coverage type that aims to demonstrate the certainty that all combinations of N consecutive paths in a process or program flow are covered. A path in this context consists of all steps between a decision point and the next decision point, or between the start and the first decision point, or between the last decision point and the end.
Step 3: Combine test situations into test cases
Every test case starts at the start of the flow and ends at the end of the flow. To be able to always go from start to end, some test situations may have to be used in multiple test cases.
In our example, TDL-1 coverage can be achieved with one test case that executes paths 1, 2 and 3. Which is:
Test case 1: paths 1, 2, 3.
In our example, TDL-2 coverage can be achieved with two test cases. The first test case combines path combinations 1-2, 2-2 and 2-3. The second test case consists of path combination 1-3. Then all path combinations have been used so that all test situations are covered.
The test cases are:
Test case 1: 1-2-2-3
Test case 2: 1-3
Extra: achieve more confidence
Flows often contain loops. Just like in our example, a loop makes that a path returns to an earlier point in the flow so that combinations with the same path can occur. The first test case goes through the loop twice (using the path combinations 2-2 and 2-3). The second test case does not go through the loop at all. To achieve more confidence our rule of thumb is: when there's a loop, create test cases that go through it:
zero times (= not through the loop)
one time through the loop
multiple times (= two or more times through the loop)
In our example, an extra test case for TDL-2 should therefore be added that goes through the loop just once:
Test case 3: 1-2-3
It is a coincidence, in this simple example, that this is the same test case as for TDL-1.
Which is better: a few long, or many short test cases?
When applying path coverage on large flows with many decision points and (especially) many loops, a high number of test situations will have to be created.
The tester may wonder; which is better, combine as many test situations as possible in one test case, reaching the test depth level with few test cases, or combine as few test situations as possible in one test case, creating a great number of test cases to finally achieve full coverage. The answer (as often) is "it depends".
Our first advice is to keep the above-mentioned rule of thumb in mind; create some shorter and some longer test cases so that each loop is tested zero, one and multiple times. If there still is freedom to choose either a few long or many short test cases, the test variety at hand will usually help decide.
When members of a DevOps team are testing code using a program flow, they probably want many short test cases, so that when a test case fails, it will be fairly easy to determine the cause of the failure.
When business people test an end-to-end business process (for example in an acceptance test), they would probably like to create a few long test cases. In their situation they usually don't expect many failures because multiple quality measures have been taken by other people previously (for example in reviewing, unit testing and other quality engineering activities) so they mainly want to assure that the business process indeed works by exercising all paths or all path combinations.
A similar reasoning goes for progression versus regression testing. With progression testing, the new code and/or changes may have caused faults. Many short test cases will help test the changes and if faults exist it will be easy to find the cause. In the case of regression testing, the goal is to ensure that parts of the system that were not affected by a change, are indeed still working as expected. This can be done with a few long test cases.
More information about: Coverage types