Data-oriented testing

When we take a data item as a starting point for test design, there are two important terms to consider: the classes (also called partitions) and the boundaries between these classes.  

In our magic boat ride case, we consider the total weight of a group. The total weight must be between 100 and 300 kilograms. This means we have three input classes: less than 100, between 100 and 300 and above 300 kilograms. The output classes are "group is admitted" and "group is not admitted".  

Discrete values versus continuous value range 

In the example above, the values for the weight are a continuous value range. The input value may be any weight. The values for the output are discrete, the group is either admitted or not admitted. The difference between discrete values and continuous value range is that with a continuous range there are boundaries between the classes. When we are testing, we will choose an input value and determine if the result is according to the specification for the class of that value. 

Equivalence partitioning 

A simple technique for coverage of data items is called equivalence partitioning (EP).

Example of data item (weight) with continuous value range

This technique determines all relevant classes of values for a data item and creates one (and only one) test case per class. The reason for testing just one value per class is that we assume that if one value in the class has the right result, all other values in that class will also have that result. Therefore, this technique is used when there is a low risk, or as an additional technique.  

In this technique the classes are the test situations. For every test situation one test case is specified. In the example above, there are at least three classes: less than 100, between 100 and 300, and above 300.  

The person doing the test design should wonder if these three classes are sufficient. Maybe other classes are also relevant. Ask yourself for example: what happens if the value is below 0? Or what is the maximum value for weight? And what if a non-numeric value is entered? These are probably invalid classes for which an error message will be the expected result. This way, we can determine six test situations and also six test cases.

Example of EP test situations and test cases.
Number Test situation = class Test case = input value Expected result
1 0 kg through 100 kg 50 kg Not admitted
2 100 kg through 300 kg 200 kg Admitted
3 300 kg through 999 kg 500 kg Not admitted
4 -1 kg and less -200 kg Error message
5 999 kg and more 2000 kg Error message
6 Not numeric A kg Error message

Note that in test case four we choose -200 kg. It could have been any negative value but by choosing -200 we might get the result "admitted" if the minus sign would not be considered by the system. 

Boundary values

You have probably noticed that we have not specified the classes in a very exact manner. In the specification above it is not clear if 100 kg is admitted or not.  

When choosing values for test cases with equivalence partitioning, it is common to choose a value that is not close to the boundaries. In practice, most faults in systems will be related to the boundary values. It is therefore very important to describe the boundary values in a very precise manner and to test these boundaries as well.

Example of boundaries and boundary values

This technique we call boundary value analysis (BVA). When applied to the example, this would result in the improved six test situations.

Example of BVA test situations and test cases.
Number Test situation = class Expected result
1 0 kg <= x < 100 kg Not admitted
2 100 kg <= x <= 300 kg Admitted
3 300 kg < x <= 999 kg Not admitted
4 x < 1 kg Error message
5 999 kg < x Error message
6 Not numeric Error message

Boundary value analysis 

To most people that start doing test design, boundary value analysis is an intuitive test design technique. When asked what to test if a group of 100 kg and more is admitted, most people will answer that they will test a group of 99, of 100 and of 101 kg. We call this three-value boundary value analysis. Another way of looking at it is just choosing one value for every class. In this example we would choose 99 (not admitted) and 100 (admitted). This we call two-value boundary value analysis.

When to use three-value BVA (or even four-value BVA)?

When testing a single boundary, it is very natural and easy to apply three-value BVA. This means you will make a test case for the boundary value itself, and for the values just below and just above the boundary value.  

In our example, the test cases would be 99, 100 and 101. Please note that test case 99 results in not admitted, 100 and 101 result in admitted. If there is a high risk, four-value BVA would be an option. This would mean adding an extra test case for the output class that was just covered once. In this example the extra test case would therefore be 98 [Tuinhout 2008]. 

When to use two-value BVA?

If you are testing in a situation with a continuous value range with multiple boundaries, such as in our example with boundaries 100 and 300 kg, using two-value BVA will be sufficient.  

Two boundaries with four boundary values

We would have the test cases with input values 99, 100, 300 and 301. We would not need three-value BVA because test case 300 will give the same confidence related to boundary 100 as would test case 101. Both will discover a fault in case the programmer made an error with the > and = symbols. (Note that to keep this explanation simple, we did not go into the details of the invalid classes such as below 0.) 

Combining EP and BVA

When the risk level is high, the team members involved in testing may choose to combine the test design techniques. They will then test the boundary values but also the values in the middle of the class, which contributes to a higher level of confidence. This way, we will determine six test situations and ten test cases.

Example of combined EP/BVA  test situations and test cases.
Number Test situation = class Test case = input value Expected result
1 0 kg through 100 kg

50 kg
99 kg

Not admitted
2 100 kg through 300 kg

100 kg
200 kg
300 kg

3 300 kg through 999 kg

301 kg
500 kg

Not admitted
4 -1 kg and less -200 kg Error message
5 999 kg and more 2000 kg Error message
6 Not numeric A kg Error message

Additional considerations 

In our example for equivalence partitioning, we used the input value to determine test cases. We could also have used output value (admitted or not admitted) to determine the test cases. Only two test cases, for example 50 kg and 200 kg, would then give 100% coverage. It is up to the tester to consider if, in this example, two test cases would be sufficient to provide confidence about this feature.  

In the example of boundary value analysis, we have used 1 kg as the minimum difference. Of course, there may be a situation where the weight is determined in grams, in which case the boundary value would not be 99 kg but 99,999 kg.  

Data combination test

Most IT systems process various data items and the values of these data items influence the results of processing. When we want to test combinations of values of data items, the data combination test (DCoT) is a very useful technique.
For an in-depth explanation of the DCoT continue here.

Syntactic testing

The syntactic test, together with the semantic test, belongs to the validation tests with which the validity of input data (e.g. through graphical user interfaces) is tested. This establishes the degree to which the system is proof against invalid, or "nonsense" input that is offered to the system.
For an in-depth explanation of the Syntactic test (SYN) continue here.