Progress report

Reporting takes place in accordance with the reporting structure described in the test plan. The progress report contains data on the most recent reporting period and cumulative data on the entire test process. 

Besides figures, the report should also provide textual explanation and advice on the results, progress, risks and any problem areas. The latter is inclined to be forgotten in reports that are generated from test-management tools. It should be realised that explanation and advice are very important in the provision of quick and reliable insight into the figures. It is the most important product of testing. While the explanation can and should be given verbally, it most definitely should be contained in the written report. This forces the test manager to think carefully, as well as making the advice stronger, reaching a wider audience and helping with the process evaluation in retrospect. 

In outline, a progress report has the following content (based on the BDTM method with the four aspects of Result, Risks, Time and Costs). In practice, the list of contents may follow a different sequence; subjects may be combined, or even omitted. It depends on the report's target group. 

These subjects are further explained below.

Status of the test object (Result) - 1

Status per characteristic/object part - 1.1

It is shown per characteristic/object part:

• The status of the tests (not started, planned, specification, execution, retest X, completed), optionally with the progress percentage, e.g. the progress of the execution is estimated at 60% 
• Overview of numbers of defects (sorted by status and severity, optionally also by other aspects, such as cause)
• If test products (such as test cases or test scripts) are seen emphatically as results, they can also be included in the overview, with an indication of whether a start has been made on the product and whether it is ready. 

The closer the end of the test period approaches, the more attention is paid in the progress report to the consequences of open defects. In the beginning, it is less useful to include this in the report, since it is expected that the defects will be solved. But the consequences should always be included in the defect report itself. 

• Defects that remain open and their impact
• Defects not solved (known errors), and their impact.

Status of test goals - 1.2

Based on the above, the status per test goal (user requirement, business process, critical success factor, etc.) is reported. Sometimes a test goal can be directly linked to a number of characteristics/object parts and to the test status related to them; sometimes the status per characteristic/object part is not sufficiently usable and the test manager still has to determine the test status per test goal. The risk tables from the Product Risk Analysis make the link possible. 

Trends and recommendations  - 1.3

Relevant trends and related recommendations can be reported here.

Product risk and strategy adjustment (Risks)

In this part of the report, the stakeholders are given insight into the degree to which the coverage of the various product risks has changed, as well as into any process risks. 

In the test plan strategy, it is determined whether and to what degree product risks will be covered by testing. During the test process, aberrations may occur: the estimate of the risk appears different and/or the test coverage requires adjusting. The adjustments over the reporting period, with associated consequences, are reported in this part. In this, the translation is made into the test goals: what kind of impact will the changed risks have on the attainment of these goals? 

Progress of the test process (Time and Costs) -  3

Regarding the progress of the test process, the points below are significant.

Progress over the latest period  - 3.1

At the level of phases and/or main products, the following could be reported:

• Number of planned hours
• Number of hours spent so far
• Number of hours expected still to be spent
• Percentage completed
• Dates: planned/expected/actual start date; planned/expected/actual end date.

Products could be the test plan, test scripts, test-execution files and reports.

If the test manager is responsible for the budget, he will also include in the progress report information on completing the test process within budget. 

Activities in the coming period  - 3.2

Here, the activities to be carried out in the coming period are reported.

Hours lost  - 3.3

This refers to non-productive hours of the testers. If the test process environment does not meet certain preconditions, this will result in inefficiency and loss of hours. Examples are a non-functioning test infrastructure, much or lengthy test-obstructing defects or lack of support. Hours lost, and the causes, are reported here. 

Trends and recommendations - 3.4

As with trends in the status of the test object, trends and recommendations in connection with the progress of the testing should also be reported. The central question here is whether the agreed milestones are (or appear to be) feasible.

Problem areas/discussion points (all the BDTM aspects) - 4

In this section of the report, the test manager points out any problem areas or points for discussion that jeopardize completion of the test assignment within the set limits of time and costs. For example: 

• The test object being delivered later than planned
• The quality of the test basis being less than expected
• The test environment not being available on the agreed date
• Test-obstructing defects present in the test environment or test object.

Besides the various problem areas, their consequences and possible measures are shown. Here, too, the test manager makes the translation into the test goals. 

Agreements - 5

This part shows the agreements made in the current period between the test team and other parties that are relevant to the recipients of the report. 

Quality of the test process (optional) - 6

If required, this part of the report can include information on the quality of the test process. The following questions play a part here: 

• Are the significant defects being found (as early as possible)? (Effectiveness)
• How economical is the test process with time and resources? (Efficiency)
• Is the test process working as agreed? (Verifiability)

The three quality aspects of the test process.

Effectiveness - 6.1

Efficiency - 6.2

Verifiability  - 6.3

Below is an example of a dashboard, enabling the most important information to be seen at a glance.

Later in the report, these points are worked out in detail in overviews with notes. Examples of overviews (without notes):

Quality of test object – defects

Quality of test object – subsystem x causes

Progress