Maintainability testing

Teams are often focused on the cost-effective development of IT systems. In the total cost of ownership, the maintenance costs are often much higher than the development costs. In DevOps the team should find a proper balance between development and maintenance costs. Quick-and-dirty development (which causes a lot of technical debt) may result in higher total costs than well-designed and well-engineered systems. Therefore, a focus on maintainability is essential. A good architecture and structure greatly benefit the maintainability. Maintainability testing is a test variety based on the ISO25010 quality characteristic "maintainability" within to so called non-functionals. 

[More info: functional and non-functional testing]

Definition Maintainability is the degree of effectiveness and efficiency with which a product or system can be modified by the intended maintainers.

What is maintainability?

Maintainability is about modifying an IT system in easy and effective way. If more effort is spent on maintaining existing code than on writing new code this is a clue that maintainability may be poor. Modifications may include corrections, improvements or adaptation of the software. But it also relates to changes in environment, and in requirements, specifications and user stories. It also refers to the installation of updates and upgrades. Indicators for maintainability are the code quality, the documentation quality, the quality of tests and many others.

This test variety should not be confused with maintenance testing. In maintenance testing, the reason for testing is that the system was changed. In maintainability testing, the reason for testing is to see how well the system can be updated, changed and maintained.

The ISO25010 standard defines five subcharacteristics for maintainability:

• Modularity – the degree to which a system or computer program is composed of discrete components, so that a change to one component has minimal impact on other components.
• Reusability – the degree to which an asset can be used in more than one system, or in building other assets.
• Analyzability – the degree of effectiveness and efficiency with which it is possible to assess the impact on a product or system of an intended change to one or more of its parts, or to diagnose a product for deficiencies or causes of failures, or to identify parts to be modified.
• Modifiability – the degree to which a product or system can be effectively and efficiently modified without introducing faults or degrading existing product quality.
• Testability – the degree of effectiveness and efficiency with which test criteria can be established for a system, product or component and tests can be performed to determine whether those criteria have been met.

These five subcharacteristics cover quite different perspectives of maintainability, which also may be of interest to very different persons in (or even beyond) the team.

The ISO25010 contains a note that modifiability is a combination of changeability and stability. These two terms are not mentioned as subcharacteristics in the ISO25010 standard but are often used in practice. Changeability is the ease or difficulty to make adaptations to the system. Stability is the ease or difficulty to keep the system in a consistent state during modification.

We will discuss testability separate from the other four subcharacteristics because these four are primarily a development focus, whereas testability has a testing perspective.

Measuring maintainability both statically and dynamically

The quality characteristic maintainability and its subcharacteristics are determined by measuring a set of software product properties. These properties are volume, duplication, unit complexity, unit size, unit interfacing, module coupling, component balance and component independence [SIG/TÜViT 2018].

The maintainability characteristics of an IT system will partially be measured by observing the product (e.g. code-reviews), other characteristics can only be measured by observing people in operations while they perform maintenance activities. The first is pure static testing. The second can be called dynamic testing, but keep in mind that in this case the IT system itself is not the test object, instead the maintenance activities performed on the IT system are the test object.

Software artifacts that can be considered to be evaluated are for example source code files, data definition files, data manipulation files and active content of web pages. These could also be installation procedures or infrastructure-as-code.

Maintainability tests can be done by, or supported by, automated tools.

Static measurements

Examples of static measurements are:

• Volume of the code, for example measured in lines-of-code
• Cyclomatic complexity of a unit and/or for the system as a whole
• Comment to code ratio
• Nesting depth of decision statements
• Coupling – number of data items passed between modules

Dynamic measurements

The dynamic measuring of maintainability mainly consists of timing the duration of specific maintenance tasks.

Examples of dynamic measurements are:

• Elapsed time for implementing a change
• Stability of the system, for example mean time between failures

Testability

Testability both concerns individual parts of a system, as well as the IT system as a whole and even the end-to-end business process.

Definition Testability is the degree of effectiveness and efficiency with which test criteria can be established for a system, product or component and tests can be performed to determine whether those criteria have been met.

The quality of the system documentation, for example in architecture descriptions but also in epics and user-stories, greatly influences the testability of the system. Also important is the ease with which interim results of the system can be made visible, assessed and even manipulated, for example by having tools that can access the database.

The possibilities and capabilities of the test environment are also important for the testability of the system, think of possibilities for time travel to test extensive business processes such as expiry of insurance or mortgage terms.

A dynamic measure of testability is the "re-test efficiency", which is the ease of testing after a problem has been fixed.

To achieve good testability the team may need to add extra functionality that supports ease of testing. Think, for example, of adding facilities for time-travelling in a system that has a process with a very long duration (such as a system for life insurances). Also, the team may need special tools and facilities, for example to check the contents of a database by directly looking into the database tables.