Product Risk Analysis

Introduction

Product Risk Analysis is the term that was traditionally used for determining the risk that an IT system will not deliver the business value that was expected. Today we use the term Quality Risk Analysis to emphasize that the risk is about the quality of the IT system.

The below explanation of the Product Risk Analysis originates from TMAP NEXT (2006) and TMAP HD (2014). For the current status of Quality Risk Analysis please read this Building Block.

Projects should follow an integral testing and quality policy, for quality is a mindset, not a feature. Quality as such ought to be an integral part of project management. Looking at project governance, several aspects, among others, play an important role: costs, risks, time and benefits (also referred to as results or - business - value).

Although all of these aspects are important, it is worth emphasizing the aspect of risk, especially product risk, since this may help as a steering mechanism. It may help to find the balance between ‘building the right thing’, ‘building the thing right’, and ‘building it fast’.

Many projects don’t have unlimited time, money and resources for evaluating the quality of the product. Such constraints in terms of time, money and resources represent constraints on the result to be achieved and therefore often reduced evaluation possibilities of the product risks. As such, it is important to achieve a well considered balance between the investment in money and time on the one hand, and the results to be achieved and the risks covered on the other. The result of the product risk analysis provides the justification for this balance.

Based on the insight resulting from the product risk analysis, high risk products can be evaluated more intensively than those representing a lower risk. Be aware that risks and how to cover these risks are directly related tot the acceptance criteria. These acceptance criteria are available in various forms. As a section in a test plan, in the confirmation part of a story card, in a definition of done, etc.

There are many approaches on how to determine risks. However in general one could say: These approaches involve analyzing the product to be evaluated with the aim of achieving a joint view - for and with all stakeholders - of (the properties of) the product to be evaluated that represent higher and lower risk levels, such that corresponding measures can be assigned to this view.

What measures can be taken in order to cover these analyzed risks is decided when determining the quality strategy. Remember, quality is designed and built in, not tested in! Meaning, from the beginning of the project, everyone should embrace risk and quality thinking. When for instance a requirement is described (e.g. user story, use case) start thinking about possible risks at stake and how to cover these, for instance by executing a Fagan inspection. Or when the system architecture is (being) defined, again think about possible risks and how to cover these, for instance by means of a proof of concept. A last example of risk coverage is assigning specific test design techniques in order to cover identified (product) risks. Read more about risk coverage as part of the quality strategy in building block ‘quality strategy’.

Too often product risks are seen as important for testing. Nothing could be further from the truth. Product risks deal with the risk for the organisation when the product does not have the expected quality. And organizations are still plagued by projects getting out of hand in terms of both budget and time, owing to software defects during the operation of the developed information systems. (Better) testing contributes to the prevention of those kind of defects, but the general idea that the key lies with testing is a persistent misconception. Many analyses of defects show that testers certainly don’t find everything. But even more: it turns out that the cause of defects lies within the development, design and requirements.

First things first. How do you determine the risk level anyway? You’ll find many approaches if you read one or more of the many books written about this topic or just execute a search in the internet. Typically you’ll find approaches like:

  • BDTM Product Risk Analysis
  • PRISMA®
  • PRIMA®
  • Risk Poker in scrum
  • Product Risk and Benefit analysis (PRBA)

Although many approaches exist, the base often proves to be surprisingly similar. Just look at the definition of product risk, which will differ in wording, but not in meaning: A product risk is the chance that the product will fail in relation to the expected damage if it does fail:

Product risk = Chance of failure * Damage

Sometimes ‘chance of failure’ is referred to as ‘likelihood of occurrence’ and ‘damage’ as ‘impact’. It really doesn’t matter because in the end the result will be similar if not the same.

There are two types of risks to distinguish. When the primary effect of the potential problem is on product quality, potential problems are referred to as quality risks or product risks. When the primary effect of the potential problem is on project success, potential problems are referred to as project risks or planning risks.

The above is about identifying risks and risk coverage. Sometimes people look at this in a different way. They try to answer questions like: What is the importance of a particular product? Or what is the urgency of a particular product? However this has nothing to do with analyzing the product risks but with determining the planning, especially the desired order of realization. For the allocation to a specific delivery time, an approach as MoSCoW is often used. This is - of course - important because if a product is realized at the right time, the product delivers the most added value. And if this is not done correctly it may have a huge negative impact on the success of the project (which is an example of a planning/project risk).

  • Trademarks: 
    • PRISMA is a trademark of Improve
    • PRIMA is a trademark of Valori

 

Responsibilities

Product risks have a strong relation with acceptance criteria/ requirements, the chosen architecture and the functional and technical solution. One of the first moments to think about product risks therefore is during the requirements phase or the defining of user stories.

Requirements and user stories are usually prioritised. What does this prioritisation mean? That the requirement or user stories are in a certain degree important for the success of the product or service. If the requirements with the highest priorities are not being met then this has a huge impact on the organisation. To make it easy on ourselves the established priority may be translated one on one into the damage component of a risk. Now only the chance of failure (likelihood) is needed. The Business Analysts could very well have a leading role in recovering the requirement related product risks, as part of the knowledge area Solution Assessment and Validation [A Guide to the Business Analysis Body of Knowledge®].

We already discussed that product risks may act as a steering mechanism for projects. But even when – as often seen – only testers are interested in acquiring product risks, their goals is to test the most relevant parts of the test object and thus providing decision makers with information to make a substantiated decision about going live with the solution. So, they do this in behalf of the project. The project manager is therefore responsible for the execution of a product risk analysis. He may however delegate the actual execution to, for example, a quality and/ or test manager.

 

PRA, testing and coverage

Testers are interested in product risks. There are several ways to determine test effort (exploratory, process/standard compliant, regression-averse, etc.), but risk-based testing is a powerful way and recommended, even as an addition to other approaches.

A disadvantage that people experience with performing a product risk analysis is that product risks are often abstract, while making test cases implies a very concrete and detailed level. Test cases cover very concrete things like boundary values, combinations of data, paths, statements etc. (even if no specific coverage types were applied). These are other aspects than the ones we find with product risks. Many therefore have the feeling that along the way the relation between risks and the test cases gets lost.

That is very serious. Designing test cases is indeed the essential link between the test strategy (and thereby the product risks) and the concrete test cases with which this test strategy is executed. The derived test cases should be a substantiated elaboration of the test strategy: the agreed coverage on the agreed place.

The solution is to ensure traceability when creating a test strategy from product risks, or when executing a product risk – where possible – performing a deepening process. In this deepening process a declaration should be made on risks on the level of coverage types. Because of limited information it is usually not possible to directly link coverage types to requirements. As soon more information becomes available, such as functional specifications, the risks should be elaborated towards coverage types, by means of some intermediate steps.

 

How to perform a PRA?

As mentioned, there are several definitions of product risk. One formula is, according to Wikipedia:

Risk = ((Vulnerability * Threat) / Counter Measure) * Asset Value.

We like to keep it simple however and use a variation of the following generic definition:

Risk = Likelihood * Impact.

 

Translated to testing we apply the following definition for product risk:

Product Risk = Chance of failure * Damage

where

Chance of failure = Chance of defects * Frequency of use

Or in plain words: a product risk is the chance that the product fails in relation to the expected damage if this occurs. The Chance of failure is determined by the Chance of defects and the Frequency of use. The chance of defects is the chance that a product (component) contains a defect. The presence of a defect in the product, however, does not mean that that defect will actually manifest itself in production. When the defect is in a part of the product that is never used, the product will not fail. The more often the product component with the defect is used – or, the higher the frequency of use – the greater the chance that the product will fail.

There is not one right way of performing a PRA, but in general you may want to follow these five steps: 

  • Determining participants
  • Determining the PRA approach
  • Preparing session/interviews
  • Collecting and analysing product risks
  • Completeness check

We have already seen that requirements may be directly linked to product risks. There are more sources of input to identify product risks:

  • Business processes
  • (Sub)systems
  • User stories
  • Uses cases
  • Functional specifications
  • Technical specifications
  • Relevant quality characteristics