One of the things I like the most about Agile is its focus on Value. Someone recently sent me a link to a Google blog on End-to-End testing that stated: “A failing test does not directly benefit the user”. Value is only created when the bug is fixed!
All too often test strategies identify several test types (often depicted as a Test Pyramid) and focus on identifying the system/software components that pose the highest risk and tune the test approach and activities to that end and that’s it. Done! We know how to most effectively find bugs! From then, it’s up to the developers to fix it.
This is a misconception! Just finding bugs does not add any value to the end user. So, should we then leave it out altogether? No, I don’t think so, but we should be more aware of where, when and how testing does provide value.
Testing is instrumental in deploying bug-free (bug-poor solutions. Finding the bugs is merely the first step. Root-cause analysis and bug-fixing are the next steps and without it, just identifying the bug is of little or no value. Fixing the bug can only be done properly when its root-cause is known and that’s where many test strategies fail: they do not take ‘making root-cause analysis easy’ into account. Or better still, set up the test strategy in such a way that root-cause analysis becomes superfluous: a failing test reveals not only a bug, it also inherently reveals the underlying cause!
So let’s take a look at the root-cause analysis. Is it always so difficult? Or is there a pattern to the easiness of root cause analysis? How about this: there are tests that don’t need any additional root-cause analysis. Most (automated) unit tests are so specific that when they fail the cause is clear. And there are tests that are notoriously difficult when it comes to root cause analysis: End-to-End tests. These tests are on opposite sides of the Test Pyramid.
The Test Pyramid as depicted above is a generalization, in practice, we have much more layers of Integration, like the Test Pyramid directly to the right. If a test strategy takes the Integration Strategy of the development lifecycle into account and tests are tuned to the Integration layers, then bugs relate to specific Integration aspects and the bug’s root cause must be related to the Integration of the already tested, underlying parts.
So let’s fully align our test strategy with the development lifecycle and eliminate the need for root cause analysis: that’s when testing really brings value to the table!
Published: 3 May 2018
Author: Ben Visser
Ben Visser (*1960 - +2018)
Ben was a highly experienced test manager and test consultant. In a career of over 15 years as a tester, he fulfilled a wide range of test functions, from programmer of automated scripts, to test manager of a large international program. We thank him for his work with Sogeti and his contribution to TMap.