Experience-based testing

The test strategy defines how intense (specific parts of) the test object must be tested to be able to establish the required level of confidence that the pursued business value will be achievable. This test intensity is achieved by applying experience-based and/or coverage-based test approaches.

Definition

A test approach is a way of working for designing and executing tests. There are two groups of test approaches: experience-based testing and coverage-based testing.  

In the following sections we will describe the experience-based testing approaches. Please keep in mind that in practice it is always wise to combine both coverage-based testing and experience-based testing. The coverage-based testing approach and the test design techniques are described in the section "Coverag-based testing".

Experience-based testing approaches

Experience, skill and intuition are the basis for experience-based testing. These approaches leave the tester free to design test cases in advance or to create them on the spot during the test execution, mostly testers will do a bit of both.

The experience-based approaches, by definition, don't follow strict rules but instead rely on skill, intuition and experience. People are creative so many approaches to experience-based testing exist. Some are more structured than others. The circumstances dictate which approach(es) is (are) applicable and which can better be avoided. We describe four experience-based approaches.

Error guessing

The four experience-based testing approaches briefly described

Checklists

Bringing relevant experience together in a checklist is common practice in many industries. Checklists can be used separately or as a tool in another approach [Koomen 2006]. For more information see the section "Checklists".

Error guessing

This is a largely intuitive and ad hoc approach to testing that often does not involve any form of documentation or logging; it is entirely unstructured [Myers 1979]. For more information see the section "Error Guessing".

Exploratory testing

Testing based on a charter, using test ideas and keeping a structured log of the tests executed and the results experienced. This approach is the most structured among the experience-based techniques [Hendrickson 2013]. For more information see the section "Exploratory testing".

Crowd testing

Taking advantage of the contribution of a large group of people, in which way many different experiences, insights, skills & intuition can be used [Boersma 2014]. For more information see the section "Crowd testing".

Flow of the experience-based testing approaches

Some of these approaches can be combined. Using checklists in crowd testing for example is a good way to guide people in a very scattered group of testers.

The basic flow for all experience-based approaches is shown in below figure.

 

The iterative flow of activities

Experience-based testing is commonly performed in test sessions. There is some preparation for each session (whether or not documented), and during the test session the involved people iteratively design tests, execute them, evaluate the result and learn about the system. Based on this, they design new tests, etcetera. At the end of the test session there is some sort of debriefing and reporting to relevant stakeholders within or beyond the team.

Combining coverage- and experience-based testing approaches

Based on the test strategy, the team decides whether to use coverage-based testing or experience-based testing as the main approach in a specific situation.

Always combine experience-based and coverage-based testing

 

However, we strongly advise to always have some level of combination of the two approaches. If, for example, the team chooses to use experience-based testing as the main approach, the tester that encounters a boundary, should still do a boundary value analysis for that boundary. On the other hand, if the team chooses to use coverage-based testing as the main approach, for example because they will automate all tests, meaning they will script them, the team should also do some experience-based testing; for example an exploratory test of a team member together with a user.

One reason for such an exploratory test is that a testing tool may not notice an obvious fault. For example, if there are white letters on a white background, the tool will still pass the test that checks if the right letters are displayed, whereas people will not see anything and raise an anomaly report. So, keep in mind to always combine experience-based and coverage-based testing to some extent.