Crowd Testing

Crowd testing, or crowd sourced testing, is when a virtual group of people throughout the world (a crowd) is involved in testing. This will be a group beyond the DevOps team, for example consisting of people from the user organization or people beyond the organization, such as users throughout the world.


Crowd testing is testing performed by a variety of individual testers from outside the team. They usually test on remote locations. It is often used to test on a wide variety of devices by people with a wide variety of backgrounds and experience.  

"Crowd testing" has its roots in "crowd sourcing". The key to the success of crowd sourcing is that a lot more ideas can be generated within a larger group of people, and those ideas can influence and support each other. Current cloud and web technologies enable the sharing of ideas within large groups. This same advantage applies to crowd-sourced testing: many testers can find plenty of places to look for bugs, and the bugs that one tester finds can influence other testers to look for similar bugs.

Crowd testing has a second advantage: it is not just the ideas and experience of the different members that are more diverse; but also the different hardware and software configurations that the software under scrutiny is tested against. Crowd testing enables a test where software can be tested on a large number of devices, browsers and operating systems, where all tests are executed in parallel, minimizing the throughput time

Crowd testing


of the tests. Therefore, crowd testing is not only very popular in testing applications for mobile devices, it is also very popular in the online gaming community.

Because many people are involved in the crowd tests, often the duration can be much shorter than when a dedicated team of only a few testers has to perform the same amount of testing. (But when the test execution needs to happen in a certain order, or if testing in later steps of a process depends heavily on earlier steps, the duration is not necessarily shorter.)

A large group also provides more flexibility; crowd testing can be applied to progression testing as well as regression testing.

Managing the crowd

Managing a crowd of testers entails a different set of challenges than managing a traditional test team. Organizations may use an internal or external crowd. An internal crowd usually consists of volunteers from the organization that participate in the testing. An external crowd is often managed by specialized organizations that match a request for testing with a subset of their crowd.

If managing the crowd of testers seems like a burden for your organization, there are various companies that offer to professionally organize your crowd testing for you using a professional crowd testing platform with a controlled group of testers. For example, when using a crowd testing platform, it is possible to request the variants of mobile devices (that is operating system, hardware brand, etc.) that must be included in the test. And for internationalization tests it is also possible to specify which countries (or languages) should be covered by the testing crowd.

In the following sections we describe some considerations to take into account.

Testers or end users

Do you want your crowd to actively look for bugs or do you want them to use the software as they would in real life? In the first case, choose experienced test professionals as crowd testers. In the second case, choose typical end users.

Organize your own test crowd or use a crowd-testing company
In some cases, it is not possible to release the software beyond the bounds of your own organization. Financial institutions have an issue with releasing software to a large crowd of relatively unknown people. If this is the case, you can organize a crowd test among your own employees. However, if there are no restrictions on releasing a test version of your software outside the organization, there are numerous companies available with a global crowd of testers that can amount to hundreds of thousands.

Distributing testing tasks

Making clear to the crowd of testers which tasks they are supposed to execute may be a challenge. An assignment like "test this app" may not result in the focus required. It is wise to use exploratory testing charters to distribute the tasks (see Section 47.4). The charter must also contain clear information on the timebox that is assigned to the testing task.

Anomaly management

The anomaly reports are usually the most important feedback from the crowd. Make sure there is an easy way of registering observations and anomalies, for example in an online tool. At the end of the timebox, all the feedback from the crowd testers should be assembled and processed into a clear and concise overview of testing results.

Rewarding the crowd

There are different ways to manage the crowd. One thing to consider: how do I reward my testers? If you are managing a closed beta group within your organization, rewards are often for executing the whole test. For instance, if your company members test a new kind of device, they can keep the device after the test. If you are managing a public cloud, there may be other ways to reward testers. Two possibilities are often used:

  • Payment for each bug found
  • Payment for each test case executed.

A disadvantage of rewarding the crowd for each bug is that, with a primary focus on finding bugs, it may be difficult to get a good picture of the overall quality of the system under test.

Test environment = live environment (beta testing)

Since the testers work remotely, the test object must be made available to them, usually through the internet. The environment often has characteristics of a live environment. The crowd testing may even take place in an actual live environment, which makes it testing in production. Crowd testing may be used to perform beta testing.


Beta testing is testing performed by people from outside the organization that develops the system, at the site and in the environment of the people that perform the test (often regular users).

Disadvantages of crowd testing 

Crowd testing offers some clear advantages as stated above. However, there are some disadvantages that limit the use of crowd testing as it is currently developing:  

  • Confidentiality: the larger the crowd and the more it is outside the sphere of influence, the harder it becomes to manage confidentiality.
  • Knowledge: not every application is suited for crowd testing from a knowledge perspective, many applications are used within a company and specific knowledge of company products and processes is required to operate the software.
  • Testers who are paid "by the bug" will often look for easy-to-find bugs instead of looking for the most critical ones.
  • Ensuring total test coverage can be difficult with crowd testing.

Crowd testing and IOT

Crowd testing is a proven test variety in IOT testing: Read more.