What is Crowd Testing?
Crowd testing, or crowdsourced testing, is when a virtual group of testers throughout the world (a crowd, first element from the Figure) is involved in testing rather than a traditionally managed test team on a single location. It is a form of test organisation.
Crowd Testing the IoT
An IoT solution covers a lot of interaction of the product with end users, other IoT products or IoT solutions. Short time to market and large amounts of IoT devices (and thus users) call for more interaction with this end user. Crowd testing is an already existing technique to apply to a system under test. In applying it to an IoT solution, the problem of infinite amount of other IoT connections can be tackled. So now not only a wide variety of hard- and software configurations are tested but also the influence of other IoT (connected) devices are covered in this test (second element from the Figure).
The key to the success of crowd sourcing is that within a larger group of people a lot more ideas can be generated and those ideas can influence and support each other. Current cloud en web 2.0 technologies enable the sharing of ideas within large groups. This same advantage goes for crowdsourced testing: a lot of testers can find a lot of places to look for bugs and the bugs that one tester finds can influence other testers to look for similar bugs.
Crowd testing has a second advantage. It is not just the ideas and experience of the different members that are more diverse, but also the different hardware and software configurations that the software under test is tested against. Crowdtesting enables a test where software can be tested on a large amount of devices, browsers and operating systems where all tests are executed in parallel by different testers, minimizing the throughput time of the tests.
This requires a big effort on how diagnostics is implemented in the product. Through logging data (error) situations from a crowd test can be analysed and fixed. The amount of feedback from a crowd test by the crowd testers can be overwhelming. They will give bug reports of varying quality. When organising a crowd test by the own company the quality can be questionable. Compared to that the quality is much higher when using a professional test company which has an excperienced crowd. Good diagnostics in your product, give good logging that can be used in analysis the feedback from all these end users. When diagnostics are also connected and available to product development a strategy can be set up to anticipate to bugs in the field, spot trends in use and also misuse of an IoT solution.
All this data coming back from a crowd test (being it log files, diagnostics information, reports from the users, emails and more) require a very good coordination (Third element from the Figure). The crowd test is a test project in the small and sometimes it can be even seen as a large test project. Before an IoT crowd test starts a base level of quality needs be present in the IoT solution.
Generally, there are three parties involved.
- The client
- the organizer of the test, and
- the cloud of testers.
The artifacts of a cloud test can be the same as of any other test. In most cases, the client is mostly interested in finding bugs, so a bug report is the most obvouis artifact. But in most cases, test plans, test cases and all other test artifacts can be made as well.
The success factors of crowd testing depend on the goals of the client.
The size, skills and diversity of the crowd.
The goals of the client. Generally, a cloud test gives very valuable results in terms of finding bugs and device coverage. If the client is looking for guarantees and testcase coverage, the results can be less clear.