I think it’s worth having an honest discussion about crowd source testing from the client and testers perspective. I’ve been in meetings where crowd sourcing was seen as the new silver bullet, “No need to hire a test team or more testers, we’ll just crowd source it”.
I’ll start with my perspective on two different situations.
I’ve taken part in crowd sourcing through a reasonably well known crowd sourcing company. I use this to supplement my income. If I have a big bill or holiday coming up, I keep an eye out for projects with guaranteed payment. I do this because any with no guaranteed payment (bug only payment) take a LOT of work for very little or sometimes no return.
I’ve had mixed experiences with the guaranteed payment ones. My first one there was no communication from the client. In fact, if you tried to communicate with them you risked a lower payment at the end! This was really difficult because there was little information about the product under test and I felt like I wasted a lot of time figuring things out that should have been documented. For example, there wasn’t even a high level overview of what was the purpose of the product.
Contrast this with my recent experience where there was a document explaining the purpose of the product, pages/tabs that don’t or shouldn’t work and known bugs. It was a mobile application and some bugs seemed to appear randomly in their internal testing. They had a specific list of devices they needed the app tested on. In their Google sheets test case list they had context listed for each test case e.g. "Turning off mobile data on some devices stops location tracking but we aren’t sure how long it has to be off for or if environmental conditions such as driving through a tunnel affect this."
If we had any issues, there was a point of contact in the company assigned to each group of testers and no financial risk associated with contacting them. I only needed to contact them once because their documentation was really helpful.
In that situation I can see how crowd sourcing was good. They seemed to already have a really great test plan in place and device specific issues would be difficult to pinpoint. Paying each of us $200 was cheaper than buying our devices.
As I said in the beginning, I’ve been in meetings where I asked for another tester to be hired and crowd sourcing was offered as the solution. As the potential client in that situation I saw so many issues with this.
I feel (even as a crowd source tester) that the passion for the product is just not there when you crowd source. Of course, this can happen with a permanent tester too.
From a financial perspective for the company I thought about the two situations I’ve encountered above. In the first, I would have little contact with the testers. I would have to do up test case documentation and then check each set of results. Given the little contact with the testers, how much value would I actually get?
In the second situation, I would have to prepare all the documentation (as in the first) along with an overview of what the product was aiming to achieve that could explain it to testers from any background. Then someone would have to be available to answer questions the testers may have. Given many of these would not be doing this as a full time gig, I’d need to be able to answer those questions pretty quickly. I would no longer be a tester but I’d be in some sort of management position, most likely with no recognition of the fact.
At the end of either situation I’d also have to go through all of the bugs logged, make sure they had enough information and that they were prioritised correctly (critical, low, etc).
In the end we opted for a single, contracted tester for a fixed term. For me this was the best option because they were able to contribute to requirements, have face to face communication with the Devs and we only had to onboard one person rather than a crowd.
So, what do you think the pros and cons of crowd source testing are from either the client or the testers perspective?