I think itās worth having an honest discussion about crowd source testing from the client and testers perspective. Iāve been in meetings where crowd sourcing was seen as the new silver bullet, āNo need to hire a test team or more testers, weāll just crowd source itā.
Iāll start with my perspective on two different situations.
Iāve taken part in crowd sourcing through a reasonably well known crowd sourcing company. I use this to supplement my income. If I have a big bill or holiday coming up, I keep an eye out for projects with guaranteed payment. I do this because any with no guaranteed payment (bug only payment) take a LOT of work for very little or sometimes no return.
Iāve had mixed experiences with the guaranteed payment ones. My first one there was no communication from the client. In fact, if you tried to communicate with them you risked a lower payment at the end! This was really difficult because there was little information about the product under test and I felt like I wasted a lot of time figuring things out that should have been documented. For example, there wasnāt even a high level overview of what was the purpose of the product.
Contrast this with my recent experience where there was a document explaining the purpose of the product, pages/tabs that donāt or shouldnāt work and known bugs. It was a mobile application and some bugs seemed to appear randomly in their internal testing. They had a specific list of devices they needed the app tested on. In their Google sheets test case list they had context listed for each test case e.g. āTurning off mobile data on some devices stops location tracking but we arenāt sure how long it has to be off for or if environmental conditions such as driving through a tunnel affect this.ā
If we had any issues, there was a point of contact in the company assigned to each group of testers and no financial risk associated with contacting them. I only needed to contact them once because their documentation was really helpful.
In that situation I can see how crowd sourcing was good. They seemed to already have a really great test plan in place and device specific issues would be difficult to pinpoint. Paying each of us $200 was cheaper than buying our devices.
As I said in the beginning, Iāve been in meetings where I asked for another tester to be hired and crowd sourcing was offered as the solution. As the potential client in that situation I saw so many issues with this.
I feel (even as a crowd source tester) that the passion for the product is just not there when you crowd source. Of course, this can happen with a permanent tester too.
From a financial perspective for the company I thought about the two situations Iāve encountered above. In the first, I would have little contact with the testers. I would have to do up test case documentation and then check each set of results. Given the little contact with the testers, how much value would I actually get?
In the second situation, I would have to prepare all the documentation (as in the first) along with an overview of what the product was aiming to achieve that could explain it to testers from any background. Then someone would have to be available to answer questions the testers may have. Given many of these would not be doing this as a full time gig, Iād need to be able to answer those questions pretty quickly. I would no longer be a tester but Iād be in some sort of management position, most likely with no recognition of the fact.
At the end of either situation Iād also have to go through all of the bugs logged, make sure they had enough information and that they were prioritised correctly (critical, low, etc).
In the end we opted for a single, contracted tester for a fixed term. For me this was the best option because they were able to contribute to requirements, have face to face communication with the Devs and we only had to onboard one person rather than a crowd.
So, what do you think the pros and cons of crowd source testing are from either the client or the testers perspective?
Testing can be done in a shorter timescale Cons: -
*Setup prior to crowd source activity
A lot of hand holding with testers not fully under your control
Testers that probably arenāt fully engaged with your product
Potentially a lot of stress and work for little return
Tester Perspective Pros: -
Quick easy money
Lot of quick experience Cons:-
Experience that may not be recognised
I have worked with crowd sourcing companies the same as you @heather_reid just to supplement my income at times. I feel that i get a good bit of experience from it but it doesnāt come stress free. At times it does feel like a lot of hard work and maybe i donāt put my full concentration into it. I feel for the tester its more of a side project and you wont get people invested into the projects and products as they are only looking for the easy way to make additional money from their skills.
For a client i have worked in a company that outsourced some of the device testing to get a quick test coverage on a lot of devices. This worked well as it was a small team doing mobile testing on a large product, that team being me on my lonesome and not all my time dedicated to mobile. So i feel that for a client it can work well in certain situations such as supplementing the efforts already in place. I would never feel that it is a permanent option.
I think a better solution would be a managed test solution. Plenty of companies now on the market offering testing solutions such as Edge Testing Solutions, Scott Logic and others who will carry out testing for the company by trained professionals. People that you can employ and know their expertise and experience so you know you are getting a quality service.
I have also signed up and taken part in some crowd sourced testing, though for me this was near the start of my testing career so it was less about the money and more to work on my testing skills and see how the bugs I found and wrote up compared to bugs found by other testers and how they wrote their bug reports. It was really useful in this regards and I learnt a lot about how to improve what I do but also validated for me that I did know what I was doing as a tester and I compared quit well to other people working for the company.
For the client one of the cons I have seen is that the testing performed and the bugs raised are generally quite shallow and superficial. When you set testing up as a race where the first to find an issue wins and gets paid there is no incentive for someone to take their time and perform thoughtful, detailed testing on the product to find those hard to find but critical bugs. The client then also has to spend a lot of time sifting through lots of badly written bug reports about issues that ultimately arenāt that important.
For testers:
Pros:
Somewhere to practice your craft whilst getting a bit of extra cash
Exposure to testing in areas that you may not deal with in your day job
Cons:
Monetary return for time spent is generally not that good
i really liked it, to be part of crowd source teams for testing.
The only cons that is found is the lack of feedback that you receive with your tests, or that kind of communication that you can have with your client that can help you to gain more experience.
Besides of that is a good way to practice and learn more about testing.
About 7 months ago, I started using Rainforest QA, which is a crowd testing platform, and I must say that I have been extremely pleased with it. I write test cases in human readable format, launch or schedule test runs whenever I want, and Rainforest runs them all in parallel on the browser/OS combinations that I want. It functions for me in exactly the same way as front-end selenium automation would and, in my experience, itās both faster and cheaper. I know that this is not the typical experience and use-case for crowd-sourced testing, but I think that sophisticated crowd-sourcing platforms could potentially revolutionize āautomationā and help more organizations move toward continuous delivery.
Quite recently I was the sole tester for a large, complicated and not well understood program that had no automated testing of any sort. Any change that was made had a completely unknown impact and required a fully manual regression test.
Crowdsourcing was really valuable to me when we started a project to refresh some of the UI of that product as it offered a non-stop inflow of information. I could cast a very wide net, or a very restricted one, but the inflow of fresh people trying similar tasks in different ways really made a difference.
The gamification model many suppliers use obviously mean that testers predominantly reach for low-hanging fruit and it can be quite difficult to request investigation of more complicated or less obvious-looking lines of inquiry. Also, I had some trouble properly validating the results when crowdsourced testers ran test cases to check bugs were fixed because by and large they followed the steps robotically without much awareness of what they were looking for.
On balance Iād probably recommend it but warn that itās not a fire-and-forget sort of thing: if you take the resource on you will find that most, if not all, of your time is spent wrangling the crowdtesting resource and interpreting the results so be careful.
Not so keen on the testing competition pay per bug only model, seems like a huge testing footprint with a lot of waste not to mention that employment laws have just not caught up with the competition model and it may be dancing around potential minimum wage issues. Some test practice opportunities though.
Small crowdsourced teams though would really work well in my view, provide a contact point and you get allocated a team that meets your skill and coverage needs alongside a āfairā rate for the work. I can see this as a win/win all round, company gets the coverage it needs at a decent cost alongside the tester getting that flexibility they might be looking for.
Iām using Crowd Sourced testers via an agency for the first time in my current role and itās an arrangement which was in place before I started.
We use them to run a suite of test cases across a number of test environments. Yes, there have been challenges but these have been worked through and valuable defects are found as part of this service.
We have a good relationship with the project leads as part of our agreement but we donāt have visibility of the people doing the testing and thatās one of the constraints which I donāt particularly like.
Iām not sure if Iād use Crowd Sourced testers again if I had the option, itās a balance of scale versus the less tangible value of in-house testers.
Iāve just started using Crowd Source testing one or two weeks ago. I cannot really talk about the client perspective, but because I think Iāll probably have to be the client someday using Crowd source testing, then itās valuable to me to use it as a tester and see how it goes.
From a tester perspective: Pros
You will learn new things, specifically if you look at other issues found by other testers. Also because sometimes Test leaders ask you to use some specific tools
Money. Legend says that some can make this activity a source of sufficient income. I think it really depends where you live because everyone in the world is paid equally
Cons
You cannot value anything else done outside the scope which is sometimes very narrow. This is frustrating
Communication is really hard sometimes: due to timezone, and because the Test Leader is just an intermediate and he just have to answer the client you donāt have to.
Several times I spend a lot of time before realizing that I couldnāt do anything for the project (no account available, all accounts were already used by people that probably already test the application previously, environment not available for youā¦) and this is really boring to invest a lot for nothing
As a freelance tester myself Iāve got quite a few thoughts on crowd source testing - which Iāll outline below:
when I started out in iOS and Android app testing (2012) it was very useful to take part in some test projects with uTest and other platforms, in order to build up experience with apps in all different sectors (casino to retail) and different app-specific technologies (touch screen, GPS etc)
having built up a decent client base and worked on many different projects - some one-off and some ongoing - I can say from the client side that what clients have told me they most value about crowd sourced testing is the large array of different devices the testers tend to have, from the latest to some older devices to some really obscure devices, and also it has to be said they get a different view point on what bugs may be hidden in their systems, rather than just hearing from me or (in some cases) their internal testers
I havenāt gone back to testing on the crowd-testing platforms in a long time, as they had started bringing in fairly restrictive and annoying ābest practicesā such as insisting on (firstly) screenshots for every bug, then (more annoyingly) videos for all bugs - lots of bugs donāt require either screenshots or videos, but the platforms were covering their asses, probably due to some previous problems
Yes this is annoying if youāre running it on your personal device. Iāve had to re-capture many videos because of messages popping up from family on my screen like āIāll pick you up in at 6 and we can head to thenā. I didnāt know how the video captures would be used and thatās potentially personally identifiable information.
@andydotpitman - āExperience that may not be recognisedā. Could you please explain that ? Did you mean to say that the clients wonāt recognize you, or employers might not recognize the temp experience on your resume ?
@heather_reid - Thank you for sharing this blog post. The bottom line is that you cannot compete with cheaper resources. 2$ will buy you candy at best in the west, but in the east it could at least buy you a small & modest sandwich. Not trying to stir the pot there, but kids now have to deal with the consequences of grandpa invading countries in the east. Haha.