Eduardo Fischer makes the following important points about mobile testing:
Device farms have solved big challenges in mobile testing to the point where they have become crucial to a successful mobile testing strategy. Without them, our only alternative is to test on only one device at a time, which means we can only āguaranteeā quality for one device.
Even though implementing and adjusting our pipelines and test suites to handle multiple devices can be challenging, convincing our entire user base to use only one brand of smartphone is impossible.
The technical aspect of instrumenting tests to run in device farms can be a little challenging but certainly, there will be good returns to implementing this type of solution into your pipeline, regardless of the vendor that you choose.
From the article: A Must For Mobile Testing: Device Farms
What things have you learnt about using device farms for testing? How have you dealt with the challenges of and reaped the benefits of running a whole suite on many devices with different configurations?
2 Likes
Wellā¦
But jokes aside.
Here are my 2Ā¢ to the subject of device labs.
Device labs are a major part of mobile testing for quite some time, years to be more precise.
Iāve been designing local device labs since 2015, itās a great fun but also a lot of work.
Iāve made a presentation on my first attempt, but itās only in Polish.
Right now, there are so many device labs to choose from, and they are so cheap, itās wonderful!
Sad part is that the market started to get too competitive, prices are sometimes too low to assure good quality of the product, which should support my team in assuring better quality of our product. Ironic a little, isnāt it?
I will not name any names, but some device lab providers:
- Donāt add new devices
- Donāt update OS on devices
- Struggle with installing an app faster than 10ā15 minutes on a single device
- Struggle with ping over 200 to their devices
- Donāt clean devices properly after test was done
- Fail 33% of the test because of their devices and infrastructure instability
- Donāt add any devices apart from top popular
- Donāt update device list in documentation
- Donāt reply for paid customers for over 72h
Right now, there are so many device labs to choose from
And I mean, so many, thatās just some of them that support manual testing:
- BrowserStack
- Bitbar
- Sauce Labs
- Perfecto Mobile
- Kobiton
- AWS Device farm
- Experitest
- pCloudy
- TestBirds
- T-Systems Device Cloud
- Sigos App Experience
- NTT Remote TestKit
- Testsigma
- Samsung Developers Lab
- Huawei Digix Lab
- HeadSpin Local
- SmartDust
- AirLab 163
- TestingWhiz
- TestingBot
- LambdaTest
- WeTest
Longer list: MobileTestingBookHandbook/device-labs.md at master Ā· pwicherski/MobileTestingBookHandbook Ā· GitHub
2 Likes
Agree that the cost dynamics in the mobile device testing business made it difficult. We (CrossBrowserTesting) provided devices starting in Oct 2010, when we added iPads, iPhones, and Android (Androids started off with just simulators, which were NOT good ;>). Ended up with hundreds and hundreds of devices. Maintaining a working farm is real work - batteries swell and you have to have a maintenance process, keeping processes working across a wide range of devices (iOS/Android, and in our case, Macs and Windows), all with different OS versions was very challenging. The permutations add up!
Another interesting challenges is the lack of information about how to provide testing on the devices, much less best practices. Even the physical management of the phones took āhackingā and time to develop the best approach. We started off by putting the phones in a dish drying rack (impressive, huh?) but ended up migrating to hanging the phones, as it made the cable management much cleaner. By the time I had left, we were putting 5 or 6 rows of phones in the front of the rack, and 5 or 6 in the back, and would have well over 100 in one rack.
Fun business but challenging and with more costs than most.
2 Likes
