What things have you learnt about using device farms for testing?

Well…

But jokes aside.

Here are my 2¢ to the subject of device labs.

Device labs are a major part of mobile testing for quite some time, years to be more precise.

I’ve been designing local device labs since 2015, it’s a great fun but also a lot of work.
I’ve made a presentation on my first attempt, but it’s only in Polish.

Right now, there are so many device labs to choose from, and they are so cheap, it’s wonderful!

Sad part is that the market started to get too competitive, prices are sometimes too low to assure good quality of the product, which should support my team in assuring better quality of our product. Ironic a little, isn’t it?

I will not name any names, but some device lab providers:

  • Don’t add new devices
  • Don’t update OS on devices
  • Struggle with installing an app faster than 10–15 minutes on a single device
  • Struggle with ping over 200 to their devices
  • Don’t clean devices properly after test was done
  • Fail 33% of the test because of their devices and infrastructure instability
  • Don’t add any devices apart from top popular
  • Don’t update device list in documentation
  • Don’t reply for paid customers for over 72h

Right now, there are so many device labs to choose from

And I mean, so many, that’s just some of them that support manual testing:

  • BrowserStack
  • Bitbar
  • Sauce Labs
  • Perfecto Mobile
  • Kobiton
  • AWS Device farm
  • Experitest
  • pCloudy
  • TestBirds
  • T-Systems Device Cloud
  • Sigos App Experience
  • NTT Remote TestKit
  • Testsigma
  • Samsung Developers Lab
  • Huawei Digix Lab
  • HeadSpin Local
  • SmartDust
  • AirLab 163
  • TestingWhiz
  • TestingBot
  • LambdaTest
  • WeTest

Longer list: https://github.com/pwicherski/MobileTestingBookHandbook/blob/master/device-labs.md

2 Likes