Tonights TestChat about Device Testing, our second of 2018 got us all talking about what our test labs look like and how we decide what devices to test on. A brief summary of the answers is below.
Q1. How do you decide what devices to test on? How do you prioritise those devices?
@deborah.lee was full of chat on this one 
âTry and get stats to give an idea on what the website/app users are usingâ
âIf no stats available, find general stats for popular models/OS/browser combosâ
âYou canât cover everything but aim to cover popular devices/OS/browser combos. If users report issues, ask them what environment they are on and try and reproduce with real or virtual device(s)â
âIt also depends in which country/countries the app is used i.e. popular phone models vary according to the countryâ
@ranjani read my mind âFor android it is more challenging as there are n number of devices across the globe!â
Doug sounds like heâs had a fun time âFragmentation is still an issue, so simplifying this to a âdeviceâ without considering OS version is shortsighted. As weâve found out the hard way!â
Q2. How does your mindset differ based on the device youâre testing on? For example, for mobile testing you need to test for recoverability after signal loss. Do you have particular heuristics you use for each?
@deborah.lee was on fire for this one!
- LONG FUN CUP
- SFDPOT too - not specific to desktop/mobile
- I SLICED UP FUN
- Mustnât forget testing without a stable wi-fi or ethernet connection for mobile apps. One of the biggest differences between desktop/mobile. That and orientation
Iâve seen some great orientation specific bugs, have you?
Q3. What does your device testing lab look like? Do you use physical devices? Or Emulators, cloud, simulators?
@ranjani âFor automation, I use Firebase labs - mostly emulators coz physical devices are expensive.â and âThe only caveat is Xamarin test labs were expensive than Firebase.â âI have also used AWS Device Farm in the past but ran into a lot of issues with the test failing on their device farm due to popups/device updates, etcâ plus tonnes more awesome insight, check out the transcript!
For me âI had a handful of real devices that I borrowed from anyone and everyone getting rid of their old phones. We had a good variety of phones & laptops in the office too. The rest was browserstack & Chrome dev toolsâ
Simon Gilmurray âMix, mainly physical as it feels to replicate reality better, but Simulator in iOS and Genymotion. Used to use Xamarin Test Cloud which was a really nice way to submit automated tests up and run them simultaneously against many different devices and OSâs.â
@ustanisic was the first to mention a crowd sourcing solution âmostly emulators and physical devices. Weâre also using some crowd testing approach between testers from different projects in order to cover more ground and more real devicesâ
Q4. What has been your most interesting bug discovery while device testing?
Zena experienced âcrashes at Boundary Value analysis!â and âthere is an issue with Captive WIFI on current IOS update, once u try to connect it and lock the screen then on UI you can still see Captive WIFI login screen, I have raised it, lets see when they can fix it!â
@ustanisic âcrashes when switching from wifi to 3G. When I took the device outside :)â
@deborah.lee âOne issue was actually specific to the iOS version (i think iOS 8.0), where you couldnât upload a photo in Safari. Took a while to work out (alongside dev) that it was not the website or the phone causing the issue!â
My own one ânumber picker on Android appeared as a nice numeric keyboard, iOS it appears as a huge scrollable wheel, try get to 30th of the month using thatâ
Ian Emery âcouple of rendering bugs specific to an iPhone 7 were a touch weird. The product owner also found a bug by pressing to one side of a button this week, caused an invisible option to pop up!â
If you missed the TestChat, why not continue the conversation here?