My name if Rtest, it has been a while since I have reached out to the test community. I need some help updating my testing methodology and processes. I have been testing an install client app on mobile devices for several years now. Both Android and IOS. I lead a small group of 4 QA testers in India. We rely heavily on manual test cases and use Qmetry to store and track our test cases and suites.
We are an Agile shop and practice Kanban and daily stand ups for the QA team and weekly check ins with the S/W developers. I have done some exploring with automation with Appium, Cucumber, Ruby and Java, but find cross platform testing with Android and IOS to be time consuming with large maintenance of tests. Is testing UI/UX better suited to testers and Manual testing?
We current maintain a device inventory of legacy devices and OS and as well as current popular devices that is quickly consuming our annual test budget . Testing Real devices and OS is a good practice but comes at a cost.
I am looking to see what others are doing for mobile testing of Installed apps (not browser). Devices, Device Farms online, simulators, automation, methodologies and process.
This sounds like a lot to ask, I know. But I would like to get my projects up to the 2021/22 thinking. I think I am stuck in the past…
Not just you Buddy, not just you.
If you are supporting legacy /wide selection of devices, going to one of the online farms looks like a good option. I’m eyeing them up, but it feels like far too much networking pain to get machines on our private network to talk to the “Farm” via a gateway that I would have to design and write myself. Probably worth going to an online Farm though if your test infra can run in the public network. You effectively end up paying per hour, and for the same money I can buy a new phone every few months and so having a local test environment is still the game I’m playing, because it’s a lot easier to debug too.
As for actual test languages and frameworks problem, a move to a “Device+browser Farm” may force you to rewrite a lot of your tests anyway. A move you probably only want to do once you have written down all of your constraints. Constraints can help clarify balanced business goals, the value of each, and cover problems you want to solve in the future. That would be a good point at which to “abstract” the IOS and Android interactions into a “layer” or classes of page-objects or whatever fits best.
I’ll offer a different perspective.
From 2018-2020 I’ve worked in a mobile team that created/maintained a popular iOS and Android app in the Netherlands, with over 1 million users.
We didn’t use a device farm. We barely had UI tests. Because the “cost” was too high and the risk too small.
Fragmentation on iOS is barely a problem, on Android it’s a slightly bigger problem. I looked at the analytics and made sure we were testing on the most prevalent OS + device combo’s and that was good enough. On Android we sometimes had weird bugs that only popped up on certain devices, but that was it.
The type of UI testing we did do was Snapshot testing, to spot visual bugs on devices with different screen sizes. But other than that…I never really noticed people complaining that the app didn’t work on their device. Sometimes on Android, but it was rare.
The only disclaimer I can give here is that in the Netherlands a lot of people use high-end devices so our situation might have been easy compared to yours.
We also were quite strict with minimum OS requirement. iOS was current version -2 (so right now iOS 12 is minimum). Android was a little more forgiving, we ended up supporting Android 4 & 5 for waaaay longer than I liked.
Thanks for you perspective. I agree… For iOS , Xcode will drive the minimum version of the operating system. I find IOS users like to stay current on updates and upgrades. Android user seem to stay on their legacy longer until forced to update or replace hardware. We currently support Android 5-12 along with Kindle Fire and Chromebook
Thanks for bring up the Private verse Public, and debugging. We have bad players trying to hack our DRM all the time…
Yeah, I have gone the same route as @maaike.brinkhof, as I’m a bit of a control freak and the cost of renting is still high. I just bought a shiney new ipad and a new google phone. I just don’t want to sound ant-test-farm, but it has it’s place. I plan to buy a new iphone later in the year, and then get a new Android tablet too, keeping coverage is easy if you have on small phone screen, one large tablet screen in both O/S. And then make sure to have one brand new device, and one lowest supported version device. A lot of the development-type tests the dev team run on the simulator are not really representative we have found. So I am able to give those to my developer team to test a tiny bit with because they are working at home now, it’s otherwise a pain to get my devs to test on a real device.
We have a “4 years” rolling policy, but have been lax on Android, and only recently upped the minimum supported to 7.0 which was not too hard since the app is free, and the stats showed that very few users were on Lollipop anyway. Chromebook may be my next testing device requirement, a pity you reminded me now. Nothing “official” has been said.
I tend to take the platforms on step further …IOS/ Android. iPhone , iPad mini , iPad / and for Android: phone, 7 and 8 " in Tabs and 10"+ Tab… This is kind of our Minimum testing…
Then we have larger Plus phones/Xls and iPad Pros, NOTES and iPencil
Buying devices for India testing and the COVID remote plan, makes sharing the devices impossible .WE bounce Jira Bug tickets back and forth as well as the implementation/freatur tickets so we get good coverage. Lots of places for issues to be missed.
Mostly we have 2 UI to test, Tab and Phone to take advantage of the extra space on the Tabs. We try to keep iOS and Android the “same”