We have a new article out today from @ashutoshm on testing for connection issues with mobile apps:
This got me thinking:
What types of mobile-specific tests do you do?
Mobile testing is a beast in its own right and has its own unique test-design considerations and heuristics to use. What are some of your recommendations?
lots, but poorly, is the correct answer. In the middle of a release closedown and yesterday we have bugged a scenario where the app does not indicate that network is probably down, just one scenario, so not a showstopper. But the entire argument about how does an app know if the internet is down came up, and it is… well complicated. But I’d like to read the piece by @ashutoshm before I implicate my team in any crimes against quality, bwah ha ha.
Much of this post will not make sense if you are testing a web-only app. It’s likely that some of what I share here is irrelevant to you, and some of it is just my strongly held opinion, take it with some lemon and salt and a mariachi band is my opening advice.
OK, now that the band is playing in the background, I’ll start. We do use the iOS built in network throttle tool, and the Android throttly tool Throttly for Android - App Download not free, but costs less than a hamburger. The biggest troubles however is that the tools limit all traffic, so any automated tools and “webdriver” automation platforms fall over in a CI/CD context, so you end up with limited manual time. and at that point it becomes easier to use that time doing real life simulations, not scripted testing. Also be aware that toggling wifi on/off is NOT a valid test, that is because it actually tears down the network stack. You may have done this when testing your app on your PC, unplugging the cable does the same thing, it notifies the network stack that the LAN is down, that notification instantly terminates most kinds of network connections, most. And it is thus a very poor simulation of a network outage, just don’t use it as a test, it’s akin to testing if a car starts with no battery, it’s not a useful test.
Hybrid apps
If you have a purely web app, your life is easier, but if like me you are testing something that is not just a browser on a piece of wet string things change a lot.
We all work from home, now right? We do a story/technique here sometimes where we grab a team mate and get them to be the tech support guy or get them to be another user, both of you log into the “platform” while on a teams call (other collaboration tools exist), and drive through the use of the app while walking around in the park, or just your back yard. I have a shed outside with a metal roof, blocks signal like a charm. You could use the network limiter instead of walking about in real life, but the limiter limits your real lived experience, and can distract you because changing the limit means backgrounding your app briefly, which is another kettle of fish altogether. So choose a profile and stick to it. We use the 4G profile as a check of the worst usable experience, although I prefer to also give 3G profiles a go with a end-to-end. It does pass, but you need more time, so you cannot buddy on the 3G experience test.
I do not bother with API level tests, here is why: Environments can become non-trivial, the journey touches 3 or more different web services, and for most testers they are complicated to set up, run on 3 VM’s and editing the bootstrapping of a mobile app to point to a environment you can intimately control with a signed secure binary is no joke. All comms is encrypted and the time spent installing certificates to look at API call timings is not worth it when our app has pretty decent network logging anyway. So that’s an easy win for people I hope, get the devs to add logging.
User experience for native apps… well if you have an iphone you will notice a little toast thing sometimes appearing warning that you have no internet sometimes? Well, that is driven via the OS, it provides apps with a way to detect network disconnection is likely. Apps can feed that status back in the UI. Some browsers do, some don’t. Some apps do, your app should! This warns the user to quickly move to a better reception spot before their page or operation times out. Apple decide that your network is flakey differently to how Google/Android do, and if your app is designed to work off-grid or behind a firewall, that API indicator may not be useful in your context. But whatever your app does, it needs to warn people in a consistent way. Yesterday we found one small part of the journey in which it does not warn the user.
There are more things you do need to test with fresh eyes on mobile, but I hope this story has given people some ideas to make their lives easier for native and hybrid app testing.
Quite informative and engaging comments here.
One thing to note: While switching off Wifi On/Off is not the best idea, it is a pretty valid scenario. I have seen applications not handling internet connection issues at all.
So, while it is a very basic test it is worth a try to see what happens when there is a total connection loss while using an app.
Yes @ashutoshm , it is definitely a good thing to make your life easier as a tester to put all the common test surface controls into the quick bar, Wifi, GSM data and Bluetooth as well as light/dark mode. Just for other testers out there, to be aware than when you turn wifi off, it takes significant time to really come back on, and then also make sure you can easily find things like how to toggle between different on-screen keyboards and accessibility settings. This delay varies from device to device, particularly found tablets of the fruit and also non-fruit based vendors do not like loosing their data connection, so very good to be patient in that area. Generally it’s also much better to actually have 2 phones not one, and configure them differently enough to make noticing issues sooner. For example in light-dark modes by stepping through on 2 phones at once.
I once was testing rugged mobile devices for industrial usage (logistics) and it was part of our routine to wrap the devices with metal meshes. We did this to break the Wi-Fi and mobile network connection without any interaction at the device.
We where looking for retries, changing from Wi-Fi to mobile network as fallback and if it could recover once we removed the mesh.
Because the touchscreens were resistive (needs physical pressure instead of an electric contact) pens with pointed tips were the default. As our meshes had holes with around 1-2 mm we could still operate the devices through the mesh.
I wonder how it is done with capacitive touchscreens which typically have broader tips to simulate fingers. I do not know which hole size of a mesh is sufficient to break the connection.
Maybe you can also wrap the device in such a way that the screen is free. Or break the connection in another way. Going into the basement is an easy one but makes you leaving your desk.
