What tools would you use for Mobile Automation testing?

Hi all

I have been given the task from my manager to start looking into automating our mobile app testcases for our Android app. Its now becoming an urgent request as it would free up time for me to do exploratory testing.
He asked me to look into MonkeyRunner but, its been proving a lot of pain to get the setup up and running. Both on my Linux and Windows environment.
I have done some coding with Javascript and all ready learning Python. Which I know most of these tools would need some form of coding in order for them to work.

Our requirements are:

  • Has to be able to use test Physical devices connected to PC
  • Has to be able to run Android. iOS is a bonus but priority is Android
  • Iam running on a Linux Debian environment but Iam trying to first on my home Windows 10 setup first. But ideally this tool should work on a Linux environment
  • Can test multiple devices at same time. Iam thinking of testing different OS phones at the same time. This would greatly cut our times down a lot.

Any help or suggestions would be great.

2 Likes

Anyone?.. :frowning:

I can further explain if need be or if anyone wants to ask me more questions about this, Iam more than happy to clarify.

Hi @mazza, I’ve shared this in a few places today so hopefully you get help very soon :slight_smile:

1 Like

You tried Appium?
Never used MonkeyRunner, but it does appear to be no more than a webdriver wrapper, from the tutorials I encounter. Appium is pretty much the same thing, but it is free. You may get more joy going the more popular Java bindings on Appium than to using the Python bindings, but a number of folk here used Appium at some point on both. Why free, well the team behind Appium want to sell you their cloud device access. Appium has decent support although their forum community online is small, and my experience has been that getting started has not been too hard.

As for multiple devices, that is going to require a hub, because webdriver is unstable if you run multiple devices attached to one PC simultaneously. It can be done, but many people either run test devices sequentially, or buy into a subscription in the various cloud offerings.

When starting off with, using a device farm is an extra complexity level. Physical devices are very expensive - which makes device farms attractive. But your developer cannot use the device farm to develop and debug on, so you will need to split your budget anyway. My team decided against device farms simply because the benefits of being able to test on the latest devices was not worth the subscription for the low risk that the app fails on a flagship device. I have used a different kind of device farm in the past, and can say, they are awesome. The benefits of using a tool that can eventually plug into one of the farms without requiring you to rewrite your entire test suite is a large factor in framework choice.

3 Likes

Hi @mazza , how important is it that you use your physical devices? There’s probably better options available if you offload this burden of physical devices.
You could use Amazon Device Farm https://aws.amazon.com/device-farm/faqs/
You can test on a far wider range of physical devices. I’ve had success with it in the past. Have a read of the FAQ and see if it meets your needs.

You could even look at Firebase https://firebase.google.com/docs/test-lab . I’ve never tried it but heard good things.

3 Likes

Thanks for the responses everyone :slight_smile:

I havent tried Appium but I have heard of it a few times and it seems to be getting popular. it does sound something I would need to look into. And theres more guides and tutorial for Appium vs monkeyrunner. So I think I would get started very quickly. I think I would be a bit worried on how user friendly Appium is with our app and just trying to get a basic function to work on our app to respond on Appium.

For us, having real devices vs emulators is a real mix up. We have used emulators in the past and work fine. But, if there was a bug on a very specific device and we couldnt replicate on the emulators, its a hard pill to swallow vs if it was on a physical device.
Having the device automate on more than one device might not be a high priority right now as just trying to get it to automate our app is more important. And I can always disconnect and reconnect a device for each testsuite.

Hi @mazza, I have worked on a variety of android and iOS apps, especially setting up automation frameworks.

I’d recommend you to go with the native testing framework that android supports. I’m not sure if you have heard about it. It is called Espresso.

https://developer.android.com/training/testing/ui-testing/espresso-testing

  • Has to be able to use test Physical devices connected to PC - Supported
  • Has to be able to run Android. - Supported
  • Iam running on a Linux Debian environment but Iam trying to first on my home Windows 10 setup first. But ideally this tool should work on a Linux environment - Supported
  • Can test multiple devices at same time. Iam thinking of testing different OS phones at the same time. This would greatly cut our times down a lot. - Supported (will run tests on all connected devices or selected ones according to your preference.

On top of satisfying all of your criteria, it also has a few other pros.

  • Can talk to the app directly.
  • Have access to app code to modify state, etc
  • Really good mocking support
  • Tests are fast.
  • No additional requirements to setup. Running ./gradlew connectedAndroidTest will run all your tests in all connected devices/emulators
  • Enables dev/tester collaboration, since it sits inside the same framework as the app.

That said, there is a little bit of up-skilling that will be required. You can use both Java and/or Kotlin for writing tests, but you will need to spend a bit of time learning about it. But you can get some help from the developers to get you started.

Please feel free to get in touch if you need a hand with this. Happy to help. Ta!

4 Likes

From my understanding, testing on a Mac would be the only way to test an iOS device successfully. Or by using a third party device farm.

Thank you soo much Jaswanth. That was a really helpful post and I really appreciate the effort taken to respond to my post.
I will reach out if I have any issues but Iam just gonna try following some of the online tutorials but already found some obstacles with these tutorials being compatible only with Android studio 2 and not 3 or even the newer release of 4.
I’ll let you know how I get on and thanks again :slight_smile:

I would recommend using Appium and Apptim together.

Yeah cool. Feel free to ask me any questions. Will do my best to help. Ta

1 Like

Correct , Appium requires XCode be installed in order to use the underlying Apple automation xcutil and so on parts of instruments. Which makes Appium and thus Sauce labs probably the simplest route to go for cross-platform apps if you are not a developer with Guru-level Java and whatever Apple language you use skills. Right now looking to see if my Appium tests can be ported into Sauce, the certificate pain and VLAN pain frightens me though.

Hi jaswanth. Iam not sure if this is something you encountered before or if its just me.
I got a question about following a tutorial for Espresso. Iam using Tutorials point. I can just understand most of whats been said so far. The real problem is, most of the online tutorials are out of date. And so, when I try to follow the Tutorials Point guide for Espresso, the code is deprecated and doesnt work.
I think the issue might be that Iam using a newer version of Android Studio and the guide is using an older version of Android Studio.
Whats the best of avoid pitfalls like this? Can you recommend any good online tutorials that are still up to date? Or should I just downgrade and just follow the tutorials as they are?

Hi @mazza,

I’d recommend the official codelab from google. You can find it here - Espresso Codelab

Let me know how you go. Good luck!

1 Like

@mazza

Need resources to learn espresso?

Here ya go:

Online courses:

Cheat Sheet:

Book:

Misc:

DSL:

1 Like

I would go for:

  • Only Android automation: Expresso
  • Android + iOS: Appium (or JS Webdriver.io). Remember that iOS automation implies a MacBook :frowning: