TestBash Mobile 2022 - Test Automation Frameworks with Jaswanth Manigundan

For TestBash Mobile 2022, @jaswanth chose 4 frameworks he has used in high profile projects that he’s worked on with so many challenges:

  • Framework #1: Appium (iOS and Android)
  • Framework #2: XCUI Test (iOS)
  • Framework #3: Espresso (Android)
  • Framework #4: Detox (React Native)

We’ll use this Club thread to share resources mentioned during the session and answer any questions we don’t get to during the live session.

4 Likes

Question by @bartzet : Referring to your mocking example, did you add any mocking library to achieve that or pure swift?

3 Likes

From @arcigo regarding Jaswanth’s last talk :love_you_gesture:

About the talk :mega:

Web automation frameworks have evolved over time and have become very mature over the years. But the same is not true for mobile apps.

Did you want to kick start your journey into mobile test automation? Or have you found it harder to choose the right one for your team? Let’s get started!

Why these 4? The frameworks I have chosen have been used in high profile projects that I’ve worked on with so many challenges. So I’m speaking from real experience, rather than just the theory behind it.

Also, all 4 frameworks I’ll be showcasing are well-documented and well-supported open-source code-based solutions, something that you can use in a software development team right away without any blockers (like money, approvals, etc).

Framework #4: Detox (React Native)

We will be using the same workflow for all the test framework demos. We will write a UI test using appium for the chosen workflow.

Detox is an open-source framework specifically built for react native apps. You can run the same test on both Android and iOS without having to change anything, given the app is also written using react-native.

Takeaways

  • What does it take to write an Detox based UI test for android and iOS?
  • In what scenarios should you use Detox?
  • When not to use Detox?
  • What type of teams can benefit in choosing a framework like detox?
  • What level of skill set is required to write a test using Detox (so you can go and prep yourself for it).

About Jaswanth! :female_detective: :bug: :iphone:

I am a lead test automation engineer working as a part of the QE practice at Culture Amp led by the legendary Anne-Marie Charret. I’m a developer turned tester. Even though writing code to build applications is so much fun, my now best friend showed me a few years ago that helping write good quality apps is much more fun. I jumped into testing from being a developer and have never looked back. I’m very passionate about native testing frameworks for web, backend, infra and mobile platforms CI/CD is something I am very passionate about. On the web side of things, AWS CDK has been quite intriguing. I have been an advocate of Infra as Code since working on it. Devops is something that is quite exciting for me, especially given it is always evolving. I have been speaking at a few international conferences on testing and CI/CD. Saucecon’21, Ministry of Testing’s Testbash and UI testing week events to name a few. It has been so much fun building testing frameworks for a wide range of platforms including mobile, web, infrastructure and microservices. Please feel free to reach out to chat anything testing.

Social networks :globe_with_meridians: :computer:

Questions!

Live answered

  • Saif Siddiqui: Last time (around 1 year back)when I used Detox the android setup was not at all stable? How is the situation now?
  • Karen Todd: Which video was your favorite to make? And which was most user friendly?
  • Saif Siddiqui: I see mocha is being used as a test runner, I used jest at that time since it is also from FB? Can you share your opinion which is better jest or mocha?
  • Anonymous You join a completely green field mobile dev project tomorrow, what are you choosing for Android and iOS?
  • Bart Ziemba: Referring to your mocking example, did you add any mocking library to achieve that or pure swift.
  • Saif Siddiqui: last question :sweat_smile: Test-id incase missing is added by QA or by the developer since it is a grey box testing tool? How you deal with the Test-ID challenge?
  • Anonymous: I’m investing £5m in Jas’s Mobile Automation tool, what features would it have?

Not answered :frowning:

Everything was answered! :smile:

Useful resources shared

By @friendlytester .-

1 Like

Look forward to the clips going up live next week or so, bookmarking here so I get a ping.
I have questions about how we can use these automation tools as “assistive” tooling. A bit like going back to the roots of automation and help with “board bring-up” or “app bring-up” using the fancy frameworks to help in situations where you don’t want to actually write conde, and don’t want to spend ages installing tools either.

2 Likes

Hi @conrad.connected , sorry for the delay in getting back to you. You can now re-watch all the talks on the site!

3 Likes

@jaswanth Hi

  1. I was a bit confused by the way you wrote test code in a style and smell that I would never do when you started comparing the various test tools. For example when coding up the appium script you perform a list scroll using a “Touch” and some hard-coded screen co-ordinates to tap and swipe. If we want to do that in a test, it’s too easy to scroll too far or too short and even to completely ignore different device screen sizes and the flakiness it would create for a learner.

  2. I was then further a little bit surprised in the XCUITest example to see the test use a scroll-until-element-visible approach, and not to Touch and swipe using XCUiTest, like you choose not to use in the appium example earlier? Why this change of tactic?

2 Likes

Hi @conrad.connected,

Good to read your thoughts and feedback. If I’ve understood correctly, it seems you’d take an alternative approach.

Is there a question you’d like to ask @jaswanth on point 1 of your feedback or something else, in general, you had in mind when sharing your feedback?

Great question @conrad.connected :+1:

I’ll answer it with some code examples when I get to the computer in the morning :blush::sunny:

2 Likes

I found a number of useful pointers in these segments in order to understand strong points in each of the tools, So this was a very useful talk, very. It covered a lot of the pain points. I care about this topic a lot.

But when you compare 2 , 3 or 4 tools, you want to compare the same task fairly and you also want to follow best practice for each tool, so as not to mislead anyone who is new. So using hard-coded swipe co-ordinates in a test even though it is just an example is going to lead people to believe that it’s OK to do so. I know time was limited. I would also have liked to see a tiny bit more of an explanation of the differences between Swift, Objective C and what they do for test readability versus Java. Segment 2 just jumps right into XCode without much warning or background of why the languages are good ones to learn as a tester in general.
I’d really like to see a more polished version of the segments, but with a broader target audience in mind, in a future TestBash stream somewhere.

/edit Would also be more specific about the advantages and the actual boundaries that “Hermetic Testing” advantage of using Xcuitest and Espresso gives us, what dependencies does xcuitest specifically eliminate for us? How is hermetic testing different to mocking?

I’m also not finding the video upload segment covering the Detox Framework in the MOT Pro recordings.

2 Likes

Thanks a lot for a very honest feedback Conrad :slight_smile: Really appreciate it!

I’ll try and answer all the questions below. Please feel free to add your comments too, I’d love to learn from you as well :pray:

1. Thought process behind the swipe coordinates

The whole premise of the talk is to compare how best each frameworks can handle, when testing a list view. So when I tried to do this with appium, the best I could find was to use scroll function Scroll - Appium. However, this could only work when I have an anchor element provided. Unfortunately, this doesn’t work on different types of list views as I wanted. (I assume it works for scrollview?). Therefore in order to keep the comparison fair across all frameworks, I had to use touch actions to perform this on appium. Hope that provides some clarity :pray:

2. Language readability - Java/Swift/Objective C

I really didn’t even think of the readability aspect of the languages. I just assumed this. Thanks a lot for pointing this out. I’ll make sure to add it to the improvements, if I had the chance to do this talk again.

3. Advantages of Hermetic testing

When it comes to hermetic testing, the difference between mocking is that hermetic testing (at least in the context I meant) is to setup and control different dependencies, within your test itself. For example, able to control and manipulate database, network, sdk and all dependency layers, within the test itself. I spoke about it here. Happy to answer more questions regarding this too :slight_smile: Tooling for Automated Testing with Jaswanth Manigundan | MoT

To be perfectly honest, I kinda underestimated the work it required to put this talk together. It was four different talks and combining it into 10 min presentations was very challenging. So, apologies if some of the thoughts were lost in that translation.

Once again, thanks for the feedback. Really appreciate it! :pray:

1 Like

Hmmm never thought about the scrollview being possibly so problematic. I have some swipe coordinates based code in some of the code I inherited too, so it seems like the only way sometimes. I guess it differs for web and native apps. I see your huge amount of effort, very good putting together a example app that was consistent at showing a very hard problem to crack.

Hermetic testing is really a good thing for programmers to be doing, I guess, having a plan for testing corners of an app that testing in a “wild” environment makes really hard to do. Probably why I do not often hear the term used. Thanks for reminding us all about options we may have nearby if we get help from a developer. I was a coder once, and did write such a test tool ages ago.

To be fair, you did not get enough time to cover the background detail, I’ve only read up about the other frameworks, and the bit of detail you gave has helped me be more confident about sticking with Appium, mainly because it’s at the right “altitude” or size for the job. So far your little segments have been my favorite part, and highlight the fun part of test automation for mobile.

2 Likes