What Have You Learned About Running Automated E2E Tests Against Flutter Apps?

🙋 Questions
,
1

Hi there,

I’m going to be starting a new role as QA Lead for a company who is developing iOS and Android apps in Flutter, and the automation testing landscape is a greenfield site. As I’ve never worked with Flutter before I’m curious to find out what other people’s experiences have been, especially regarding automated E2E tests for mobile apps.

What test frameworks have worked well, and what ones not so well? E.g. Appium, the official Flutter test framework etc.

What device farm/s have you used and how did they work out? E.g. local real devices in the office, a cloud-based solution etc.

What CI/CD quirks did you discover?

Any other insights would be welcome too. Thanks in advance. :slight_smile:

Dean

2 Likes
2

Welcome back @weetabixhands . I’m keen to know too, we have a bit of jetpack/flutter code in one of our apps - it’s all a confusing landscape sometimes for testers. Not tried the Flutter testing tool yet, aaaand because we have a single hybrid browser launch step in one of the user journeys, I’m reticent to start over. Does flutter drive Chrome well and robustly alongside apps?

The devs want to rewrite the UI in flutter, which I’m a strong believer in doing, so keen to hear of people’s Flutter experiences.

3

I’m experimenting with Patrol at the moment.

It expands on the flutter integration test layer but drops the find lines which makes the code cleaner and it also adds in the ability to to interact with the native OS which is a bit of a gap with the flutter integration tests.

Its a very early look so far so I’ve not hooked it up to CI yet but likely will look at something like browser stack for this.

Apps rather than web I’ve found I needed to get a bit more code friendly and that has been a bit of an extra layer on my learning patrol, developers readme for local builds often miss a lot of things they have set up as default and I do find myself adding a lot of Keys which are equivalent to ID’s throughout the code myself. Though doing this level of detail has given me a bit more insight into risks that I can use in my day to day testing.

Because of the above I recommend pairing with a developer or at least having one on hand, it is designed for developer use though which I like.

It does have a few early quirks, using environment variables from files for example needs work, the build every run seems a bit odd but I think they are also working on options for that and they are also looking at parallel runs which I did not come across with flutter integration tests.

The company behind it were also at two recent flutter meetups in Europe with presentations so looks like they are serious on taking it forward even if its a small group. Whilst my own thinking at the moment is light end to end flow coverage which I think it will work for, they have talked about more complex banking style apps they have used it on.

I like testing flutter apps and generally I’ve found them less problematic and more consistent across OS than some other languages, my risk focused exploratory approach though at this point generally offers much more value than my e2e automated contribution so this side is still a minor part of the flutter testing coverage for me.

2 Likes
4

Thanks for your thoughts, Andrew. I’ll take a look into Patrol, it sounds like you’re having an interesting time!

5

I’m a tool vendor and speak to a few folks who are coming from or considering Patrol.

Things to consider with Patrol:

  1. You’ll need to plugin your own infrastructure to run tests in the cloud. This has the potential to draw team time away from “above-the-line” work.
  2. It’s a grey box solution. You’re creating a dependancy between the app code and the test tooling.
  3. It’s not app framework agnostic. If the dev choose to rewrite the app into React Native (for example), then your automated tests will need to be re-written and you’ll not be able to use them for regression testing the re-write.
  4. It’s a full coded solution. The tests are complex compared to low-code test creators.

Of course, those aren’t necessarily problems if you have the right team and infrastructure. But they can be problematic for lean teams.

My (very biased!) advice, would be to consider some more managed and declarative services as part of your search.
Such as:

  • Moropo (my tool)
  • Waldo
  • Sofy
  • testRigor

disclaimer: I’m a founder of Moropo and completely biased

6

I think its a very valid point that patrol is not exactly plug and play, to the point I do recommend pairing with a mobile developer for a few days to get the infrastructure up and running.

A couple of questions on Moropo, I may give it a quick look.

Is it aimed at mobile apps and not just mobile web apps, apps tend to be a bit more complex with the interactions.

When an app transitions to webview or native side, that could be clicking on a notification, a google login, a 3rd party overlay payment system or even just native device configuration does it handle that as smoothly.

Does it handle parallel runs?

With 2, 3 and 4, sometimes I find the cons of avoiding those outweigh the potential benefits.

Often my first question is are the developers building the product going to be comfortable using the tool, they in my experience tend to be more comfortable when 2, 3 and 4 are in place. Even things like selenium I find developers wont touch themselves so they drift more towards tester rather than developer friendly which in turn creates both a communication and collaboration distance and inefficiencies.

That might be worth looking at in terms of promoting the tool, is it developer friendly and are they going to be happy using it. Do you have a base of developers who would recommend the tool or do you feel it is more aimed at less technical testers and see that a postive?

I think its fine for some teams to see some things as a strength whilst others see the same things as a weakness, different contexts are important and like you I have my own bias.

Lean teams can go both ways, if a distance is created from the developers and that team only has developers or they decide at some point to drop the tester out will they actually continue to use the tool?

Either way if it is app focused and not web app focused I will likely give it a look and see if I can provide a better informed view than I have now.

7

Hey Andrew,

There are some great questions there.

Moropo is focused on mobile apps. Our Flutter docs are here: Flutter - Moropo

We can certainly handle WebViews and most other “off-app” interactions including OS settings etc. We sometimes have to provide a workaround for some very advanced use cases. I like to be challenged, so suggest a flow, and I’ll build it out or confess that we can’t do it :slight_smile:

We support parallel runs and actively encourage them to get test suite results under 5 mins.

Our Test Creator is low-code (ungated sandbox). Some devs hate that; some devs love it. Folks who want to build their own testing system and control every part of the infrastructure won’t like Moropo. Folks who just want something quick, simple and easy to use will like it.

My opinion is that devs should be writing features and bug fixes. They shouldn’t be “marking their own homework”. QA should belong to another role, and they should be empowered to run test automation without needing to be an engineer.

Let me know if you have other questions :slight_smile: