How many failures do you get from legitimate UI changes and how do you mitigate that from happening?

[dianadromey]

We were very happy to be able to invite @alexburlton to TestBash Autumn to give her talk Snapshots Through Time - How We Evolved Screenshot Testing at Our Company.
Alyssa Burlton who works for Glean had a great Q&A session, but plenty of questions were left to be answered. One of those questions was from @jack.taylor :

How many failures do you get from legitimate UI changes and how do you mitigate that from happening

What about you? What would you say?

[alyssaruth (Alyssa Burlton)]

Apologies for the delayed response to this!

The screenshots do see a fair amount of churn due to feature development causing intentional changes. Looking at our git history, it looks like we’ve had 10 pull requests so far this month where screenshots have been changed or added.

To mitigate it from happening, there are a few options available:

• Ensure screenshots are as focused as possible, to reduce the chance of the churn happening to start with. For instance, if your screenshot is of a modal, then it’s better to have it on a blank background than with your actual app behind it to avoid having to update it because of the background changing.
• Make sure the screenshots are easy to update when it does need to happen. It’s inevitable that screenshots will need updating once in a while - if they don’t, then you’re not screenshotting the right parts of your app! So it should be as easy and fast as possible for any developer to update them locally.

In our case, we’ve also got a mechanism where screenshots can be updated by the CI pipeline itself - including “[US]” in a commit message automatically triggers a screenshot update, which gets all the screenshots passing again if there’s been a change. So it’s not too painful when a failure like this occurs - the developer is informed and can simply tell CI to update them to get their change passing.

[sebastian_solidwork (Sebastian Stautz)]

I quote myself from this other thread:
https://club.ministryoftesting.com/t/what-makes-automated-tests-checks-flakey/72173/17

@sebastian_solidwork

I find “End 2 End” being done by GUI very wasteful. I know that is often a escape path for check automaters without much support for testability of the product.
Welcome to automation hell if you lack time to adapt the automation to the changes in the UI … I have been there.

In my current project we check most business logic via API. And the GUI part is more about “Does the UI works at all”, which includes the connection to the server.
The API parts needs around 1,5 hours and is quite stable in terms of check code calling the application. The GUI parts runs in around 10 minutes.

Part of the problem is sometimes the “We do our (most) automation through the GUI.”.

[ipstefan (Stefan Papusoi)]

@dianadromey

How many failures do you get from legitimate UI changes and how do you mitigate that from happening

In what role am I?
Failure of what? And for whom does the failure matter?
Legitimate UI changes in what sense?
Mitigate failure or the effects of failure on the product, team, business, testing?