I’m unsure what you agree on. I perceive quite the opposite to my writing.
I read your text like you say automation should be done only be the GUI.
I advocate for at least a mixed approach, also using the API instead of just the GUI as entry point.
Also I would heavily differentiate between automation, which works mostly for regression (as explained before, mostly on GUI level, you can finish automation only when the application works).
And testers (exploratory) testing the application itself. Just using it finds often many bugs fast. Also many the automation cannot find.
It’s typical test pyramid mythology to think we can produce software that meets all of the requirements if we unit-test and component-test it to death.
We have both read Michael Bach, who often preaches, “test the thing the customer sees”. We do still need to test both, because if we only test the UI, it makes triage and verification take longer and it also delays defect discovery to the point that defect cost goes up. And the fact that defects are cheaper to fix if we find them in unit testing can end up feeling like the customer actually cares about how many unit tests you have. So when we test, we divide and conquer, and must try hard to write UI tests that cover the UI behaviour, not the business logic. When I book an airline ticket, I actually tend to only worry about if the system has actually found me a unique seat on the aircraft and will print me a ticket with the same seat number I chose when I looked at the graphic of the plane. Users do use the GUI, not the innards Sometimes those UI behaviours can be broken out into mini-tests and avoid the flakiness of E2E, but testing at web-component level.
It’s typical test pyramid mythology to think we can produce software that meets all of the requirements if we unit-test and component-test it to death.
Wouldn’t this be true, or at least find 90% of the bugs and issues. By focusing down on the base, we would have much higher coverage of tests. While leaving QAs to do exploratory and follow the users ‘journey’ to catch what’s left in the UI? I am not saying UI E2E/Regressions aren’t important. This could be due to my inexperience as a whole, but I found that on the UI. QAs or BAs, tend to find the bugs and issues for better than UI Automations.
So, I guess it comes down to the purpose of the E2E ‘test’ we’re building. Are we actually using it to test and discover issues or to prove that future changes don’t break the application, but then with a changing UI we’d almost always expect the UI tests to break.
Any unsupervised " test automation" can just cover a comparable low number of scenarios (which they do reliable). Also automation needs at first to be developed before it can find bugs, which effort.
Humans (experienced testers) can way faster explore/test different scenarios (semi-automation/tools can help here). Just a subset of this will be automated for regression.
Also it happens to me often that I find bugs in the application during the development of automation which block this. By that I did a “weird” type of exploration/testing.
For short you need a, at the relevant places, bug-free application to develop your automation. First lest humans test/explore the application and once it is stable enough you can do development (you maybe can prepare some things, but especial at GUI automation much development work needs a working application).
This my experience of 15 years doing exploration as well as development of automation and test tools.
I’d take a step back and consider the value you are getting from your UI tests.
If you look at the last couple of years what really important things have they caught that would not have been caught otherwise?
Of those issues could they have been caught better somewhere else, maybe with better testing, developers flagging increased risks on certain changes to testers, maybe lower in the stack.
What areas within the coverage caught these issues, is their any pattern regarding the things the coverage caught, how they got introduced or types of issues, could process change prevent them?
I realise I have not addressed the ever changing UI element but if you can assess the value of the coverage first that can then help you decide the value of continuing given extra challenges.
If the tests are regularly catching critical issues, yes there may be better ways to do this but its likely worth keeping them up despite the challenges until you get those better ways in place.
The fear of telling people something is very low value is real, particularly if there’s even a very small chance it can blow up, history can help the decision be more informed.
Having skimmed the thread, I’m going to try to give a perspective that I haven’t seen addressed.
First, why is the UI changing so frequently? Does it really need to? Seems like there may be more systemic development practice issues, design churn, bandaids, or something else unhealthy driving such frequent change.
If it does need to, are a lot of the changes stylistic where, for example, more stable selectors could help limit the impact on the automated checks? (For example, XPath selectors are notoriously brittle. Class or ID-based selectors are better but could still be changed by developers for product-related functionality or styling reasons. The option I would recommend is working with devs to add data-testid attributes that are only used for tests to access elements, then there’s no reason for development to change them later).
Also, how are the automated checks structured? Are you using Page Object Models, or are UI element selectors scattered throughout the tests where you have to change many places when one UI element changes? I highly recommend Page Objects for their impact on maintainability.
As I said, the test-after-the-fact approach provides only regression value for future changes, with the cost of slower development time and harder test coding work due to the product not being designed with tests in mind (because they were there).
It’s typical test pyramid mythology to think we can produce software that meets all of the requirements if we unit-test and component-test it to death.