Hey everyone!
Would like some advice on the following please.
We have UI automated tests running on a nightly build for several apps/projects and in the last few weeks we are seeing some errors on flaky tests. I would normally raise it in the stand up and fix the issues unless there’s quite a few and then the devs would jump in and help out.
We went through a phase of about 3-4 weeks where some tests were failing and none of us looked at them. This maintenance seems hard work though.
I enjoy fixing the broken tests although I would prefer writing new selenium webdriver tests instead but I don’t think the team would appreciate me spending half a day every few days fixing broken tests. My role is a mixture of manual and test automation
The team appreciates its not just the QA’s responsibility for looking out and fixing broken tests but does anyone have something in place or have they gone through something similar and can shed some light?
Yes our tests should be robust but sometimes they just fail and may not fail again for a few weeks
If you’re too deep into problems, do a reevaluation. You can start alone, present the problems and ideas to the team, and get their feedback.
Here’s what I’d question, among others(depending on the situation):
Are we automating the right things;
** things that bring us valuable information from risky areas which significantly reduce the time it takes to check them otherwise? Can we reduce the number of checks in noninteresting places?
Are some checks duplicated at lower levels(unit/integration/web-services)? Can we reduce the UI ones?
Do we have enough resources to maintain existing automation? What would it take to allow us to do so fully?
What would we give up from the tasks of the current people, if hiring new ones wouldn’t be possible and the existing ones would have to maintain it?
Is the UI automation approach the right one?
** too many unstable elements used, hardcoded data, async behaviors acting differently, large chained dependencies, data/environments?
** technology is not appropriate? is there a link between tools used, coding methods and the stability of the checks? are there tools that would be more appropriate?
** coding/debugging skills could be better? are people coding them creating messy stuff? ask a developer to review and give some guidelines on improving coding.
** too much fluff added on top? some UI automation frameworks are developed in unneeded complicated ways, because the coders like shiny stuff and experimenting.
How easy is it to debug? Enough loggers, can we run it easily through an IDE with debugging breakpoints, screenshots or recordings available,
I’ve seen these before:
stop using it; review what would happen in x month if you don’t; with fresh things in mind take a different approach, trying to have a sustainable thing.
rewrite the framework(change tools, language, process) each few years;
hire more automation engineers; of course experiments with devs, other testers, business failed;
drop it and do smarter, professional testing;
reduce the scope significantly, only have a few dozens long sequence scenarios;
externalize it(and pay others to handle it);
do more at lower levels where things are a bit more stable.
throw it away, product is not changed anymore(under life-support/maintenance only), or product changed completely(any old stuff is irrelevant).
We love fixing tests that break actually, but yes, temptation can be to try and fix all of them as an ongoing task, that eventually starts to consume 100% of your time. Most testers, like yourself, detect this starting to happen, and say woa!
Basically just like you, I get the devs involved a bit to either make the product easier to test, with more hooks or logs, or make the product behaviour more consistent so that test code can be re-used in more places. We all have flakey tests, I prefer to gradually disable them one at a time, but leave them until I have a fair bunch, and then grab all the related ones and re-evaluate and re-write them in one go.
I find the Five S’s a good structure for maintaining automatd tests. I have used them with the low code tool Ghost Inspector and now use them with Playwright. I wrote this about them: testandanalysis.home.blog/2023/01/10/the-five-ss-create-a-structure-for-test-automation/
This is good but (still, AI might change it) also quite expensive. One should be aware of that before starting test automation of something tested manually. Too many stakeholders and managers do not realize that.
Hi, really appreciate your detailed response.
Some of the UI automated tests were written by devs about 5-6 years ago, the functionality still exists and small changes to the app are made here and there maybe a couple in a month.
Spending time on the automated tests is not easy as developers are focussing on new features and fixing bugs and I’m trying to balance between manual testing and writing automated tests. Devs would help out in fixing broken automated tests when there is a need.
For the last couple of years i’ve tried to have a balance between manual testing and test automation as test automation is the route i wanted to go into. So in my squad I am the only QA and there’s 5 other developers.
We definetly don’t have enough resources maintaining the automation tests although it can work maybe if everyone in the squad took turns in monitoring the test results every morning and fixed broken tests
Another thing is finding the time to go back to the very old tests, which i think is the main concern and either re-writing them or going through them to see if all of them are needed, this seems a time consuming exercise and not sure the higher up people would allow that time.
Yep it is time consuming but its not everyday but the thing with me is that i prefer test automation/writing code/fixing tests not sure if i’ve had a burn out with manual testing if that’s such a thing