Hi guys,
Hopefully you can help me with figuring out how much of automated visual testing it’s worth having. Is it worth prioritising visual tests in general? Is it worth using tools like snapshotting for visual regression?
Right now we use WebdriverIO Selenium for UI functional tests.
3 Likes
You might have a look at eyes of Applitools which can be combined with Selenium. There are some free courses on Test Automation University:
A few years ago I wrote a serie of blog posts how to combine Visual Testing and Test Driven Development. Here is the first one:
2 Likes
Hi @mariam-sargsyan,
Thanks for asking here on The Club.
If not on your radar, consider joining a Ministry of Testing event called Test.bash();. They’ll be lots of chat, talks and demos of visual automation tools. I have a feeling it would help answer questions you have about visual testing.
1 Like
Hi Mariam, the answer for implementing any automation framework in your project depends on what you’re trying to achieve. It’s important to consider that rather than go for tools straight away. You mention you have UI functional tests already, what purpose do they serve? Are they giving you useful results so far? What insights have you gained from that in the past?
Hi! Thanks for your reply. The main purpose for us right now is to lighten the load for manual regression testing by having a solid regression test coverage.
The framework was there long ago, and the acceptance tests were integrated into the CI, but the test cases weren’t documented as automated and so the main purpose we wanted them to serve wasn’t fulfilled.
Also, right now we don’t have any reporting tool to give us insights and that’s also something we’re considering to implement.
The problem is the more rigid your toolset gets the more brittle it tends to become. If you have a stable UI you very rarely intend to change and it’s very important that it stays the same then you have a case to be made. If you don’t then you’ll have a circus of failing tests that need to be fixed and the suite run multiple times - a heap of stress and cost you absolutely don’t need. You need to find your balance between cost and utility - every automation tool and suite is it’s own project with it’s own code with its own rot and usability problems and stack integrations and before you know it you’re fighting bugs in the software testing the software you’re trying to find bugs in.
Know your situation and what you’re buying for your costs. You probably have read a whole load of stories of people who get into tooled solutions then find themselves abandoning it because the costs sucked the flow out of the system - automation feels like it’s a lot cheaper than it is. But if you have a non-volatile system and your UI is very important to someone important then it might be worth it, or some other outlier like if you’re Google. Consider working around your problem if you can - your tools must serve an intelligent test strategy or you’re working for them instead.
You’re going to need to deal with some issues if you go visual, and the three I’m thinking of right now are dynamic changes (anything that has to change, like a system clock), false positives/negatives and managing all your comparison images. Image comparison has some problems with things like compression artefacts, scaling autofill, sub-pixel information and so on that make comparisons difficult. Most systems to deal with visual comparison will have things like fuzzing systems and error ratio percentages and whatnot to make matches more likely - essentially turning up the “how much wrongness is okay” dial until your tests pass without complaining. Just make sure they can fail, and that your reporting system also works and is readable/traceable/useful.
With regards to reducing regression pressures I’ve done this before, and I’ve written on it before, but generally speaking find the actual human purpose behind the regression tests and see what you can simply cut out, what you can deformalise, what you can push elsewhere and what absolutely must be done. Use purpose to reorganise.
You can organise into what must be done the same way each time (lab conditions, formalised instructions, high integrity, expensive, rarely finds new things) and what can be investigated with more open ideas (start from different states, complex and challenging actions, finds more problems, cheaper maintenance, more adaptable). This will bring down the cost, help you find more problems at regression, but you won’t lose sleep over the things that matter - but first you need to know what matters to people who matter.
Another thing might be to look at change risk - remove some of the regression suite and have teams bring along things they want to look at this regression. It will depend a lot on what you’re making and how.
Hope some of that is useful.
1 Like
Hi Mariam,
Would you like to try testRigor? You can do things like check that certain part of the page looks the same way or just plain use Applitools easily on a certain page. Check it out here: testRigor Language Support Documentation - testRigor AI-Based Automated Testing Tool and you can create your free account here: Sign Up - testRigor Test Automation Tool if you go for open source option.
Disclaimer: I’m a testRigor co-founder. 
Artem
1 Like
wow, all of this was really helpful! Thank you for taking the time to explain
1 Like