How would you approach testing the effectiveness of AI-powered 
visual testing tools in detecting subtle UI changes, such as color variations or font adjustments, across different browsers 
and devices? 
2 Likes
Hey @ansha_batra
Obviously the first step would be defining the test scenarios. These should be covering various screen resolutions/browsers, critical UI elements that are sensitive to changes.
The next would be creating the baseline screenshots. These screenshots would represent the expected appearance of UI.
Later I would introduce subtle changes. These should be small & challenging to notice visually.
Next I would run visual tests. In this step I would my Cypress Tests with Percy, which compares the present state with the baseline screenshots. Then, verify the results of the tool whether it accurately detects the introduced subtle changes.
Also, Cross Browser/Device Testing ensures in maintaining visual constituency & responsiveness of the AUT.
Finally, Continuous Monitoring through seamless integration into CI/CD pipeline helps to identify visual changes in each build.
In this approach, we can effectively use AI-powered tool such as Percy for detection of subtle UI changes. 
2 Likes
Rather than using AI when I do snapshot testing I:
- try to eliminate sources of indeterminacy in the code.
- try to block out areas of the snapshot which change due to indeterminacy.
The former would mean something like fixing a random number generator so that a randomly selected picture would always be the same, making sure all SELECT statements come with an ORDER BY or fixing the data in the database before doing a snapshot test.
The latter would mean ignoring the random picture when snapshot testing.
The former is preferable, not always possible. The latter is usually preferable, not always possible.
1 Like
I have been around using various forms of approval testing / golden master testing / snapshot testing for years, and so far I fail to see the promise in AI in this space.
A friend implemented a mobile application screenshot testing across huge farms of screen sizes and devices, no AI and very useful, not at all the maintenance nightmare.
One of my team members just recently added playwright snapshot tests. Making good choices on the scope of snapshots and recognizing the environmental dependency seemed to do do trick.
I believe we have a lot of algorithms work in this space, but much less AI than the marketing messaging seems to insinuate.
3 Likes
@poojitha-chandra has outlined a thorough and systematic approach to testing the effectiveness of AI-powered visual testing tools in detecting subtle UI changes.
I would add that it’s crucial to ensure that the test scenarios cover a wide range of potential variations, including different operating systems, device types, and browser versions.
Additionally, it’s important to establish clear criteria for what constitutes a successful detection of a subtle change, as this will help in evaluating the accuracy of the AI-powered tool.
Overall, this comprehensive approach will provide valuable insights into the capabilities and limitations of the tool in real-world scenarios.
1 Like
@hitchdev
Your approach focuses on minimizing sources of indeterminacy in snapshot testing by either fixing them in the code or selectively ignoring areas prone to change.
While both methods have their merits, it’s essential to consider the benefits of AI-powered visual testing tools, which offer automated detection of subtle UI changes across different browsers and devices, providing a more comprehensive solution.
@maaret
It’s valuable to hear about your experiences with approval testing and snapshot testing. While AI-powered visual testing tools may not always be the best fit for every scenario, they offer automated detection capabilities that can be beneficial, particularly in complex environments with numerous browsers and devices.
It’s essential to consider the specific requirements and constraints of each project when determining the most suitable testing approach. Thank you for sharing your insights.
@maaret @hitchdev @poojitha-chandra
Thank you for sharing your valuable insights💞 and taking the time to contribute to the discussion🙌Your perspectives have provided valuable considerations for approaching visual testing effectively.
1 Like
Your approach focuses on minimizing sources of indeterminacy in snapshot testing by either fixing them in the code or selectively ignoring areas prone to change.
While both methods have their merits, it’s essential to consider the benefits of AI-powered visual testing tools, which offer automated detection of subtle UI changes across different browsers and devices, providing a more comprehensive solution.
Speaking as somebody who has some background in data science, machine learning and whose primary work is with LLMs right now, I’d venture to say that it isn’t at all essential.
I think @maaret here is right on the money:
I believe we have a lot of algorithms work in this space, but much less AI than the marketing messaging seems to insinuate.
I can see lots of potential for AI (I wouldn’t be working in this sector if I didn’t) and a lot of snake oil. Most of what I see in terms of “testing AI” is mostly on the snake oil side and it has a lot of undeserved hype.
With snapshot testing specifically, it is not as popular as it could or should be but that is purely because the tooling/software we write and build (application code and test code) has issues with creating deterministic output and validating non-deterministic output deterministitically. This is not a problem that AI is an ideal fit for.
In fact it is a problem where adding AI can cause more issues than it solves - when I ask my LLMs at work a question today they will often give a different answer to the one they gave yesterday. They are stochastic parrots - not a new kind of magic.
1 Like