🤖 Day 23: Assess AI effectiveness in visual testing and discuss the advantages

[dianadromey]

Welcome to Day 23! Today, we’ll assess the effectiveness of AI in visual testing compared to non-AI visual testing methods. The use of AI to detect visual anomalies within graphical user interfaces (GUIs) has great promise. So, let’s explore the potential advantages and pitfalls of adopting an AI-assisted visual testing approach.

Task Steps

Let’s begin this investigation by selecting one of two options based on your current experience and access to visual testing tools:

Option 1 - For those actively using or looking to get hands-on with visual testing tools

• Select a tool and examine the capabilities: Select a tool or platform that boasts AI-powered visual testing capabilities. Review the documentation or marketing materials to understand the AI approach and their anomaly detection claims.
• Test the claims: Design a time-boxed test (e.g., 30 minutes) to evaluate one of the tool’s AI-powered visual testing capabilities. For instance, if it claims to detect layout changes, intentionally modify the GUI and see how well the tool identifies the anomaly.
• Consider failure scenarios: Assuming the tool performed well in your test, construct a scenario where you think it might fail to detect a visual anomaly.
• Share your findings: In reply to this post, share your insights on AI-powered visual testing. Consider including
  • Which option you selected
  • Which tool you selected and the AI visual testing capabilities it claimed
  • Your findings from your timeboxed experiment
  • The potential advantages and risks of using AI for visual testing.
  • How likely you are to continue using the AI-powered visual testing tool.

Option 2 - For those new to visual testing or without access to tools.

• Research AI visual testing: Find resources (research papers, blog posts, documentation, video demos) discussing how AI is used for visual testing and GUI anomaly detection.
• Critique the AI approach: Try to identify the core benefits AI brings to visual testing and the techniques used by AI systems to analyse GUI images/screenshots and identify visual anomalies. Then hypothesise scenarios where an AI system might struggle to detect a visual anomaly.
• Assess if AI visual testing is for you: Consider whether an AI-powered visual testing solution would benefit your team based on the challenges you currently face.
• Share your findings: In reply to this post, share your insights on AI-powered visual testing. Consider including
  • Which option you selected,
  • A summary of what problems AI-powered visual testing claims to solve and how
  • The potential advantages and risks of adopting an AI visual testing approach
  • How likely you are to adopt AI-powered visual testing tools.

Why Take Part:

• Deepen your knowledge: Maintaining robust visual testing as UIs evolve can be challenging. This task helps you understand how AI could potentially streamline this process.
• Develop a critical mindset: When evaluating new testing tools or approaches, it’s crucial to think critically about their capabilities, limitations, and impacts on your team. Today’s task hones that skill.

Learn smarter with your team. Go Pro!

[adrianjr (Adrian)]

Hi my fellow testers,

For today’s challenge as I have some previous experience with visual testing in the form of Applitools Eyes and have documented this in previous days. I’m going to link to my posts from previous days challenges where I have already talked about my findings:

Day 5: Identify a case study on AI in testing and share your findings

Hi my fellow testers, for todays challenge I thought rather than search for a case study I would instead share my own experiences so far with trying to use AI tools within my workplace, I hope that’s ok. Brief background on the case study Back in October last year my workplace devised a learning week for us all where the challenge was to spend a week learning something that helped us develop individually and also could indirectly have a possible future benefit to the software we develop. I cho…

and

Day 17: Automate bug reporting with AI and share your process and evaluation

Hi my fellow testers, I’ve struggled with finding a new tool for this challenge so I have fallen back on an AI tool I have used previously, which is Applitools Eyes Automated Visual Testing & Monitoring with AI - Applitools Eyes as this uses AI its in detection of issues Evaluate the Reporting Quality: Assess the accuracy, completeness and quality of the bug reports generated by AI: Are the bugs identified by the AI valid issues? This was a mixed bag as they could have potentially been issues…

I hope these are useful to everyone, I’d be very interested in hearing experiences of visual testing tools that aren’t Applitools Eyes, as this is the tool I am aware of.

[poojitha-chandra (Poojitha Manam)]

Hello all,

I’ve chosen Option 1 & selected Percy. It claims to automatically detect visual UI changes & regressions by capturing screenshots of web pages & comparing them pixel by pixel.

@dianadromey

Your findings from your timeboxed experiment

I purposely changed the look of a test web page by adjusting its colors & moving some parts around. When I used Percy’s visual testing, it quickly found & showed me the differences on the page. It accurately spotted the color changes & the moved parts, proving it can accurately find layout & visual changes.

@dianadromey

The potential advantages and risks of using AI for visual testing.

The potenial advantages would be the efficiency, accuracy & consistency. But the risk would include False Positives/Negatives.

@dianadromey

How likely you are to continue using the AI-powered visual testing tool.

I see the benefits of using AI for checking visuals, I want to keep using Percy. But I’ll need to keep an eye on how well it works over time, especially for catching small visual changes & dealing with complicated websites.

[connmc (Conn McKay)]

@dianadromey
Hi Diana, I didn’t get this in email and only saw this today because of a link on Day 24.

I don’t perform any UI testing so maybe give this a pass. hope that doesn’t offend but i would like to spend the time on Security as I think AI is escalating the need to have proper security in place.

[poojitha-chandra (Poojitha Manam)]

Yeah, same with me, I too didn’t received the email on this challenge details.

[dianadromey]

Hi @connmc , thanks for raising it with us.
From our mailing system, I can see you received Day 23 email, on the 23rd of March, at 11:01 AM (UK Time). Have you checked your spam folder?

[dianadromey]

Hi @poojitha-chandra ,
Thanks for bringing it to our attention. I’ve had a look at our mailing system and indeed, Day 23 email was not sent to you for some mysterious reason as you’re meeting all the conditions for the email to be sent… so we’re now investigating with our mailing system why it was not sent to you.
Apologies about that.

[ash_winter]

Day 23

Option 2

Research AI visual testing

This was quite interesting and made some bold claims:

How AI in Visual Testing is transforming the Testing Landscape | BrowserStack

It talks about Computer Vision vs Visual AI. Computer vision allows a machine to derive information from visual stimuli, Visual AI goes beyond that with analysing and learning and apparently ‘forming thoughts.’ They both have a role to play in image classification, such as optical character recognition and object detection. Then object tracking where an object can be recognised and then following it in different contexts. For visual testing, Visual AI locates elements ‘as a human would’, even if the underlying element was renamed. The end of the article is surprisingly forthright that visual testing creates tons of false positives, even when augmented with AI capabilities.

Critique the AI approach

The article lists three scenarios where AI still struggles to recognise valid positives and negatives:

1. Anti aliasing of text with pixel by pixel comparison being too sensitive a medium.
2. Some parts in the app may change, click through adverts for example or dynamic campaigns.
3. Different browser engines mean images and fonts are treated differently but not indicative of a problem.

I understand why it is desirable to introduce a learning system into this process, as going through endless screenshots comparing them by eye is extremely tedious. Visual testing without some kind of AI augmentation seems very limited to me.

Assess if AI visual testing is for you

As I’m working on a Flutter app, and using golden tests for visual comparison (with or without AI) is a better option I think. It can be done at individual widget level which makes the comparison easier for people and also more programmatically stable. It doesn’t however visually test everything together in a page. That still needs to be done by a human. Perhaps some limited comparisons might be OK to add, but I would be extremely wary of committing wholesale to visual testing.

[mirekdlugosz (Mirek Długosz)]

Few years ago I faced a problem where I felt visual regression testing might be a right tool for the job. I created simple solution using open source tools (Selenium, ImageMagick and Python). The complete write-up is available at my blog. There was no AI involved in that.

My website displays full URL in content, and only after running a tool I realized that my “old” and “new” versions are running from different domains. This resulted in ImageMagick finding differences in cases that I was not interested in. I imagine better visual testing tool would allow me to mark certain areas as irrelevant. Maybe it could use AI to say “hey, all your screenshots are different only in this section, is this intended or would you like to ignore it?”.

Occasionally, a change somewhere near the top of website content would cause everything below to move down a bit. That resulted in huge differences, as almost entire website content was considered “changed”. Again, a better tool - maybe with help of AI - would be able to figure out that all changes are caused by elements moving in position by some constant factor. Ideally, a tool would be able to account for that and still find the differences in this part that was moved that.

Other than that, all obvious problems with my script were related to performance. A more mature tool, be it SaaS or self-hosted, would better utilize available resources.

[nao (Nao Deng)]

For today’s task, based on the goal of continuous learning of AI testing tools, I’ve chosen Option 2 to trial a new AI visual testing tool.

1. About Choosing the Tool and Checking Its Features

Having previously tried Applitools Eyes in an earlier challenge, this time I opted for a different AI visual testing tool, Percy, to learn about its capabilities.

Percy’s official introduction highlights:

• Eliminating the risk of shipping visual errors by getting visual coverage of your entire UI and confidently releasing every version.

• Rapidly performing comprehensive visual reviews by running visual tests with every commit and getting fast, deterministic results to efficiently debug.

• Cross-browser and platform rendering: Percy renders the same pages across different browsers and platforms (desktop and mobile), highlighting visual differences specific to each browser and platform. Percy captures DOM snapshots and resources, renders them, and compares them to previously generated snapshots to detect visual changes.

• Responsive differences: Percy highlights visual differences by rendering pages at configurable responsive widths and grouping visual changes while ignoring noise, facilitating faster, more accurate visual reviews.

• Snapshot stability: Percy’s proprietary snapshot stabilization technology automatically freezes animations and other dynamic elements to minimize false positives. Percy ensures the consistency and determinacy of page rendering.

However, initially, there seemed to be no mention of AI in the promotion. Soon enough, I found another official promotional piece about Percy: Introducing App Percy: An AI-driven Automated Visual Testing Platform for Native Applications.

• AI-driven visual testing: With App Percy, you can automatically detect visual defects across devices and screen sizes with every commit, ensuring your UI meets every customer’s expectations from day one. App Percy’s lightning-fast infrastructure captures screenshots from selected devices with every code push and compares them to the baseline to discover visual defects. The key here is our underlying computer vision-driven algorithm — the Percy Visual Engine — which reduces false positives, such as those caused by dynamic elements, and highlights only those visual deviations discernible to the human eye.

• Percy Visual Engine: The powerful AI algorithms of App Percy detect changes users truly care about. It aids in reducing noise and simplifying image comparisons by detecting page shifts, handling anti-aliasing noise, stabilizing intelligent texts, and ignoring regions. Learn more about the Percy Visual Engine.

2. About Testing the Statement

Since Percy is a commercial tool, I registered for a trial account.

Introduction to Using Percy

After registering and email verification, I could start a trial with a new project, similar to the steps with Applitools Eyes.

• New project setup: options for web or mobile app projects and choice of code management tools like git or GitHub.

• Selecting code writing tool.

• Obtaining Percy token.

• Setting up Percy local environment.

• Writing tests.

• Running tests.

• Viewing reports on the Percy platform.

Starting the Percy Trial

This time, I chose Percy’s cypress sdk for a demo project trial with the following steps:

• Node project initialization in a local folder via command line.

npm init

• Cypress project initialization.

npm install cypress --save-dev

• Installing Percy dependencies.

npm install --save-dev @percy/cli @percy/cypress

• Configuring Percy token.

export PERCY_TOKEN="<your token here>"

• Writing demo test code: visual testing my blog’s homepage.

import '@percy/cypress';

describe('Integration test with visual testing', function() {

it('Loads the homepage', function() {

// Load the page or perform any other interactions with the app.

cy.visit("http://localhost:1313/");

cy.percySnapshot('Login page responsive test', { widths: [768, 992, 1200] });

});

});

• Running the test.

npx percy exec -- cypress run

• Viewing command line results.

After the test runs successfully, the command line displays a link to the Percy platform for detailed visual verification results.

Test Scenario 1: Multiple Visual Difference Recognition on a Page

• Test preparation: Changes to the top content of my blog’s homepage.

• Rerunning the demo test.

• Viewing Percy recognition results.

Test Scenario 2: Small Icon Visual Difference Recognition

• Test preparation: Removing an external link icon from my blog’s homepage menu.

• Rerunning the demo test.

• Viewing Percy recognition results.

3. About Considering Failure Scenarios

Continuous recognition verification on multiple page changes was successful, with no failure scenarios encountered yet. Potential failure scenarios might include subtle color differences and minor variations in font styles.

4. About the Potential Advantages and Risks of Using AI for Visual Testing

• Potential Advantages: A professional AI visual testing tool can significantly improve project visual testing efficiency and ensure quality.

• Potential Risks and Limitations: Using an AI tool, especially for an unreleased product, comes with risks associated with data privacy, security, and result bias uncertainty. Another concern is Percy’s use of a token to upload local test data to view visual test reports on the Percy platform, posing a potential data privacy risk.

5. About the Likelihood of Adopting AI-driven Visual Testing Tools

It’s currently unlikely for the project I’m working on to adopt AI-driven visual testing tools, but I’m open to trying and learning about new AI-driven visual testing tools for personal projects.

blog post link:30 Days of AI in Testing Challenge: Day 23: Assess AI effectiveness in visual testing and discuss the advantages | Nao's Blog

[vishnuar]

Hi there

I’ve gone through this article - https://www.ericsson.com/en/blog/2022/12/visual-regression-testing-ai

The problem of pixel comparison
The pixel method applied to visual testing poses some serious challenges. It struggles with false positives and flags minute changes arising from font anti-aliasing, image rescaling, browser and graphic card rendering which are not visible to the human eye. Besides, it cannot handle dynamic content like a blinking cursor or a page where content is changed regularly. Furthermore, once the subsequent pixels differ, the reported differences can mask a real issue further down the page.

Visual AI testing uses Image and DOM comparison together and also wait for Tester confirmation for false positive cases

Benefits over traditional pixel comparison

• Ignore minor visual differences caused by shift of a few pixels during rendering on different browsers
• Recognize which elements are visual and which are not, just like a human can do, and
• Gauge which visual elements are allowed to move on a page and ignores them across all screens

Thanks
Vishnu

[parwalrahul (Rahul Parwal)]

Hello, @dianadromey and fellow learners!

Thanks for this interesting challenge. Here are my learnings for this task:

I have also video blogged my learnings here:

Visual UI Testing via AI | How does it work? | Benefits of using AI for Visual Testing -Day 23 of AI

Do share your thoughts & feedback.

Thanks,
Rahul

[manojk (Manoj Kumar Barik)]

Hello Everyone

Research and Understanding AI in Visual Testing:

Core Benefits of AI in Visual Testing:

1. Increased Efficiency: AI automates visual anomaly detection, saving time and effort.
2. Accuracy: AI algorithms excel in recognizing subtle visual differences.
3. Scalability: AI-powered systems efficiently handle large volumes of visual data.
4. Adaptability: AI models continuously improve to adapt to evolving interfaces.

Hypothetical Scenarios of AI Struggles:

1. Complex Contexts: AI may struggle with dynamic interface changes, resulting in false results.
2. Subjectivity: Human judgment may be needed for subjective anomalies.
3. Unforeseen Anomalies: AI might miss anomalies not in its training data, causing gaps in coverage.
4. Performance Issues: Processing large-scale visual data may lead to delays or resource constraints.

Potential Adoption of AI-Powered Visual Testing:

Evaluation of AI Visual Testing Solutions:

1. Example Tool: Applitools Eyes
   • Problem-solving: Applitools Eyes streamlines anomaly detection across browsers and devices.
   • Advantages:
     • Faster Testing
     • Consistent Results
     • Scalability
   • Risks:
     • Dependency on Training Data
     • False Positives/Negatives
     • Technical Complexity
     • Cost
2. Advantages and Risks:
   • Project complexity and resource availability influence adoption.
   • Alignment with testing goals and priorities is crucial.

Conclusion:

AI holds promise in streamlining visual testing but requires careful consideration of benefits and risks. Balancing automation with human judgment is key for reliable outcomes.

Thank you

[lisboalien (Aline de Farias Lisboa)]

Hi there
Another late answer, but for this one I’ll share my experience with visual testing.

I tried to implement visual testing in my project and I had some issues with that:

1. Pricing: For you to have all the advantages of visual testing you need to pay, and for a small company as the one I am working in right now it is not possible.
   I tried to implement a visual testing using a free lib, and it was great at first, but then it started to fail with anything, and it was terrible to adjust, so I just stopped
2. More tools to verify: If you are not validating only the visual of your system, you need to see the results on more than one tool. Applitools for example, you need to enter in their system to see what’s happening and if you are using cypress cloud, as I am, you need to go there too to see the other results, I don’t like having multiple systems to see what happened on my tests.
3. Selecting exactly what needs to be verified: I know that this is the mean feature of visual testing, but imagine that you have a huge e2e, your system changes with data, you need to specify exactly what the visual testing tool needs to verify for it not to fail every time. It’s easy for a few test cases, but it can be a nightmare if you have tons of TCs and you re-use functions to easy your life.

My conclusion on this is:
Visual testing can be heaven if you construct your test project involving it from the beginning and your testing logic needs to be built from its perspective, if you have to implement in a project that are there huge and healthy you can have your taste of nightmare

Note that I didn’t mention AI, that is because for me to see the AI improvements I need to overcome these barriers and the AI doesn’t seem to improve those sides that I find negative on visual testing.

See ya.

[sargam (Sargam Panwar)]

Greetings fellow Testers!
For this task, I have chosen Option 1 and the tool considered is ‘Aplitools eye’ to perform automated visual testing.
Difficulty while performing the task: No clear steps on how to integrate the tool in Selenium IDE i.e. very limited information available so far.
About the tool: The tool is very friendly. It provides options like using the tool for ‘End to End testing(Aplitools Preflight)’,’ scripting(Applitools Autonomous)', or ‘UI testing(Applitools Eyets)’.
Findings: I found multiple doc files for integrating applitools with Selenium with java but none solved the issues I was facing when importing the applitools.
So if anyone is using this tool for visual testing on ‘selenium with java’ please share blogs/videos with me.
Thanks!