Hi my fellow testers,
I’ve struggled with finding a new tool for this challenge so I have fallen back on an AI tool I have used previously, which is Applitools Eyes Automated Visual Testing & Monitoring with AI - Applitools Eyes as this uses AI its in detection of issues
Evaluate the Reporting Quality: Assess the accuracy, completeness and quality of the bug reports generated by AI: Are the bugs identified by the AI valid issues?
This was a mixed bag as they could have potentially been issues, they were reporting on differences within the screenshots but in the specific examples I tested it on, they weren’t actually bugs but timing issues so the state at the time of the screenshot was slightly different which caused different text to be on screen. If this wasn’t a timing issue though it could have genuinely been a bug
Are the AI-generated reports detailed, clear and actionable enough?
Yes the screenshots it generates are really clear as they display & highlight the differences between its baseline image & its test image
How does the quality of information compare to manually created bug reports?
I would say screenshot wise they are better than I could produce manually but I would need to add a comment or some explanatory text as well to the bug report
Identify Risks and Limitations: Reflect on the potential risks associated with automating bug reporting with AI: False Positives: How likely is the AI to flag non-existent issues?
Depending upon the match level the tool has been set at the potential for false positives is high as it could detect pixel level differences that aren’t bugs and that a human would never notice
False Negatives: Can the AI miss critical bugs altogether?
I’ve not found this in my experience so far, any changes present have been detected even if they haven’t turned out to be real issues in the end
Data Anonymisation: Is your data anonymised before being used by the AI?
I don’t think the screenshots are anonymised in anyway
Data Security: How is your data secured within the tool?
They state here that they are protected by SSL and HTTPS https://help.applitools.com/hc/en-us/articles/360006914892-Applitools-Eyes-Security-Guarantees
Data Ownership: Who owns the data collected by the AI tool?
I would hope that I still own my screenshots but I can’t find any specific information on that