Hey,
I was curious if there are other people like me who are really bothered by the state of flaky selectors in the current web browser world.
Sometimes a selector will work for a session, a week, a month and then suddenly it will fail.
This can be attributed to many reasons: The flakiness happens because tests become dependent on the current state of the code, not the visual representation of the webpage.
Imagine a test looking for a button with the ID “submit-button”. If a process in the code “optimization” renames the ID to “csj9323h-btn”, the selector breaks even though the button itself remains visually unchanged.
so I am working on a library to fight back:
- This uses vision: It uses “AI vision” models to determine the state of an application without having to use CSS or HTML selectors. Just describe what part of the page you want tested and what you want tested and it will give you a pass or a fail
- Requires OpenAI Key: Right now the best vision model that can be accessed quickly is provided by OpenAi
- Can be used to test other things: Accessibility, application state, complicated UIs (browser games)
Here’s an example:
const { test, expect } = require("@playwright/test");
const goodlooks = require("goodlooks");
goodlooks.configure("an_APIKEY_for_OpenAI");
expect.extend(goodlooks);
// Let's write our test which takes a webpage and validates that the web player will not autoplay after the page has loaded.
test("rickroll", async ({ page }) => {
await page.goto("https://www.youtube.com/watch?v=dQw4w9WgXcQ");
await expect(page).goodlooks("the video on the website is not playing");
});
As you can see all it takes is to add this part to a typical Playwright test:
.goodlooks("Anything you want the AI to verify for you")
Output:
âś… TEST PASSED.
The page shows a video with the play button available
and a timeline that is not progressing,
indicating that the video is currently not playing.
This could be asking the AI whether it sees thumbnails on YouTube, whether there’s a picture of a cat on the page, whether hovering a menu has produced a specific change to the page, accessibility info on the visual state of the elements etc etc.
Was curious what you thought of this idea and whether it would benefit the community?
Or have you found other ways to achieve this?