Why is my cypress-axe test behaving unpredictably?

🙋 Questions
,
1

I have a Cypress test script using the cypress-axe library to run an accessibility test on any given web page to check it for WCAG 2.2 AAA rule violations. It is fairly straightforward. And it worked perfectly good only 4 days ago. Today however, it is passing my tests even though it should not.

describe("Test accessibility for different WCAG levels", () => {
  const page = "whateverurlyouwant";

  it("should run accessibility audits for WCAG 2.2 AAA", () => {
    cy.visit(page);
    cy.injectAxe();
    cy.checkA11y(
      null,
      {
        runOnly: {
          type: "tag",
          values: ["wcag22aaa"],
        },
      },
      (violations) => {
        cy.writeFile(
          "./cypress/a11y-reports/a11y-AAA.json",
          violations,
          "utf8"
        );
      }
    );
  });
});

Test case: I run this test against a webpage that has a known issue and fails WCAG 1.4.6 which is a AAA rule.

Expected result: the issue is reported in my test results file. This is what i experienced last Thursday.

Actual result today: the test passes successfully, no a11y problems found.

Things I have tried to remedy the situation:

  • put in a wait statement. just in case it was due to the page not being loaded completely. but no luck.
  • told cypress to click a button further down the page which reveals more text. this was to make sure that the page was scrolled down enough to reveal the text block that is causing the contrast fail. also did not help.

My question is: Is there anything wrong with my test that I could improve to make the results more reliable? And actually, I also wonder if this even is a case of flaky test results or something else. I have in fact contacted Deque to check if they have changed anything their end. But since between my successful test and today lies mostly a weekend, I think that’s unlikely.

3 Likes
2

In the code above: values: [“wcag22aaa”],
I don’t see the value here in Axe-core Tags:

Watch this for compatibility discusions:

This is another tool using the same API/library Axe. They are talking about:
case ‘WCAG22AAA’:
return {
type: ‘tags’,
values: [‘wcag2a’, ‘wcag21a’, ‘wcag2aa’, ‘wcag21aa’, ‘wcag22aa’, ‘wcag2aaa’, ‘best-practice’]
};

So if you want 22 AAA you could do similarly using cypress-axe.
values: [‘wcag2a’, ‘wcag21a’, ‘wcag2aa’, ‘wcag21aa’, ‘wcag22aa’, ‘wcag2aaa’, ‘best-practice’]

1 Like
3

Thank you so much Stefan!

I have come to the same conclusion, it must have to do with the fact that wcag22aaa is not listed in the axe api docs. I have found it in these docs but I think that maybe these are for a paid aspect of deque devtools? I actually struggle finding out exactly what are the free and the paid-for features on their site. Or even how much they charge for which aspects of their products. So I just don’t know. But it seems the most likely explanation: In the few days between me writing my test and running it successfully, and me trying to demo it, the axe team changed something. I would have prefered the change breaks my test. Because right now, it looks like the test is running and it does not find any issues. Which is misleading to say the least. Fortunately I tested my test :laughing:

Might look into pa11y! I know it was used in my company in the past so need to find out why they stopped. Thanks for the link!

1 Like
4

Going by their git code for the axe-core library used by cypress:

else if (rule.tags.find(tag => tag.includes(‘aaa’))) {
result = descriptions.wcag2aaa.rules;

1 Like