Why is my cypress-axe test behaving unpredictably?

🙋 Questions
,
1

I have a Cypress test script using the cypress-axe library to run an accessibility test on any given web page to check it for WCAG 2.2 AAA rule violations. It is fairly straightforward. And it worked perfectly good only 4 days ago. Today however, it is passing my tests even though it should not.

describe("Test accessibility for different WCAG levels", () => {
  const page = "whateverurlyouwant";

  it("should run accessibility audits for WCAG 2.2 AAA", () => {
    cy.visit(page);
    cy.injectAxe();
    cy.checkA11y(
      null,
      {
        runOnly: {
          type: "tag",
          values: ["wcag22aaa"],
        },
      },
      (violations) => {
        cy.writeFile(
          "./cypress/a11y-reports/a11y-AAA.json",
          violations,
          "utf8"
        );
      }
    );
  });
});

Test case: I run this test against a webpage that has a known issue and fails WCAG 1.4.6 which is a AAA rule.

Expected result: the issue is reported in my test results file. This is what i experienced last Thursday.

Actual result today: the test passes successfully, no a11y problems found.

Things I have tried to remedy the situation:

  • put in a wait statement. just in case it was due to the page not being loaded completely. but no luck.
  • told cypress to click a button further down the page which reveals more text. this was to make sure that the page was scrolled down enough to reveal the text block that is causing the contrast fail. also did not help.

My question is: Is there anything wrong with my test that I could improve to make the results more reliable? And actually, I also wonder if this even is a case of flaky test results or something else. I have in fact contacted Deque to check if they have changed anything their end. But since between my successful test and today lies mostly a weekend, I think that’s unlikely.

3 Likes
2

In the code above: values: [“wcag22aaa”],
I don’t see the value here in Axe-core Tags:

Watch this for compatibility discusions:

This is another tool using the same API/library Axe. They are talking about:
case ‘WCAG22AAA’:
return {
type: ‘tags’,
values: [‘wcag2a’, ‘wcag21a’, ‘wcag2aa’, ‘wcag21aa’, ‘wcag22aa’, ‘wcag2aaa’, ‘best-practice’]
};

So if you want 22 AAA you could do similarly using cypress-axe.
values: [‘wcag2a’, ‘wcag21a’, ‘wcag2aa’, ‘wcag21aa’, ‘wcag22aa’, ‘wcag2aaa’, ‘best-practice’]

1 Like
3

Thank you so much Stefan!

I have come to the same conclusion, it must have to do with the fact that wcag22aaa is not listed in the axe api docs. I have found it in these docs but I think that maybe these are for a paid aspect of deque devtools? I actually struggle finding out exactly what are the free and the paid-for features on their site. Or even how much they charge for which aspects of their products. So I just don’t know. But it seems the most likely explanation: In the few days between me writing my test and running it successfully, and me trying to demo it, the axe team changed something. I would have prefered the change breaks my test. Because right now, it looks like the test is running and it does not find any issues. Which is misleading to say the least. Fortunately I tested my test :laughing:

Might look into pa11y! I know it was used in my company in the past so need to find out why they stopped. Thanks for the link!

1 Like
4

Going by their git code for the axe-core library used by cypress:

else if (rule.tags.find(tag => tag.includes(‘aaa’))) {
result = descriptions.wcag2aaa.rules;

1 Like
5

Hi Scout,
This response might be a little late, but thought to do it in case could be of help to you or other fellow testers.

Pa11y actually use underneath htmp_codesniffer and optionally axe-core for it’s accessibility analysis.

In fact when you check the source code of pa11y plugin, you enable the option to use axe-core, it the internal code maps wcag2aaa to wcag2aa (for axe-core).

Regarding html_codesniffer, according to it it should do the job for wcag2aaa (I have not checked personally so I cannot confirm this). However if you check the html_codesniffer repo the last update was 4 years ago, so seems it’s not really being much maintained).
On the other hand axe-core is updated very often.

According to axe-core package documentation:
“Axe-core has different types of rules, for WCAG 2.0, 2.1, 2.2 on level A, AA and AAA as well as a number of best practices that help you identify common accessibility practices […]”.
So it seems that should support AAA, so adding to the list as proposed in a previous comment in this thread it’s suppose to work (I have not checked myself though).

If you are using specifically Cypress, you may want to consider to have a look to an open-source Cypress Accessibility plugin I released last month, called wick-11y .

This is a blog I published about Accessibility in general how the wick-a11 plugin can help you addressing it:

It uses underneath axe-core and cypress-axe (what ever the latest axe-core package at the moment of the install), and provide off-the-shelf detailed reports of the accessibility violations in the Cypress log, in the console, flag in the web page graphically all the violations in different colors based on severity allowing the tester to interact graphically with the violations and details about how to fix them, and optionally generates a detailed html report with a screenshot of the violations also based on severity.

It’s pretty convenient and installation and configuration is very straightforward.

Hope this might be of help!
Cheers