Power Hour - Accessibility Testing

Hi Julianne,

So this is one of the biggest challenges I’ve seen around Accessibility. I’d be lying if I said I’ve always been clear on the value of accessibility myself. It’s interesting how life guide’s us and my engagement in/increased awareness of the importance of accessibility is in a big way down to my son being Autistic. Seeing how efficient he is with technology to then come across challenges because the app/site/product he’s using isn’t as accessible as it could be. As I mentioned in the answer to Oli, the three factors (and I picked these up from James Sheasby Thomas) I tend to focus on when trying to convince people of the importance of accessibility testing are – Legal Requirement (Equality Act 2010), Morality – it’s the right thing to do, market share – greater potential customer base, competitive advantage – a better user experience therefore increased user satisfaction.

The important thing here is that you need to know your users to be able to grasp how important Accessibility Testing is to them, and if it’s important to the users then that makes the sell to higher ups a little easier.

Again without knowing how you are working currently, I’ve always been a big believer in testing being present at every stage of the development process, in which scenario, if you can identify areas of concern around accessibility based on previous releases you can raise those concerns early and the overall cost of accessibility is reduced. Compare that to the cost of a lawsuit or finding fundamental accessibility issues in a product just before release and the cost saving is quite big.

Interestingly the government website has a plethora of information on both the importance of Accessibility Testing and involvement of the team in accessibility.

https://www.gov.uk/service-manual/helping-people-to-use-your-service/making-your-service-accessible-an-introduction#accessibility-is-the-whole-teams-job

2 Likes

Hello Ben. When you say resources do you mean places to practice? If yes then literally anywhere on web, app or mobile you can put those skills to use. I contact site owners if I find accessibility issues, some are more responsive than others! You can also google ‘bad websites’ and there’s all sorts of terrible ones highlighted to play on, errrr I mean professionally practice your craft, ahem…
I suppose one of the most famous bad websites in https://www.lingscars.com/ as an example of what not to do!

2 Likes

I’ve seen teams take a copy of the products they develop and request changes to actually build accessibility issues into the product in order to validate that accessibility tests and tooling is performing as expected. The bonus value from this is that it built awareness with the development team around accessibility issues so they could be understood and considered early on.

Otherwise I know there are sites built to have accessibility issues to allow people to practice accessibility testing against. A couple of examples are:

https://www.washington.edu/accesscomputing/AU/before.html

https://alphagov.github.io/accessibility-tool-audit/test-cases.html

2 Likes

I’m sure there are a few here too but my WiFi has crashed

2 Likes

Thank you for your questions Pekka. There are some things you can add to a CI pipeline, for example we have checks on commit related to a skip to all content being present, the ability to tab to all elements and missing alt-text. From what I’ve read automation can only cover up to 40% of the guidelines and not always completely so there will be things you can do. I’d be very interested to hear how you get on.

From my experience and reading, the most common pitfalls of failing accessibility are often the simplest to fix. Missing or badly described alternative text for images. Contradicting labels which confuse screen reader users. Tabbing and keyboard traps. Poor contrast and small text size. Overly complicated language use. ‘Click here’ links rather than something clear and descriptive.

3 Likes

Hi,
Accessibility is no different than any other testing with regard to the fact that the quicker the feedback loop the better. So as long as you don’t treat these checks as the full solution to accessibility challenges and understand what value they can give you before you implement then they are definitely worth it. Again the gov website has an interesting Accessibility tool audit which I found interesting:

https://alphagov.github.io/accessibility-tool-audit/index.html

With regard to pitfalls I don’t profess to have done accessibility testing against enough products to really identify pervasiveness with any confidence however based on two things, accessibility testing I’ve been involved in and seeing my son (who has Autism) use software, I would say the following:

  • Clarity of controls and links etc – often a lot of effort goes into creating a design that looks slick and in my experience this can often result in clear buttons, links, input boxes etc being sacrificed for a ‘cool design’
  • Lack of keyboard accessibility – Having areas of the software including buttons or fields that just can’t be accessed through the keyboard.
  • Lack of or poorly named alt attributes – obviously causes issues for screen readers
  • Lack of consistency of design and behaviour – My son can’t read but he often figures his way through software and websites using some degree of his own intuition/trial and error. Where he tries to use things in a similar way in various points in a product/site or across sites and the behaviour is different or the design in inconsistent, it can result in violation of expectation and in a worst case scenario a melt down.
2 Likes

Hey great question. That’s an interesting challenge you have their and I’m not going to profess to have any experience with them however I am fascinated as online ads can be an accessibility nightmare for my son in their own right.

With regards to question 1 - How do you typically test them, do you already have or is there an option for some kind of test harness (non frame host) to be developed so you could test the ad in isolation of any page it may appear in? Is testing the iframe itself for accessibility relevant in your context (do readers pick them up as ads for example) or is it just the content that is within your context? Are they cross domain iframes? You’ve probably already looked into them but could Bookmarklets provide any value?

http://pauljadam.com/bookmarklets/index.html

For question 2 I can definitely give an opinion. So whether it’s advertising or otherwise the clarity of information being provided is important in accessibility but there isn’t a clear right and wrong as, for example, autism is a spectrum and therefore the challenges can be diverse. If I take my son as an example, typically he has a short attention span so if it’s not clear or doesn’t grab his attention he will move on however if you get his attention he may become quite engaged. He can’t read so if the message is clear through some kind of visual representation then he is more likely to pick up on it.

That said it’s a fine line because he is very focused and builds expectations of what he is expecting to see/do on a given site. He’s more likely to leave a subtle add on a page however an add that pops up, has sound or in any other way gets in the way of what he’s focused on, causes a violation of his expectations and worst case scenario we end up with a meltdown, best case he puts his ipad down and walks away from it. As I mentioned in my answer to Gem, I find understanding your emotional response to something important in considering accessibility – does it shock you, annoy you, bore you, frustrate you etc. Then consider how you can compare that emotional response to the emotional response of others, ideally a diverse group including disabilities and neurodiverse people.

2 Likes

Hi Bruce. One of the main reasons for agreeing to do this was to push myself and hopefully learn too.
So this is a brilliant question. But… The honest answer to 1 is I’ve no idea! This isn’t something I’ve any experience of or with but Twitter and The Club might be good places to reach out. The only thing I’ve come across on this is discussion on the frames having titles. This article on WebAIM gives some practical advice, https://webaim.org/techniques/frames/. I’d also experiment with keyboard only use, a screen reader and if applicable, accessible mode on mobiles.

For 2 I’ll refer to an earlier answer and my accessibility quadrants. Readability is important and one of the simplest ways of checking is to copy text into a Word document and spell check with ‘Show readability statistics’ enabled in options. (File – Options – Proofing). When spell check is done it pops up counts, averages and readability details. The three sections are % of passive sentences, Flesch Reading Ease and Fesch-Kincaid Grade level. There are other online tools of various quality available too.
I’d definitely be interested in reading how you get on, obviously in your own inimitable style! :smile:

2 Likes

Hi Heather and thanks for having us at the club.

I’ve never been involved in designing persona’s specifically for accessibility testing but I’ve worked with a UX expert in the past to build persona’s to better define the user journey and this overlapped into accessibility. Obviously this will very much depend on your product and the potential target audience however I’d be tempted to look online for some examples first before reinventing any wheels, for example the government website has two fairly cool pages, one with interviews with people who are affected by poor accessibility (https://accessibility.blog.gov.uk/category/accessibility-and-me/) and also some ready made persona’s/profiles tied into accessibility (https://www.gov.uk/government/publications/understanding-disabilities-and-impairments-user-profiles)

It’s difficult to really recreate the true extent of someone’s accessibility issues because of the diverse nature of disabilities however in a team I worked in previously, we actually blind folded one of the testers and setup a screen reader against the site to see if he could navigate the site whilst blindfolded. Unfortunately, it’s harder to really recreate the challenges that an autistic person for example might have. I’ve mentioned in a couple of earlier questions building emotional maps of the software/site ideally from as diverse a group as possible.
To this end I would always encourage companies to look at the diversity of their teams, if your teams aren’t inclusive then you will struggle to build an inclusive product. Peoples biases are affected by their challenges and their input into the product will include their own biases. You could look at outsourcing that accessibility testing but ultimately the responsibility for the accessibility of the site sits with the development team (product, dev and test).

2 Likes

Hello Heather, thanks for your questions. I’m going to try to answer them together if that’s ok? There’s a couple of ways in which I’ve included accessible traits into personas and my testing in general. They come at this from a couple of different directions. We have existing personas that relate to different mortgage ‘states’ such as in arrears, has an arrangement to pay back arrears etc. I test as those but add an additional trait such as, keyboard user, screen reader user or some other trait. One of the tools I use is called Funkify a disability simulator available as a Chrome extension. Details can be found at https://www.funkify.org/. So I’ll enable a trait and test in that mode.

Another way is to start with an accessible persona and test as them. There is a great alphabet of accessibility issues that can also be used as personas on The Pastry Box Project at https://the-pastry-box-project.net/anne-gibson/2014-july-31. There’s also a gov.uk set that can be found at https://www.gov.uk/government/publications/understanding-disabilities-and-impairments-user-profiles. Let us know how you get on and good luck.

2 Likes

Wow Paul, I think potentially there might be enough there for the answers to fill a book, or possibly two! There’s certainly not enough time in an hour to really get into all these. Hopefully I can say some things that may help.

I will say that finding industry standard tools may be problematic and I’d suggest thinking about what you want the tools to do first, then find the ones that best match. Several tools have been brought up in the answers so far that are popular and very useful.

I have no experience or knowledge on an SDLC (Software Development Life Cycle) specifically for a team performing accessibility testing, or testing of any kind. My experience is with an SDLC for development that mentions something like, ‘testing techniques will be employed through the development processes. It could also reference approach or strategy documents, which might suit your purposes more closely.

It may well also be worth thinking about what’s most important first and making some decisions. Those may well inform the questions that follow nullifying any quick decisions now.

2 Likes

Wow lots to get through here Paul so bear with me, I’ll try my best.

As far as your question with regards to industry standard tooling, I’ve never really bought into the idea of industry standard tooling. There are different tools that do different things well and which tools are best for you will depend on how you work, your tech stack, what you are trying to achieve etc. What you need to look for and ideally trial is which the tooling supports and adds value to your approach/strategy.

I’ve used manual tools to support my accessibility testing, things like colour contrast checker (https://contrast-ratio.com/as recommended by James Sheasby Thomas) and ChromeVox screen reader. I’ve never used it and I’m not sure how much value you would get from it unless you are working in a domain requiring a lot of compliance analysis and auditing, but there is an accessibility test management type tool - Worldspace suite from Deque systems.

With regards to Accessibility testing automation, the tools I’ve used most are WAVE and ASLint, of which I’d probably say I prefer WAVE but that may just be because I used it for longer and ‘got used to it’. I’ve not used aXe but I’ve heard positive feedback on it as a tool, mainly from developers who have used it. If you haven’t already seen it the gov.uk team did some pretty comprehensive analysis of accessibility testing tools at:

https://alphagov.github.io/accessibility-tool-audit/

As far as the SDLC is concerned, again it depends on the team, the current approach, tools/skills and knowledge. As with testing in general in my opinion the ideal strategy is for testing to be included from analysis/design right through to release and then continued analysis against live. I’d want to see Accessibility considered in story analysis and in defining the technical solution. Where accessibility is called out in story analysis prior to development it can be treated as any other required behaviour identified in the story and tested accordingly including the development team building it into any integration tests run on build. Including accessibility in any continuous integration/continuous delivery pipeline helps to keep focus on accessibility and identify issues arising against existing product.

I’ve worked in teams where tests were built by developers in aXe for accessibility testing and then testers picked up ‘manual’ accessibility testing at a story and a release/regression level and teams where testers picked up all of the accessibility testing retrospectively before release as part of regression testing and the gaps identified are those I’ve already mentioned. Build an accessible product rather than checking if it’s accessible after building and you will reduce the pain and cost of accessibility.

Hopefully I’ve touched on most if not all of your points. Happy to discuss further if there is anything not clear or not answered.

2 Likes

Hi Dan.
I’ll start by pointing you at the gov accessibility project and the published results. Whilst it might not be 100% up to date, the link below shows a breakdown of accessibility issues intentionally included in a test site and which tools caught which issues.

https://alphagov.github.io/accessibility-tool-audit/results.html

It’s interesting looking through the issues that no tools picked up. There is an interesting array of issues many of which are visual and I think fairly likely to be picked up through completing manual accessibility checks.

The ones that I think are important (and that may be based to some degree by my own biases) because they are potentially too difficult to identify/understand are around Neurodiversity. Things like consistency of behaviour throughout the site, clarity and organisation of content, avoidance of excessive stimulation like visual noise. Making sure non-neurotypical people don’t feel uncomfortable or unable to use the site because there are too many violations of expectations.

2 Likes

Hello Dan, firstly I really hope ‘if accessibility testing gets done’ isn’t even coming into the equation!
I covered this (hopefully) in my answer to Pekka regarding the most common pitfalls of failing accessibility. Some of those can be automated to some extent but things like consistency, tabbing orders etc. need a human assessment. In terms of priority I’d say the top two are keyboard only use and supporting assistive technologies (screen readers etc.). Ease of use, language and contrast following those. From the updated Web Content Accessibility Guidelines 2.1 it covers third party tools so it is really not getting in the way of or restricting them.

2 Likes

Great question and screen name Professor! Particularly over the last few years, accessibility developer tools have improved in browsers as has the ability to provide accessible mobile devices. Progressive web applications have meant, in some ways, the best of both worlds. HTML5 is a step towards accessibility as standard in the way we build online services.

We are making progress; we just need to keep pushing the will for everyone to consider accessibility. It is something we should be doing and legally it is moving towards having to do this or be punished.

There’s a lot of discussion about the future of testing being synonymous with Artificial Intelligence and Machine Learning (AI and ML) and they could definitely play a big part. But I believe the future of testing is definitely tied to accessibility testing not only as a fundamental requirement but as a basic human right.

As an example of the way accessible software can improve peoples lives take a look (and everyone else reading can too :blush:) at this buzzfeed.com piece and see how technology is helping make people’s lives better, https://www.buzzfeed.com/watch/video/35480 I’d highly recommend a look.

2 Likes

Hi,
This is a great question if you don’t mind me saying.
As with much of this context is important. Touch centricity introduces an abundance of accessibility issues across the accessibility spectrum especially smaller touch devices. That said however I think this in turn has pushed providers of touch devices and apps to come up with solutions to those accessibility issues.
You only have to look through the Accessibility options on the iPhone/iPad for example to see the emphasis accessibility has in modern touch devices.

Interestingly I think arguably the proof is in the pudding. The uptake of touch devices is a reflection of how they generally are opening up the market. Would they be doing this without some reasonable level of accessibility. Certainly for neurodiverse accessibility I can tell you my autistic son finds touch devices easier to use than pretty much anything else.

2 Likes

Thank you, everyone, for all your brilliant questions and patience with me and @ruarig

Thank you to the Ministry of Testing for inviting us and to @heather_reid and @mcgovernaine for your support.

I really enjoyed this experience, learned quite a lot and as I said on Twitter, I feel like I’ve kicked a little ‘impostor syndrome’ butt along the way. If you have any follow up questions or want to ask something more directly please feel free to contact me or post here.

Thanks, Ady

Ps, I’ve started posting more on accessibility on my website at https://www.thebigtesttheory.com/ and this is where the free guidelines with test hints are. Please feel free to visit or even download the word version so you can add your own notes in your context. My site is ad-free so you won’t be hassled by PPI and such :wink: so I’m not trying to get you to go to make money. Plus you can read testing poetry while there, what more could you ask for?
Slides from my accessibility talk mentioned earlier are available at https://www.slideshare.net/adystokes/edit_my_uploads

3 Likes

Similarly I’ve had a blast. Thanks to Ministry of Testing for the invite and for Ady for sharing this whole experience with me.
I’m more than happy to follow up with wider discussions on any of the questions.

I’m currently spending some time promoting and raising awareness for Autistic Accessibility. Take a peek at my blog post if you want to know more, there will be more coming soon:

I’ve also started a group on LinkedIn to discuss Autistic Accessibility:
https://www.linkedin.com/groups/13703932

Thanks again,
Matt

3 Likes

In a related way this post is about having an inaccessible site that tools think is fine. Well worth the read.

3 Likes

I’m late on this thread, but just wanted to say the ‘most inaccessible site possible with perfect Lighthouse score’ article is very interesting… Thanks

2 Likes