CAPTCHAs in automation

A while ago Ministry of Testing had their online #TestChat around automation. One of the questions asked was:

Q2. What are your biggest struggles in automation?

This reply got me thinking:
https://twitter.com/ir_emery/status/908048745983492096

As some discovered in 30 Days of Accessibility Testing Challenge, I’m not a big fan of some of the CAPTCHAs I’ve seen.

How does one go about handling CAPTCHAs in automation? Do you have to speak really nicely to a developer to turn them off for automation? Is there a tidier way to handle them? Do you think they need to be handled?

Check out https://developers.google.com/recaptcha/docs/faq
This explains how to get rid of validation while using the still having inputs.

2 Likes

First thing is that CAPTCHAs were designed so people could not use automation on the site. So if there was something you could do from your automation to disable the CAPTCHA or get around it then people who the site wants to keep out could use the same method to bypass the CAPTCHA.

A classic example of a safeguard with a bypass was all Microsoft products. In order to prevent people from stealing Microsoft products you have to enter a product key. Twenty years ago this was a big problem for testers at Microsoft. They would have to generate product keys every time they wanted to test the product. So Microsoft changed the code. You had to enter the product key OR 11111-11111111-11111. If you just held down the 1-key until it filled the input then pressed validate, it worked. Now they could have taken out this hack after they finished tested but by QA standards, if you modify the code, you have to retest it. Microsoft decided to just leave the backdoor in.

After a while word got out and soon everyone knew that enter 11111-11111111-11111 would unlock all Microsoft products. Now Microsoft products require a real product key. :frowning:

Same sort of thing for CAPTCHAs. If there was a secret bypass that always bypass the CAPTCHA then word would get out and spam robots would soon be programmed to use the secret bypass.

The best solution I have found is that the developers could put in a piece of code that checks an environment variable or system configuration. If the variable is set then CAPTCHA gets bypassed. During testing, you would deploy the binary, configure the variable to be unset, confirm that CAPTCHA works as expected. Set the variable, confirm that CAPTCHA is bypassed. Do all your testing without worrying about the CAPTCHA.

As part of the release testing, you want to make sure all deployments to production have the variable unset so CAPTCHA will be enabled.

This way you don’t have a new binary and don’t need to test everything all over again.

Darrell

6 Likes

exactly this :smile: we have the property for captcha enabled / disabled via the platform rest api on our test networks (endpoints only visible on internal networks) so we can have a setup step for tests that go through this to ensure its disabled

We do this for our test environment, works perfectly.

So I need to speak really nicely to developers and/or arrive with sweets :wink:

1 Like

Do CAPTCHAs work at the API level too? I’ve never had to use one for websites I’ve worked on. Would it be possible to do much of the functional testing at a different level? Like API or DOM? Bypass is good too, but I’m thinking about testing in stage or prod where those features can’t be turned off as easily.

1 Like

Here is my take on this.

I agree with darrell.grainger on the reason of Captchas => To disable automation.

Reason for automated testing => minimizing human intervention during tests to save cost.

In reality, having a configuration-setting to turn off captchas in test-environments will probably work most of the time… Unfortunatly if there is a problem with the test-environment-setting, this may not be clear during testing…

Thus introducing such a configuration-setting, displaces the test from QA-testers to developers, since if you want to be sure that the setting still works after a change to the initial webform, there will have to be some sort of manual testing.

Just consider what will cost more, manual test by developer (of the setting) OR manual test by tester (of the webform)

There are some advanced things that you can do with captchas (for example, using OCRs) but if you only have 1 or a few forms with captcha, it’s probably best to stick with manual testing.

1 Like

I have a few issues with Captchas as a general user so I guess my feelings towards them do come out above. A computer could definitely crack most Captchas quicker than me! I also don’t think they’re terribly mobile device friendly but it’s just my personal opinion.

Going by the replies above, I’m wondering if an answer to the “What are your biggest struggles in automation?” question (given this particular tweet) might be: getting someone or learning how to turn off Captchas before running the automation.