Common misconceptions about test automation

However much your team uses automation to support your testing, I’m betting the road to where you are now was hard fought, and full of myths to be busted and misconceptions getting in your way.

I’d also wadger, that job isn’t finished yet, and you’ve still got more blockers to adopting automation in a healthy way that fully supports human testing and development.

So, have at it, what are your war stories? What are the blockers that remain? Maybe by posting them here, we can realise we are not alone, and share tips on how to overcome the problems we face!

Misconceptions I’ve seen recently:

  • Unit tests with mocks all the things, fast is king only
  • Unit tests and TDD mean we reduce the need for other testing (partly true… at a stretch)

So much this. I would call these high speed low realism tests. It annoyed developers who were proud of how fast their test suite was.

I got so annoyed about people using the wrong type of test on the wrong type of code and the vagueness of the advice people gave I even wrote a small guide on when to use an integration test and when to use a unit test.

In writing it I realize that a lot of misconceptions crop up because the language around automation testing is so imprecise. Half of the reason why people can’t agree on when to write a unit test is because they can’t even agree upon what a unit test is.


Truth here, all the way! What a unit is, and how to test it, are so fundamental and yet lack so much agreement and clarity. I suspect I would get slightly different answers from each Dev I work with.


What misconceptions I see from my experience

  • Automation test repository is a sacred land only QA are permitted no one else can’t contribute the test script
  • Test automation have to separated from dev repo
  • Use BDD and only believe that it make test automation write/maintain easier
  • Test automation = Every QA focus write every test every scenario in UI level

Look a bit deeper I think all four point kinda link into how people do collaborate, make the team more cross functional rather than work. on their own silos.


Funny but true:
Any bug reported in production, everyone is looking at why not captured by Automation :slight_smile:


Or, related to the one by @theology:
Automated checks are supposed to catch many bugs. If not, then automation is not worth it.
As if regression finds many bugs in all but the worst of code bases, regardless of automated or not.


Some misconceptions I’ve run across:

  • Tests that aren’t in an full staging environment (i.e. using mocks) are useless
  • Tests that require a full staging environment are bloated and too slow

I think @fullsnacktester’s comment is really insightful

  • Unit tests with mocks all the things, fast is king only

I can see why you’d want unit tests to be fast if you run them on every PR or build. But at the same time you need those slow tests too, and maybe they just need to be run at a different phase of the dev/release cycle.


It’s possible to automate all the things

It’s desirable to automate all the things

Automation can completely replace manual and exploratory testing


Code coverage tells us the quality of our software.

I recently made a pitch to stop including these numbers in a review meeting where people were showing how they went from 78.3% to 78.9% coverage last sprint. That misses the fact that the tests intermittently fail. That misses the fact that the most brittle and high risk areas have sod all automation. When ranting with a colleague he told me that his previous work had a requirement to have 95% coverage before release, so folk wrote tests that didn’t even assert anything.

Also one I saw was a job advert for a Test Engineer who can perform TDD. That would be… a developer.


So many trigger phrases in so few words. This is great :laughing:


The two most common misconceptions I’ve encountered are these…
1: We can automate everything. The truth is that not all tests or testing types are good candidates for automation.
2: Automation equals CI/CD. It just doesn’t… Automation is a key piece of that process, but it absolutely doesn’t take you from your current process to CI/CD by itself.

Edited to add a third item…
3: Once automated tests are written they’re good to go! So many teams and companies don’t factor in that as their product grows and changes, the automation will need to be maintained and expanded to match it.

1 Like

I love these!

Automation will replace manual testing. cough