Why will testing never just be automation?

Archive
1

There has been some discussion lately that testing is eventually going to just become entirely automated. This means as testers we will become automated developers…not sure that’s the case. So we decided to ask you! Unsurprisingly we got lots of replies.

Because automation tells you all the unknown things about what you already know, manual tells you all the unknowns about what you don’t know. You need to find those unknown unknowns.

Because a lot of software will always be for humans, and it’s cost effective to make the first humans to use it be your employees rather than your customers.

Because you cannot automate everything.

Once people realize what is testing and what automation can do. There’s the human part in testing, automation just cannot mimic… Every system has two sides 1) engineering view point 2) better solution for human …

I agree on that.
Automation is a tool which needs to be used wisely because otherwise, it’s costs explodes (which will be not be paid at a certain point and so the tool not used any more).

Because life is complex and emerging, and automation does not like emergence that much. Cooling things down makes them suitable for automation. What to do before they’re cool enough for automation?

Apparently, this seems to depend on where you work… Seems that automate everything is still very much alive in certain places, making automation ‘testing’.

Automation is a test tool, not testing itself. You COULD have automation as the sole testing tool, doesn’t mean you SHOULD. If an application is designed for humans then it should have at least some testing done by humans.

If we’re talking test automation then we’re actually meaning automated checks. Testing is about exploration and uncovering information, for that you need testers and our natural sense of curiosity and urge to press the big red button and see what happens.

  1. Sometimes you don’t know what you’re looking for until you see it. Heuristics are doable, but serendipity is trickier.
  2. Sometimes precision isn’t desirable. Things you don’t even know that you’re doing may reveal problems.
  3. Sometimes it’s just not worth the effort. Some things are very important to test once, but pointless beyond that.
  4. Testing is literal magic and no one knows how it happens, so it can’t be replicated. If you can do automation, you are a programmer and by definition unable to test.

I don’t totally agree. It should be just automation. Automate all the things! If I find a test case while making exploration testing that is not yet automated I automate it.

Initial, random and exploratory, UX design tests cannot be automated.

I found automation isn’t always realistic in some test cases it has to be done manually for it to succeed automation is the dream but tech wise not always possible or even better not always needed.

Cos machines are incapable of opinions. #a11y

What are you thoughts on all this? Do you think it will become more dominant?

1 Like
What should not be automated?
2

There are always going to be those who want to replace human testers with coded checks. Humans are expensive and “unreliable”. Computers, once the effort has been made to build the automation, are cheap.

This belief, of course, completely overlooks the fact that paying humans to perform pre-written checks with no allowance for deviation is expensive and unreliable because humans aren’t the best at doing the same thing the same way over and over again. That isn’t our strength - our strength is seeing patterns and exploring and figuring things out.

The last I checked, AI is not up to deciding whether a user interface is counterintuitive or kludgy. A human will pick this up immediately.

Then there’s the whole question of whether or not the effort to build test automation is worthwhile. Take a print function. It’s relatively easy for a human to check that what’s on the screen matches what’s on the paper. It’s a whole lot more challenging for a computer to perform that check, and inherently less accurate. The same applies to many other devices - I’ve tested software that interacted with multiple specialized hardware devices including but not limited to turnstiles, magnetic card readers, RFID readers, PINpads, magnetic card printers, receipt printers, security cameras… The thought of trying to automate those interactions is enough to give anyone hives.

Something similar applies to third party interfaces. Sure it’s possible to automate the interaction with the interface API, but that doesn’t necessarily cover everything. The payment gateway testing I’ve done, I’ve encountered situations where the test gateway has bugs that don’t exist in the live gateway (and, of course, the test card stock was not valid to use in the live gateway. Made bug fixes interesting). Simulators are usually even further out of sync with the live gateway. On top of that, the availability of a payment gateway is completely out of the tester’s control, since it’s the responsibility of the payment provider.

Testing is as much evaluating fitness for use of an application as it is evaluating the functionality. Automated testing can help with the latter, but it’s currently not able to do anything to evaluate the former. AI would have to be much more advanced than it is now to change that fact.

So no, testing will never just be automation.

3 Likes
3

I really dislike this segregation of Manual Testers/Automated Testers.

It bothers me no end in the workplace to have to discuss the situation with management (not to mention the difficulty it causes with trying to get testers that have no automation experience on side for introducing automation).

As far as I’m concerned there is “Testing”.

Whether it’s manual or automated is a toolset to support testing. Manual is great for somethings (as Kate covers in more detail above) and Automation is good for others (again covered well by Kate so no need for me to expand).

I think what testers (and more importantly testers with automation skills) should be concentrating on is educating others within the development industry that there is no such thing as Manual Testers and Automation Testers, Automation Testers are just manual testers that haven’t added automation to their skill-set yet.

And vendors need to stop advertising their products to upper management as a solution for removing all manual testing overheads, because it’s just not going to happen and it creates false impressions that again need to be managed by the testers.

2 Likes
4
1 Like
5

As already stated. This manual versus automation needs to stop. Similarly to that we do not talk about development automation as an activity that would be instead of developers we should shift the conversation. Continuous Integration and build automation removes a step in the development process that makes have numerous benefits with it. As in allows the developer to focus on a problem that is better suited for humans. Or using snippets to generate code. All aimed at a similar thing allow the human to do the things they need to do. And not spend time on a task that is easily reproducible.

Will we have more automated tools and parts of the test process. Yes. We can still make huge advancements. With Machine Learning there is also a new set of tasks that can be aided with computations. And even if there may be a future where the tools for product owners and developers are so good that there never will be a mistake, we are not serviced about trying to replace manual tests with automated tests. For me even automated checks have a limited use since in general it is a good pattern to separate the data collection from the deductions and conclusions. Use automated tools to collect data. Use other tools to visualize and organised data but leave the deductions to humans. Treat the system under test as a patient. The tester then becomes the doctor, and the tools become these fancy machines that provide test results. The tester is still in charge of specifying which tests to make, and draw conclusions from the results. Some tests will be so cheap and common that it might make sense to always run them. Others will be very expensive so you want to exclude all other possibilities before you use them.
In all my experiences with heavily automated environments information overload is common, and you often are so busy with all the noise from the system so you miss crucial things.

My favorite example was a place where we had over 3000 automated checks. We was tasked with making a custom feature for one specific customer which basically added three new features (only for that customer) meaning that those 3000 checks did not run that. And since it was kinda a one off we did not add a lot of new tests. Due to flaky tests around 5% of the tests had a tendency to sometimes fail. But 2850 passing tests instill a sense of high quality. When the customer gets the solution they reported 27 issues with it. In this instance having done 30 specific checks, one time, would have been a way better approach than to use 3000 irrelevant checks.

tl;dr Test automation should be increase test efficiency, not replace testers.

1 Like
6

100% automated testing is just a wet dream of bean-counters who don’t like the fact that good testers cost lots on the staff budget. To get these types to shut up, mention that AI is developing so fast that in a few years’ time, we’ll be able to automate allocation of resources within a company to the most optimal level without any need for senior managers arguing over budgets so you can cut a whole load of management posts from a company… :smiling_imp:

3 Likes
7

One of these days I need to put together a talk for the Sheffield tester meet-up on cynicism and how to use it to your advantage as a tester! :laughing: