Why do we automate?

Hi all

I have a genuine question as to why we believe that writing code to test code is a good way to test that something works as it should.

It always seems a little odd to me that we have developers writing code, which they (in theory) write unit tests for, then testers come along and test the dev code, and then write automated testing code to test the dev code.

It feels like we put a lot of effort in to testing whether dev code is working by writing test code, but then how do we know that the test code is working? Is there a missing piece of the puzzle?

We rely heavily on the results of the automated tests to tell us whether something is not working as expected (as part of a regression test), but without even more code to test the test code, how do we know that we can rely on the test results. Just because something says that it has passed, is it reliable and can we believe it? After all, it seems to be quite common in my experience that automated tests fail but not always for genuine regression issues - timeouts, data issues, UI changes that have not been reflected, removed functions etc.

So how can we have confidence that the test code we write to test the dev code is any good and serves a useful purpose?

It might seem an odd thing to ask, but my mind does work in odd ways (I digress but when I was 8 we were told to write a story about a tank and I was the only person to write about a fish tank, not an army tank!!), so I usually end up thinking of things in a different way, and this bugs me. I’m not out to cause any arguments as to the benefits of automation - I just wonder how we can have any confidence in something that isnt tested itself, if that makes sense.

Thanks - I await responses with interest :slight_smile:

2 Likes

You raise a lot of valid concerns here, the part about the test code itself not being tested I’ve found the most intriguing. From my perspective, it does get tested in a way. When I write a new automated test I’ll make it fail first (if I know an endpoint will return 200 I first put the expected result as 400, for example). Another thing I do to “test my tests” is use of breakpoints and then running the test in the debugger, to get a more intimate glance of what I’m getting back.

Also, good testing design goes a long way of helping us to have pretty reliable tests. Implementing SOLID principles, like SRP in terms of what is being checked in your tests is quite usefully in a way that you minimize the potential points of failure. If you test is small and concentrated on validating just one thing then it will be much easier to figure why it is failing and it will have less potential for flakiness. I try to have a lot of small tests rather then just a few big ones, whenever possible.

For me UI testing is the most annoying by far, since there are times where you will need to automated rather complex customer journey where your test will need to have a lot of steps and debugging such tests can be a pain in the lower backside!

@sjwatsonuk

#MutationTesting for developers. Example: https://pitest.org/
It’s a way to test if your unit tests really cover everything.

Unit tests may show 90% code coverage, but what I want to know is if you have 100% mutation coverage & 80%+ code coverage.


You have a valid point of who looks at the testers code? Well in my sector all code is being code reviewed even testers code. (By Dev to see if all standards have been followed, By Testers to see if all tests are covered & By Security, to see if there are no sensitive data leaks etc… ) I haven’t seen this culture but only in 1 company so far.

It also helps if you can read the code from the dev. “< > <= >= || && int string …” tell us a lot already and of course there is so much more.

Is there enough coverage for automated tests? No never, it’s impossible to get 100% coverage, there will always be someone who thinks of another scenario. But is it worth it? (is the ROI big enough?)

What if this happens:

At some point I suppose you need to have ‘faith’ in the one who writes the tests. A lot depends on the mentality of the people.

2 Likes

you can test your test code…but lets get to the question.

I think it’s too easy to say that tests themselves can be faulty. Yes, they can, but if you have a test writing disciple that follows similar parallel patterns to app coding disciplines, the loop will close itself. and it’s super important to have strict process around the writing of automated tests, like ensuring the test suite owners have enough time to address concerns, have enough time to look after code quality, and are continuously refining their own processes. Testers need retrospectives too.

The automator who thinks their automation is perfect needs to be dropped into the same swimming pool as the dev who thinks they don’t need to code unit tests. Okay, So I started out by saying you can test your own code, here is how, it’s a mix of 3 approaches you need to be taking. I’ll start with the “cheapest” technique first.

  1. Golden images, have a golden image/environment of the app under test. A version that never changes. Every time you make a systemic test framework or suite change, run the whole shebang of tests against the golden image environment. The test report should report all the same fails that it did before you made a test code change.
  2. Sometimes the golden image technique is not fitting your context. In that case you need to break the product subtly, manually, and then run your automated tests against it, and almost all of your tests must fail. I find it’s most useful to try this when the product is in a known broken state because the devs make a breaking change to your environment. Run your suite anyway, but observe closely to ensure that all tests fail for the correct reason, the one thing the devs broke. If a few tests somehow manage to bypass the broken area, give those test cases a stern talking to.
  3. A test framework that has self-tests. And then, tests that are as low-code as possible. This means that every time you add a test case, you are not creating code churn. Test automation is much like an onion as shreck would say, they have layers. Test cases are the upper layer, your framework forms the lower layers. If test code breaks the “layer”, it needs refactoring. This gets easy to spot if your framework and test system language is well designed. This also means that test-cases often defeat the DRY principle by nature, and an exception to the disciple on app coding. This is a good thing for tests. The self-tests you write need to be pretty exhaustive, and split up into suites that attack each problem.
  • A suite that tests your test toolstack, but at a product-agnostic level
  • A suite that tests your product test fixtures themselves

And of course, test logging to help prove or disprove false positives

1 Like