When to stop your automated tests?

Archive
1

A while back someone on the testers.chat slack there was a discussion on test automation and when to halt a test:

Should it be done on the first failure, or let tests continue to run even if there are failures during the run?

There’s a few pros and cons to each (and I’ve done a few different methods):

Halting on the first failure:
You get quick results on a failure, which can be very useful if you have a lot of tests, or even just a few long-running ones. On the other hand, sometimes a simple failure that doesn’t affect much of the rest of the test (such as a typo or a value that’s just off by a small amount) can cause you to lose out on some valuable feedback without going in and fixing the first failure and rerunning.

Letting the whole test run, even on failures:
This method works well if you have some assertions that, if they fail, don’t actually cause problems down the road. This can be nice if you are checking a lot of data in a table, or some similar situation where one failure is more than likely not going to cascade through the test, making the rest of the run fruitless.

Personally, I prefer the following:

Break your test suite into independently running sets (no set relies on the other for set up or tear down).
Run each of the tests without halting on the first failure. Yes, this does mean you might sometimes get into bad states, but you might find that only one test fails out of 20, and you can get a lot of good data, even if one test fails!

Optionally, you can make your test suite fancier (and I recommend this for longer-lived and larger test suite):

Set run timeouts. This way if your test gets hung up on something, it will eventually close.
Set an error counter, and stop the test if it hits that limit. We did this for one extensive set of tests where once we hit 50 or so errors, that was indicative of a larger problem, and it was better to stop than to continue.
Hard stop on certain types of errors. This one is good for if you know a certain type of intermittent error is happening and, while you might fix it sometime, it’s best to just stop the tests in the interim.

Anyone else have any ideas on how to handle their tests failing and when to stop them?

4 Likes
2

For fuller coverage you want to continue with the tests.
For fast feedback you want a notification as soon as one test fails.

What I ended up doing with a critical but long-running suite was to configure it to keep going, but manually check periodically during the run that no errors have occurred yet. (I did this by search the console output for a “test failed” type of string.)

On the face of it, it wouldn’t be rocket science to automate that periodic test. Ideally your automation framework could be configured to send an immediate notification when the first failure happens - though I don’t know any way of doing that in our framework (Jenkins, Maven Surefire/Failsafe)

2 Likes
4

I think it is dependent on when the tests are running and their purpose.

For tests run after dev commits you should be failing fast, the fewer tests the better. For longer continuous tests then I would be running the full test.

But there is probably more to consider but that would be my starting point.

1 Like
5

I agree with some of the comments here that it depends on how you are running your automated tests-

  1. If you are just writing an automated test for some flow, and you keep running them as and when you add steps to it; it makes sense to stop it midway if you know you messed up something or you can give shorter timeout periods to quickly get feedback on the test.

  2. If you are running a huge regression test suite, say 300 tests. Just because the 202nd test failed, you dont want to stop running the rest of the tests. You still want feedback on the overall health of your system. In those cases, I will fail the 202nd test but will still try to run the rest of the tests. This becomes all the more important when running tests in a CI/CD pipeline

1 Like