A while back someone on the testers.chat slack there was a discussion on test automation and when to halt a test:
Should it be done on the first failure, or let tests continue to run even if there are failures during the run?
There’s a few pros and cons to each (and I’ve done a few different methods):
Halting on the first failure:
You get quick results on a failure, which can be very useful if you have a lot of tests, or even just a few long-running ones. On the other hand, sometimes a simple failure that doesn’t affect much of the rest of the test (such as a typo or a value that’s just off by a small amount) can cause you to lose out on some valuable feedback without going in and fixing the first failure and rerunning.
Letting the whole test run, even on failures:
This method works well if you have some assertions that, if they fail, don’t actually cause problems down the road. This can be nice if you are checking a lot of data in a table, or some similar situation where one failure is more than likely not going to cascade through the test, making the rest of the run fruitless.
Personally, I prefer the following:
Break your test suite into independently running sets (no set relies on the other for set up or tear down).
Run each of the tests without halting on the first failure. Yes, this does mean you might sometimes get into bad states, but you might find that only one test fails out of 20, and you can get a lot of good data, even if one test fails!
Optionally, you can make your test suite fancier (and I recommend this for longer-lived and larger test suite):
Set run timeouts. This way if your test gets hung up on something, it will eventually close.
Set an error counter, and stop the test if it hits that limit. We did this for one extensive set of tests where once we hit 50 or so errors, that was indicative of a larger problem, and it was better to stop than to continue.
Hard stop on certain types of errors. This one is good for if you know a certain type of intermittent error is happening and, while you might fix it sometime, it’s best to just stop the tests in the interim.
Anyone else have any ideas on how to handle their tests failing and when to stop them?