When to stop your automated tests?

(Charles) #1

A while back someone on the testers.chat slack there was a discussion on test automation and when to halt a test:

Should it be done on the first failure, or let tests continue to run even if there are failures during the run?

There’s a few pros and cons to each (and I’ve done a few different methods):

Halting on the first failure:
You get quick results on a failure, which can be very useful if you have a lot of tests, or even just a few long-running ones. On the other hand, sometimes a simple failure that doesn’t affect much of the rest of the test (such as a typo or a value that’s just off by a small amount) can cause you to lose out on some valuable feedback without going in and fixing the first failure and rerunning.

Letting the whole test run, even on failures:
This method works well if you have some assertions that, if they fail, don’t actually cause problems down the road. This can be nice if you are checking a lot of data in a table, or some similar situation where one failure is more than likely not going to cascade through the test, making the rest of the run fruitless.

Personally, I prefer the following:

Break your test suite into independently running sets (no set relies on the other for set up or tear down).
Run each of the tests without halting on the first failure. Yes, this does mean you might sometimes get into bad states, but you might find that only one test fails out of 20, and you can get a lot of good data, even if one test fails!

Optionally, you can make your test suite fancier (and I recommend this for longer-lived and larger test suite):

Set run timeouts. This way if your test gets hung up on something, it will eventually close.
Set an error counter, and stop the test if it hits that limit. We did this for one extensive set of tests where once we hit 50 or so errors, that was indicative of a larger problem, and it was better to stop than to continue.
Hard stop on certain types of errors. This one is good for if you know a certain type of intermittent error is happening and, while you might fix it sometime, it’s best to just stop the tests in the interim.

Anyone else have any ideas on how to handle their tests failing and when to stop them?

(Jonathan) #2

For fuller coverage you want to continue with the tests.
For fast feedback you want a notification as soon as one test fails.

What I ended up doing with a critical but long-running suite was to configure it to keep going, but manually check periodically during the run that no errors have occurred yet. (I did this by search the console output for a “test failed” type of string.)

On the face of it, it wouldn’t be rocket science to automate that periodic test. Ideally your automation framework could be configured to send an immediate notification when the first failure happens - though I don’t know any way of doing that in our framework (Jenkins, Maven Surefire/Failsafe)

(Heather) pinned globally #3

(James Farrier) #4

I think it is dependent on when the tests are running and their purpose.

For tests run after dev commits you should be failing fast, the fewer tests the better. For longer continuous tests then I would be running the full test.

But there is probably more to consider but that would be my starting point.