How do you “Investigate Failed Automated Tests”? - Automation in Testing Curriculum

Hi all,

We’re continuing our work to create a curriculum focused on automation that is created with feedback from the testing community. We’ve already run a series of activities that have helped us identify a list of key tasks that helped us create this Job Profile .

We’re now in the process of going through each task and analysing them to identify the core steps we take to achieve them. For this post we’re considering the task:

Investigating Failed Automated Tests

We’ve run a series of community activities including social questions and our curriculum review sessions to identify the steps we need to take to successfully achieve this task and have listed them below:

  • Regularly check automated tests runs through whatever reporting is in place
  • Discover which test has failed
  • Analyze failure information including assertion message and additional captured information
  • Review any recent code changes in the system under test
  • Determine if the issue lies with the automate check or with the system
  • Determine if the issue is caused by a difference in the test runner – tests that are very sensitive to timing may fail due to the speed of the test runner
  • If it is a problem with the system:
    • Carry out additional testing around the error
      • Verify APIs are working
      • Verify the test conditions
        • App version?
        • App install / setup?
        • OS Updates?
    • Capture details about the issue and report it to team
  • If it is a problem with automated check:
    • Analyse further to see what the issue is and if it’s a repeated issue
    • Determine if the automate should be fixed, disabled or removed
    • Fix the automated check by either correcting errors or making it less flakey
    • Rerun failing check multiple times to see if it now passes
    • Report to the team that you have fixed the issue

What we would like to know is what do you think of these steps?
Have we missed anything?
Is there anything in this list that doesn’t make sense?

What do you do when an automated test fails?

3 Likes

One thing I would do is build and run the failing versions locally and then, if the error was still there, debug it on my own machine. Clearly this wouldn’t be possible in a number of different systems.

3 Likes

If it’s automated tests written in a compiled language such as C# - have the tests even compiled? It might be that the interface to the code under test has changed in a way that stops the tests from compiling.

Has there been a change to the build pipeline that builds and runs the tests?

Has there been a change to the environment, e.g. a database connection string for a database needed by the automated tests has changed, and this change hasn’t made its way to the tests?

1 Like

Probably waaaaay too late, but this topic was suggested to me, and “I have opinions”.

Building automation to be diagnostic is a key part of the training at my old company. To the point where we built a test instance to learn from where we could cause the software to fail really easily (pointing it to another instance).

I think we overfocus sometimes on building a suite to get green ticks, and not enough on ‘what does our automation tell us when a test fails, is it useful? Can you work out what went wrong’.

In one training session, I had one tester work out what went wrong just by looking through the list of tests which passed vs failed, and make a good guess. But sometimes it’s helpful to have more information.

“What did you expect” vs “what did the system actually do” is the basics of manual testing, and automation needs to do that well as well.

2 Likes