🤖 Day 19: Experiment with AI for test prioritisation and evaluate the benefits and risks

There’s some overlap between today’s question and my answer back in day 6.

In context of automation and regression runs, I see a benefit of a tool that would select only relevant tests in the context of the change being made by developer. It doesn’t really have to be AI - you can relatively easily script something close enough, assuming you have the right data.

The main benefit is faster PR regression testing and shortening the time from opening the PR to merging it. It’s not too bad in my team, as full regression run takes 30-35 minutes. But I’ve been in teams where it took 12 or more hours, and shortening that by even 25% would be great improvement.

The obvious risk is that tool might decide to skip the test that would actually encounter some issue. You can minimize that risk by keeping nightly full regression runs. Tool should also provide configuration option, so you can decide if you want it to be liberal or conservative when it comes to selecting tests.

In context of wider testing activities, I would not want a tool to decide what am I supposed to work on and in what order.

4 Likes