The future of testing in an AI world?

Discussions
, ,
1

I’ve been thinking about the future of software testing in the world of AI - specifically how the role of a tester may change as AI tools are used to build and test software.

For organisations that write their own software (or use a third-party development team) and have a high risk appetite - perhaps an eCommerce website or internally used back-office software - the accelerated delivery of features through vibe-coding will be attractive to product owners. There may be a temptation to ship quickly (YOLO it out) and validate in production using monitoring and end-user feedback, then fix forward or roll back. In this situation, the role of a tester could morph into a quality coach, and we may also see a return to eXtreme Programming practices - working closely with product and development to validate quality as a feature is built.

For organisations that build software for other organisations and therefore have a more risk-averse outlook - such as vendors of SaaS platforms or COTS products - the demand to innovate faster may still drive adoption of AI-assisted development. Here, specialist testing expertise becomes distributed across the SDLC - advising and influencing design and development, and using AI tools to accelerate test design, implementation, and regression coverage to validate the platform while managing risk.

For organisations that buy software from third parties/ISVs, validating that the solution meets business needs will still largely be done via user acceptance testing and operational acceptance testing. However, this can be accelerated and enhanced through AI tooling- for example generating test ideas and scripts faster, building automated tests for key operations, and collating, summarising, and analysing test outputs for reporting and decision-making.

AI is having a huge impact in the IT industry - accelerating development and test processes, and this shifts testers toward coaching, risk management, and verification of what is built (by AI or human) - as long as humans use software, humans will need to test software.

How do you see this evolving in your roles?

5 Likes
2

AI Won’t replace Testers , The key is using real AI Thoughtfully. First we need to decide what is our need and what are our boundaries. AI is vague , in that vagueness we need to identify what are our limits and what we need from AI. AI will help testing evolve, but testers still need judgment. Use AI to improve tests, not just rely on it blindly.

2 Likes
3

For fun, I’ve been collecting stories on how AI is (not) going just great.

There is impact. There is FOMO. There is massive change.

It’s unsettling. It’s also exciting.

I’m doing my best to be positive, but to also have that risk averse lense.

I don’t really know what to expect, but increasingly I’m finding it comforting having actual conversations about what is happening on the ground.

We’re sharing all of this via MoT, if that’s of interest. It’s a huge responsibility of ours, that I’m taking seriously, to help us navigate a quality led world.

6 Likes
4

I like the enthusiasm many people and companies show toward testing with AI. Having autonomous agents handle tasks such as to develop the test, fix the test, launch the test, etc.

To streamline the workflow even further, I would propose adding a supervisory agent that coordinates and oversees all the other agents. This would free me from manual involvement :smiling_face_with_horns:

I’ll be watching the process from the comfort of my couch with a bowl of popcorn— what possibly could go wrong? :popcorn:

2 Likes
5

I’m reluctantly now considering that AI will replace a lot of roles in testing, QE, QA and automation and potentially rather than replace make the need for certain activities we do redundant.

My initial view was that it would be primarily over a couple of years be activities that favour mechanical strengths with the basics being scripted testing and automation.

The hands on tester who actively focuses to leverage on activities that humans excel over machines, comfortable with unknowns and ambiguity, broad risks, context aware, experiments and deep investigations empathy etc I had felt was a bit safer. This still might be whilst we are still building human centric applications where determinism was just a guard railed illusion and we accepted humans as being naturally non-deterministic, ambiguous and with the occasional hallucinations.

I am now also questioning this with a couple of strong caveats.

  1. Can multiple agent use remove ambiguity, unknown risks and move towards a model where everything is known very well? Here testers may not be replaced but could be obsolete.

  2. Will the quality, ambition, innovative products that stand out from the crowds bar drop and will that be accepted mainstream - if so the need for those human strength traits may again not be required. So again its making human strength activities redundant rather than replaced.

One thing to consider is historically their has been a accidental drive away from human strengths towards mechanical strength activities, even the tester/discoverer role to more of a QE builder/protector role could be seen as a drift more to mech favouring activities.

It’s moving too quickly, decisions and fears on this are potentially jumping the gun but I’d still be experimenting and preparing for multiple outcomes.

If the role changes to a conductor and not so sure I’d enjoy it even if very capable of making that transition.

2 Likes
6

Your closing point is the one I keep coming back to:

“as long as humans use software, humans will need to test software.”

I think this is really important. And I believe holding onto it is going to be harder than it sounds. Because the testing challenge AI presents is fundamentally different from what we’ve dealt with before. And the pressure to sideline human judgment will be stronger than we expect.

It is essentially a testability problem - specifically around observability and controllability. AI systems are opaque. We can’t fully observe how they arrive at their outputs, and we can’t reliably control their behaviour. Testing in production is essential, arguably even more so with AI. But an AI failure can look like a perfectly confident, completely wrong answer, and we’re still learning what to look for.

What makes the whole thing worse is the sheer pace of change. Model capabilities advance so quickly that our evaluation methods struggle to keep pace. Many of the most consequential capabilities are emergent, i.e., behaviours no one designed or predicted. This makes risk compound in new ways: when AI components interact with other AI components, they create feedback loops where small drifts can cascade into changes no one explicitly decided to make. On top of this, there’s the evaluation lag: by the time you’ve built new behaviour patterns into your evaluations, something else has changed.

Distributing testing expertise across the SDLC should be the default for any project I think, AI or not. A quality coach is a great way to build that culture. What AI adds is a harder oracle problem though: we need new ways to judge whether output is acceptable when the system is non-deterministic; we need sharper consistency heuristics.

Automated evaluation is essential, and with AI it needs to expand into new territory — checks that are more statistical in nature than binary pass/fail. The question is where human judgment needs to stay in the loop.

I think this is fundamentally a systems thinking challenge, and exploratory testing becomes a core approach here: building tacit knowledge about how these systems behave, codifying it, and using that to develop deeper understanding. Of course, that cycle has always been part of good testing. But with AI it becomes critical — the risks are bigger, they change faster, and the window to adjust before the next shift is shorter.

This applies to our testing tools too. AI agents can help us test AI, and, realistically, we have little choice but to use them. But those tools carry the very limitations we’re trying to test for, so someone needs to be watching them. In this context, seeing human involvement as a bottleneck grossly misreads the situation; instead, in AI-integrated systems the human feedback loop is one of the few irreplaceable control surfaces we currently have. This is something we need to keep articulating clearly, I think. Because the testing challenge with the emerging systems is genuinely new and genuinely harder. And our collective understanding of how to approach it will always be catching up.

2 Likes
7

AI landscape is changing too fast, for me definitely AI changing for testing as well and this is happening now, earlier to writing test case s manually to automation, debugging, report creation is time consuming task with AI as companion things speedup as well as quality, AI add more thoughts now but with speed always need to have human in loop (HITL) to ensure qulaity.

8

this is interesting thanks for collecting

9

AI WILL NOT REPLACE TESTERS, it’ll simply elevate their roles. Testers will evolve into strategists, quality advocates, and AI supervisors, rather than being mere executors who only write, run, and review test cases. The smarter approach is to embrace the change rather than fearing it.

10

Agree with the points above, I do not see the role of testers disappear. I do see the way we do our testing changing. More efficient work by overseeing rather than doing everything ourselves. With AI making the job much more efficient, it might result in fewer jobs for a period. Though I also really believe that together with AI assisted development, there will be an increase of product development which will also mean an increase in demand.

The most important thing: stay up to date with how the role is evolving and educate yourself on AI and new tools

1 Like
11

This is so interesting as it represents a seismic shift in how we interact with software. Collaborating with a tool rather than directing it is probably the biggest change that any of us have faced in our roles, yet the concept of AI isnt new - its been around since the 1950s at least. The surprise is that its taken so long to get where we are, but now the pace is quickening and it naturally leaves us wondering about our roles.

My take on this is that it is an opportunity to reframe Testing/Quality Engineering in the AI space. We only look at what testing is through a traditional model - someone writes code, someone else tests it and its all human driven. Our future roles will pivot into more governance led roles, as there is a need for human oversight when it comes to the models that are being created, the data sources, the inherent biases that will exist. It will be up to us to flag where tools are making decisions that are unethical, biased, unfair, incomplete, invalid etc. How will we know that the data sources used to make decisions are unbiased, comprehensive, true and verified etc (unless we provide all the data ourselves which wont always be the case).

I think its a massive and exciting opportunity but one we as a profession are not ready for. I predict a race to the top with loads of people creating processes to follow whilst we are in the vacuum, and this could lead to a lot of confusion.
I wonder if we need a set of standards and guidelines to follow - not necessarily a certification - but something that defines what we should be doing, and helps us showcase to what that is to our stakeholders and peers, so they are clear on our roles.

Interesting times indeed!
Steve

1 Like
12

“moving from test case creation and execution”, “testing needs to evolve from testing into testing” , “testing has been always based on determinism”

It’s hard to phrase this without sounding harsh but I am seeing this minimalist view of testing quite a bit it, for me it flags significant risks and at what could be a fairly pivotal time so its important to at least consider.

For me it carries a potential indication that for some almost no embracement of change has been evident over the last few decades and a risk they will attempt to do the same flawed things just faster, rather than see opportunities for real change.

There are likely very easy paths forward, many that managers and sales folk will love but the testing bar likely drops further, on this path I have no doubt AI will replace a lot of testers.

There are other paths though, for me its likely those who have previously and continuously embraced change and rather than that minimalist view of testing being waved around, have for decades had a much broader and holistic view of testing that may be in good positions to choose the path and also protect some of those values being lost should certain other paths be taken.

2 Likes
13

AI will amplify whatever definition of testing we bring to it. If that definition is small, the impact will be too.