Will demand for QA grow with AI?

I read this LinkedIn post about AI and the demand for QA.

Demand for QA will soon explode. The flood of AI-generated code will require rigorous testing and validation.

And the author emphasised the following:

Increased emphasis on QA
Innovation in QA methodologies
Shifting QA roles and responsibilities
Software development blurring into specification writing
Emphasis on soft skills

I’m curious about this all and still trying to wrap my head around how the role of the tester may need to innovate and change with the growth of AI-generated code.

How about you? Where are you at? How do think the role of the tester/QA person will adapt? Have you seen a change already when working with applications that have been supported with AI-generated code?

6 Likes

From first hand experience the A.I.'s output is very unpredictable.

Large language models like ChatGPT are very very big - and sometimes that leads to unpredictable behaviors.

For this reason people are creating more and more guardrails around generated content.

TBH - AI is a game-changer for all of us in the tech world, and others too!

Unlike traditional coding, AI is all about learning from data and making predictions. It’s opened doors to cool stuff like image recognition and language understanding, which was unthinkable before. But it’s not all roses - AI can be unpredictable, and that brings a new set of challenges. So yeah, with AI, the ‘impossible’ keeps getting redefined…and that’s unpredictable!

And when there’s unpredictability, there’s QA!

5 Likes

I agree and disagree in equal measure with the points made by this fellow on linkedin, I’ve probably been building software for too long.

But at least todays weather prediction was spot on, it’s raining in normally dry Cambridge this morning. When anyone makes predictions and your prediction is that “things will change”, I get bored; but points I do side up with are:

  • The responsibility of QA shifting, being more present in the “go-live” decision process. If only as a way of feeding back production defects more quickly, getting your QA team up to the coalface can only be good for everyone.
  • Emphasis on people skills, is for me key, maybe because I’m getting old, but that has only taught me, that people write the code together. No X10 developers, coding standards , frameworks, nor clever processes can guarantee victory, cohesive teams do. Especially in todays isolated remote-working mode.

But I generally see AI as a thing of scale, not as a fresh enemy, rather as growing up and coming of age. QA are going to have to pick up the baton, if they don’t, then someone else will. And I’m talking about using large-dataset systems, learning to work with tonnes of statistics and turn it into product knowledge or insight. Gone are the days when users would report bugs, today they just swap platform, and for QA to find new ways to detect the “unreported” bugs that cause users to leave may be just one of the “large-data-set” tool areas we need to grasp. For example: Use our test workflow analysis mindset to help orgs build using analytics to uncover workflow issues in a product. I don’t think QA people will suddenly find they have more jobs in the marketplace, I rather believe they will have to work alongside a different set of people in the org.

/edit My bias comes from how ChatGPT-4 is much like iPod was to .MP3. Nobody outside of tech circles knew what music compression would mean, until someone commercialized it. Mp3 as a format changed how artists distributed their art, and more recently how they built it. Some of the impacts will lag a lot.

3 Likes

My relevant experience is not with AI, but with testing a complex system we’ve designed and built with lots of internal rules that affect the result. The similarity to AI is that I test it by trying it out on examples and assess the results via high-level acceptance criteria and my own judgement rather than specific requirements specifying the expected result. However, I also have the option to inspect the inner workings (to an extent) and manually configure certain variables and see how they affect results.
In addition to greater visibility during testing, if we don’t like the results for a specific input, we can identify which rules are causing it and design changes to improve the behaviour. I don’t see this being possible with AI, though as I said I don’t have experience with it.
One of the big problems I’m facing with testing this system, which would apply to AI too, is getting a good overview of behaviour and how it changes. Ideally I want to see results from many different inputs and an analysis/overview of key features, and the ability to compare today’s results to yesterday’s with any differences highlighted. I guess this big picture, as well as the details of the inner workings, are the things that may present difficulties when testing AI.

2 Likes

My fear is that, as already before LLMs, people will underestimate the necessity for testing (and also overestimate automation).
Especial as LMLs output sounds most times quite reasonable. Which is dangerous when the content is just made up.

People should make more use of testing, but if they will? :person_shrugging:
Therefore they at least have to acknowledge that LLMs are not a magic wand.

So far I have no interaction, nor necessity to, with any AI.
With AIs changes one more time the subject of the testing, but the basic principals will stay the same. Finding problems which matter to humans.

4 Likes

It is always hard to predict what will happen, but it is still important to imagine what might happen. That’s why I’m enjoying reading this thread.

We can influence what happens with AI in QA if we think creatively about what might happen and what could happen, and use those ideas as we make decisions about where to spend our time now.

As everybody here already knows, we can influence how AI itself develops if we interact with it. Several months ago, I tried to get ChatGPT to write a poem in iambic pentameter and realized it couldn’t. So I tried to teach it. I started with “Write me a sentence with ten syllables in it.” It couldn’t, so I tried some more prompts like writing words with certain amounts of syllables, etc., and got an idea of how well it could deal with counting syllables (very poorly at the time).

A couple of months later, I asked it to write me a poem in iambic pentameter, and it got pretty close! So it looks like lots of people were teaching it to do similar things.

How does that translate into how QA can use AI to improve quality? I’m not sure yet, but I’m not going to wait passively to find out. I’m going to go out and explore and use my imagination!

3 Likes

From what I’ve seen so far, AI is already useful for giving an outline framework for test cases (our Tech Team are already using it for producing various scripts) and can have a pretty good stab at producing some automated tests (I’ve seen a few examples using Cypress) but what is produced is only the first cut and needs a thorough review and usually some tweaking.

Obviously AI is improving all the time (presumably the more it’s used for testing, the faster it will improve) but for now it’s only as good as the information it’s provided with (and sometimes not that good). I doubt it’s going to vastly increase the demand for QA but is more likely (in the medium future) to change the role of the tester to improve the level of precision given in test case definitions (AI is only as good as the instructions it’s provided with at present).

2 Likes

@sebastian_solidwork That’s my fear as well. I think the need for testing will increase, but the demand probably won’t.

3 Likes

I’m not sure the premises in “the flood of AI-generated code will require rigorous testing and validation” are well-founded, even if they sound logical. Are LLMs concretely leading to a significantly higher rate of customer-facing digital products being launched? Are LLM tools leading to significantly higher release cadences for existing products? Sounds plausible to me, but is it true? Will it be true? And if a lot of new products are being launched, do they require rigorous testing or are they tentative prototypes to test the waters for a large number of possible business ideas?

Anyway, if we assume we work for a company that’s found a way to produce a “flood of AI-generated code” that meets the need for new features, change requests, and new products, will demand for QA increase? Thinking out loud:

If it’s a developer driving the code generation, I expect the tools that help them be more efficient (in scaffolding code, debugging issues, rubber ducking technical decisions, and so on) to be joined by tools that also make testing more efficient (in identifying issues, generating test scripts, running only relevant tests, and so on). Perhaps that arms race will keep everything in tune.

If it’s a business person driving a fully LLM-produced product, well, I will echo that “the need for testing will increase, but the demand probably won’t.” If that demand does increase, because the business person realises they need a better result than they’re getting from the LLM, then it won’t just be a need for human testing, but also for human development. Again the demand for testing matches the demand for development.

2 Likes

I’m inclined to agree with Anna (@sles12 ) that the need for testing might increase but the demand probably won’t.

About the only thing I can see truly increasing the demand for testing is C-suite folks starting to see testing more in the line of essential insurance than a cost that gets tacked on at the end so they can say they’ve done due diligence.

It will probably take a fair few more bankrupted companies and dead patients before the idea starts to percolate, too.

And yes, I am rather cynical.

P.s. The stack overflow blog on the topic of AI in coding is rather apt to the topic: The hardest part of building software is not coding, it's requirements - Stack Overflow Blog

5 Likes

As a QA/Tester when we use the AI , we have pro and cons but how are we going to use this was the more challenging thing need to discussed .

For an example

You wants to create the billion of test data for doing your test activities for that if you need to generate the script . Yes you can go ahead but is that script more optimized one to use . That mean when u create your own script it took 3 min for 1 billion records but AI generated script took 1 min then it will be more effective way to use the AI in QA activity

As mention how are we going to use this AI to QA is that challenging decision.

1 Like

Not had to test any AI generated code yet (to my knowledge, at least)

I think the sentiment that testers will be needed to test AI generated code is right, but to describe it as a ‘flood’ - nah. Not yet. Plus if the codes getting generated, what are the developers doing? They’ll test it too, right? RIGHT??

It’s an interesting time with AI. We’ve got writers and artists literally crying out plagiarism whenever writing or artwork is generated by AI.

So why are developers not crying out when AI code is being generated?

1 Like

I’m perplexed
“use AI to generate test data”… um that makes no sense at all to me, it’s only going to generate cases that are popular from it’s training set surely, and so it likely wont include any cases that are “unknowns”.

2 Likes