It’s been interesting following AIs tool that have helped speed up processes like chatgpt or copilot.
I sometimes like playing around with chatgpt to help me with test case scenarios or automation script ideas. It is really interesting how it comes up with ideas.
How will Artificial intelligence effects Quality Assurance in terms of manual testing? Will they be more demand for automation via AI tools?
I don’t think we’re there yet (and we might not ever really get there). From what I’ve seen, current tools generate so much content that it’s a lot of work to filter out the noise (uninteresting, nonsensical, or duplicate test cases/ideas/scenarios). So useful for brainstorming like you describe, but I wouldn’t personally use it for anything where the correctness of the output is important.
There are a couple of free online courses on AI from the university of Helsinki.
These are useful to give you some insight and a level of understanding of what a lot of people are currently talking about when they use the term AI, its still very narrow and the intelligent part can be debatable.
I’ve spoken with a couple of really good developers that I’ve worked with previously and the are fairly avid users in terms of generating code, both in terms of initial generation and refinement through decent prompts, they still flag the review and use with care aspect but state they are saving a lot of time. If this applies to product code then it should also apply to Automation code.
On exploratory testing side I find it useful as a buddy, give me some risk hypotheses for something I’m working on and give me some ideas how to investigate this risk. You can probably get most of the same from google but I’ve found it refines things better and perhaps can grasp your context better so maybe even gives a more linear train of thought and less random.
I do though find the term over used. Lets take something like a professional version of burp suite that has been around for quite a while, note that I am not a regular user so no expert opinions on this, it has scanners you can leverage from and parts that will crawl your code and maybe even product, they don’t actually talk about AI but around algorithms and a tool to be leveraged from in very hands on penetration testing.
I raise this tool though because we may see other tools no where near the level of this tool but they call themselves AI so it can be a good comparison, here’s a good testing tool with a lot of capabilities so what makes your tool AI, what can it do more than this non-AI classed tool.
I believe the potential is there but it may take a bit of time, hopefully it wont suck all the fun out of testing or re-direct testing in the wrong direction which could happen if rushed and bandwagoned.
At the current state of LLM (which I have been using locally) there is already a lot of utility in these models. And other more specialized models.
We already have vulnerability and best practices scanners that can be implemented in the CI. SonarQube, for example. These AI models will enhance those techniques.
Fuzzy Testing is enhanced by LLM because it is allowed to “boil the ocean” very quickly. This is how some very obscure security vulnerabilities have been found
LLMs can already help with identifying coded tests for a given feature code block. Sure some are less useful than others but it can help to identify cases or spur the tester to consider other cases.
When coding tests LLM are very useful in helping identify issues and getting a coder unblocked. They arent always right, of course. “confidently wrong” is the term. But when mulling over a problem, they can help unjam someone who is going in circles (Ive done that)
Way back in the day I read an article about an early “AI” designed to play chess. the computer would take hours to select a next move. Thats because the computer was playing out every possible move to an end game before selecting an optimal next move. This is, I think, the advantage of the use of these LLMs - they process a vast array of inputs and potential outputs. Its up to us to decide the value of the result of that process.
Thats my thought, anyways
Replying to myself because coincidentally I encountered a post on linked in
(it is linked to a whitepaper that Im not inclined to download. but the summary is useful)
LLM generated code has a higher long term cost than is often talked about. Over the past 3 years, GitClear looked at 150M lines of code to evaluate quality changes with the addition of these new tools. It is no surprise with the tool+IDE’s ability to suggest additions, but not deletions or refactors, that short term speed gains are the focus in the marketing. Long term maintainability is a very hard thing to track or pin value to, and as engineering leaders it’s important to keep a view on how this is shifting with these new modes of code production. Ultimately, we need to work with short and long term business goals, and LLM generated code must be managed in new ways to allow for long term gains.
The implication is that more testing and evaluation is required as a result of AI assistance in coded features.