AI has begun to influence thoughts in testing, and a major debate at present is whether the QA practitioners should start relying more into LLMs on their side or into cloud LLMs as opposite.
Local LLMs provide control and privacy since they give you the environment: you can tweak those models to your own ends, and our test data can be sensitive so that it should not be allowed to leave our system. Downside is mounting and maintenance of the hardware.
Cloud LLMs give convenience with scale. You get instantaneous access to the best models with no setups, updates being automatically handled. Downside is the expense, vendor lock-in, and data-security.
Both bring some different opportunities and trade-offs for testers.
Also, if you think local LLMs are better, please share your go-to download link / tool recommendation to explore.
@juanalvarezarquillos
My best guess is an LLM that has the context of the software in test and then you just give it user stories and get returned test cases that provide good enough coverage.
But that’s easier said than done in my experience. There’s just so much hidden knowledge in a software that before the LLM might hallucinate, the very source of truth might too.
I have so many questions: How can we gather the context of a software product? How can I verify the test cases has a proper “coverage”? If I need to check the test cases are covering the user stories, then I should fire the LLM! Including a generative AI in a thinking process is a recipe for disaster.
The main thing that is going to swing that decision is company policy. If you are told not to use cloud LLM tooling then you have to work local. I’d recommend starting with Ollama, then trying different models for the usecase you have in mind. Ollama has a built in chat interface so its easy to start with.
I used the Continue plugin to hook Ollama into my IDE for local code based chats.
If you want to start expanding from there then the desktop tool from msty.ai can use the already downloaded ollama models, but it has a more flexible chat interface and you can add agents and tools, e.g. a search agent or the playwrite/chrome dev tool agent. This might add more capabilities to your experimentation.
I’ve just started using OpenRouter for the AI programs that I’m writing because they run faster and take less load on my machine, and I typically use OSS models through open-router and these are often covered by free limits. In the last week I’ve spent 0.4 cents ($0.004) on just over 1.47M tokens of llm usage.
If you have no restrictions on using Cloud LLMs then there are a lot of chat llms to choose from and with generous free plans on each you might be able to do most work on free plans.
But… security should be a concern for corporate work, even if there is no policy in place. For testing it is probably less of an issue, but if the application you are testing is never exposed on the public web, you probably shouldn’t use any code or proprietary algorithm information in your prompts and instead keep them very generic if you use them at all.
If in any doubt - use Local LLMS.
The advantage of this is that, if you can learn to use the local LLMs well, and you are allowed to use cloud LLMs, you’ll really appreciate the extra value the cloud LLMs bring. And you’ll be able to spot when they are not needed if you managed to fulfil the use-case with a local LLM.
Local LLMs provide control and privacy since they give you the environment: you can tweak those models to your own ends, and our test data can be sensitive so that it should not be allowed to leave our system. Downside is mounting and maintenance of the hardware.
Cloud LLMs give convenience with scale. You get instantaneous access to the best models with no setups, updates being automatically handled. Downside is the expense, vendor lock-in, and data-security.
Also, if you think local LLMs are better, please share your go-to download link / tool recommendation to explore.