I’m currently working in a company where all the devs are heavily involved in AI and have catchups, meetings etc around the topic. But not much talk about it in the testing space.
Not going to lie I’m quite out of the loop with AI and don’t want to be left behind.
I hear both sides of the spectrum when it comes to the benefits of AI. Was potentially thinking that I could bring some topics into my testing guild/meetup sessions.
In the testing space at your companies, what AI related topics and tools are you discussing.
I was thinking this could potentially be an area I could look to potentially specialise in the quality space.
I’m currently doing some research into AI and using the in-house tool we have at my company.
Awesome, that you are thinking ahead like this, honestly you are not alone, a lot of testers are still trying to figure out how AI truly fits into the quality space.
Some AI-related topics in testing:
AI-powered test automation: Tools like Testim, Mabl & Applitools are using AI to help with smarter element locators, visual validation & self-healing tests.
Predictive analytics for QA: AI to predict risk areas in an app based on previous bugs, user behavior or system usage patterns.
AI-assisted exploratory testing: Tools like Test.ai aim to human-like exploration to uncover bugs that scripted tests might miss.
NLP for test case generation: Platforms are emerging that turn plain English into automated test scripts.
AI for performance testing: Some tools use AI to model & simulate real-world traffic patterns more accurately.
you can even start even small by:
Hosting session
Running comparing a traditional test automation tool vs an AI-assisted one.
Setting up tool exploration, where team experiments with one AI-testing tool
Just wanted to know,
is your in-house AI tool something built specifically for testing or is it more of a general internal AI platform that you are thinking about adapting for QA purposes?
Love the list @squarerootuk , I might do some more digging myself around those areas.
Our main AI tool is github copilot. I don’t use it personally because of my role, more my team. However, I did do a lot of practice around GenAI and found there was a lot of power in the quality of my prompts and followed the guidance of Dave Birss using the CREATE and PIQPACC principles. We keep a library of our best prompts, not only to reuse but also to inspire more.
Really cool to hear @ghawkes’s team is actively using GitHub Copilot, it is such a smart way to boost productivity in dev-heavy workflows, & big respect for following Dave Birss’s CREATE and PIQPACC principles, the prompt crafting is an art in itself & having a prompt library is genius. That kind of collective knowledge sharing can totally level up a team’s AI game.
since you are already curating top prompts & seeing their value internally, have you experimented with automating test case creation or bug report generation using those refined prompts? That might be a killer next step, especially if your team is already deep into Copilot.
also how do you decide which prompts make it into your library? Is it based on outcomes, team feedback or something else?
No. I suppose should try to prove/disprove my concerns but the test cases will be as good as the information you have to hand to provide the prompt. If that works by supplementing that with a lot of domain knowledge then I don’t want people to fall into the trap of working hard to make the GenAI to produce “the right answer”, when they can just produce the right answer themselves. So in other words, a lot of work would have to be done on stories to contain enough information for GenAI to get close and if they don’t, then you’re doing the work, not AI.
Having said that, I did have specific successes. I once fed it an API spec to give me a top 10 most important test scenarios and the test scenarios were pretty good - but we wrote the tests based on those scenarios. I also had to write documented customer facing test plan and with a good prompt, I got a good result with that too and just needed to make some tweaks and customizations. I’m very interested in seeing what I can do with predictive analytics.
Yeah its results based. Whatever prompt you used that delivered a productivity benefit, put it on the list.
Appreciate your honest take, you are absolutely right that there is a fine line between making GenAI work for you vs. working to make GenAI look good. That balance is tricky, when especially the context & domain expertise still rest heavily with us as humans.
API spec example is exactly the kind of smart, targeted use shows promise that AI not to replace, but to spark smarter thinking or provide a launchpad & getting a head start on customer facing test plans? That is a real-world win right there.
Prompt quality being only as good as the input makes me wonder ==> Have you tried layering prompts like using one prompt to extract gaps or assumptions from user stories before attempting automation? That might reduce the risk of garbage in, garbage out while still making the most of AI’s pattern-spotting strengths.
Also, when a prompt gets added to your library, do you ever revisit or version them over time as models evolve? if you have seen older winning prompts start to under perform as LLMs get smarter or more nuanced.
