Have you faced any intriguing challenges while implementing AI in testing?
What innovative solutions did you discover to overcome them? 
As we embark on this exciting journey into the world of AI in testing, share your experience and insights 
3 Likes
I have found that like any “shift” in technology, there are a lot of cool ideas around AI in assisting in testing. But I am cautious. For a couple of reasons. There are always “cool new things” that get started, but they inevitably run into friction and are then abandoned. I dont mean AI itself but rather new “products” that promise to be the next big thing but end up being a poor investment professionally because they so often end up withering and dying. So I approach all these “no code”, “ai generated tests” “self refactoring tests” claims with a lot of caution, but I am watching for maturity.
Im also concerned that AI tends to be very homogeneous in its responses to problems posed to it. So it might be pretty good for automating the creation of very ridgid testing - where the tests are pretty obvious if tedious to write. Or it might be effective at “boiling the ocean” and iterating through massive matrices of inputs without giving weight to realistic scenarios. I do think it will be very beneficial in security testing.
For now I find it most beneficial as a sounding board for things I am trying to do. Asking questions about a blocking problem. Generating code snippets. That sort of a thing.
Im also very interested in how to test AI models themselves. The current “tests” of AI models are pretty fuzzy and impractical; only really useful for comparing models to other models rather than “can it do the job I want it to do, safely and accurately?”
6 Likes
It sounds like you’ve approached AI in testing with a healthy dose of skepticism, which is understandable given the hype surrounding new technologies.
Your caution about the potential for promising products to fizzle out is well-founded, as is your observation about the homogeneity of AI responses and its limitations in handling realistic scenarios.
Your focus on using AI as a tool for generating code snippets and providing insights is a pragmatic approach. Additionally, your interest in testing AI models themselves reflects a crucial aspect of ensuring the reliability and safety of AI applications.
It’s important to continue monitoring the maturity of AI in testing and to adapt your strategies accordingly as the technology evolves.
1 Like
I’m not sure I am young enough to even embrace the new AI thing we call AI at all without caveats. For me, computer learning started in the 90’s with trying to use Bayesian models in troubleshooters. I worked out that manually capturing the data was more useful of itself, than trying to work out how to model it. Remember this was the age of generative music, and right now we are in the age of generative art, and even back then computers were only good enough to fool you if there was an artist like Jean-Michel Jarre up on stage actually queuing it all up into a lightshow too.
Today I find it frustrating that asking the nightcafe instances of AI to draw something takes at least 5 attempts to produce something that although unique, is barely passable. If you don’t have hours and hour to learn it’s language, it does feel like I am “boiling the ocean” as @msh puts it. To me it’s still a rent-seeker, and is only useful while my ability to negotiate the cloud provider contract price down stays sane.
The “technical” insights will come to me too I’m sure. I hope to continue to observe closely. But as always, computers are built by humans, and that’s the common denominator, we love to build complex systems, which ultimately are controlled by someone to meet their own goals not yours. And finding that mutual benefit will be the prise always.
1 Like
Thankyou @conrad.braam for sharing your insightful perspective on AI in testing. Your experience with computer learning from the 90s and the challenges you’ve encountered resonate with many in the field. It’s clear that while AI presents exciting opportunities, it also comes with its complexities and limitations.
Your emphasis on aligning AI initiatives with broader goals and finding mutual benefit is crucial for success.
As always, value your contribution 