Ai-generated test cases are becoming very popular, but they come with challenges, while AI can quickly generate large number of test cases, it may struggle with some issues also, how can we mitigate these pitfalls while leveraging AI for test case generation.
I have explored this using various approaches, while they can create a large set of test cases, I noticed some issues such as irrelevant test scenarios and false positives/negatives. I expect AI to be more useful and relevant and meaningful test cases, and I found manual review and human intervention is still necessary.
The cool thing about testers when it comes to things like AI is we’re sceptics. We never stop testing a new tool or technique’s value. Fundamentally, Generative AI is as good as the data it gets, but as @hananurrehman states, anything missing it’ll fill the gaps with hallucinations.
Give a generative AI tool a full Swagger spec or schema definition of an API and it probably do a brilliant job of coding tests for it. They’ll probably be more unit test level, because it doesn’t know what the API is for or how its used.
Give a generative AI tool a story or 2 and the quality of output depends on how much information is in the story. But stories are meant to be brief insights into how the user uses a feature. You could very easily fall into the trap of trying too hard to “get AI generated test cases to work” and add more and more detail to your stories…completing losing what stories are all about.
So try it, use it for what its good at but don’t be afraid to not use it. Definitely, don’t spend time change your processes and standards to try and make it work.
I suggest you read what James Bach and Michael Bolton have written about the use of AI in testing. Their view and mine is that it’s far too unreliable and untrustworthy to use for anything important at the moment. Maybe that will change in years or decades to come, but not any time soon.
Contrary to what Gary says elsewhere in this thread, I don’t think that most testers are anywhere near sceptical enough about AI. The vast majority see AI as a magic bullet, perhaps because they are lazy or gullible.
And then there’s the age-old argument over whether you should even be writing test cases (spoiler alert - you probably shouldn’t) rather than using a more effective and efficient exploratory or investigative approach.
I think you lose the voice of your client that way. While yes AI can be super helpful for a lot of things but it’ll never account for the human factory, learned experience and work-a-rounds your end user does. While it’s a tool that can help, I always will recommend working with your client to understand how they use your application/service and test it more closely to how they use it