What have been your biggest challenges working on projects involving AI-coded applications?

MoT Content Discussions
,
1

Testing applications developed with AI coding tools is a unique experience—equal parts fascinating and frustrating. While these tools speed up code generation, the code they produce often lacks scalability, context integration, and overall quality. This makes the role of testers and quality engineers more critical than ever.

In my post, “Testing AI-coded applications: practical tips for software testers”, I share real-world experiences from testing projects that relied on AI-generated code. From debugging the outputs to facing integration and scalability challenges, the journey was a rollercoaster, but worth it.

What You’ll Learn:

  • Insights into the unique challenges of testing AI-generated code, such as dealing with hardcoded data and code that mixes backend and frontend implementations.
  • Tips for creating robust testing strategies and critical validation of outputs.
  • How testers can redefine their roles in AI-driven projects to ensure quality, scalability, and easy integration.

Your Turn:

  • Have you worked on projects involving AI-coded applications?
  • What were your biggest challenges, and how did you overcome them?
3 Likes
2

I was literally penning a related topic…and then your article arrived :grin:. It really helps with my problem so thank you.I’m also interested in other peoples experiences so will be watching this thread.

My particular problem is we have an initiative thats begun in my organisation to trial an AI Coding Agent “Augment”. As a quality leader, I like to be as risk averse as is sensible so I asked a couple of high level questions:

  • What is the objective of the trial? Answer “Try out the tool”
  • How will you know the trial has been successful? Answer “Engineers like the tool”

So as you can tell, my alarm bells are going off. A lot of the potential risks I tried to highlight are in your article, so very pertinent timing :right_facing_fist

Trying my best to prevent any quality shortfalls, but I fear I will also need to prepare for them. :folded_hands:

1 Like
3

Oh wow !! That is great to hear, I mean that the post was useful and it helped you to highlight the potential issues and challenges ! I also need to keep reminding people because it is easy to get super enthusiastic about it (I got really addicted, can’t deny) but then looking a bit deeper, you know AI is not there (yet!) so use with cautious I would say :joy:

QAs, Cybersecurity and Testers will be also a key in this new AI era !

2 Likes
4

I see LLM syndromes as a big issue if they are wrapping prompts around LLMs or triggering API requests to LLM in any part of their functionality.

Human testing is the key there.

Automating AI chaos will just give you faster AI chaos and unpredictable outcomes for customers.

2 Likes
5

Even better, we have AI analysis :sweat_smile: in some of our teams and it’s terrible :stuck_out_tongue:

So some PO’s are having meetings with business about their requests, they create a transcript of that meeting with AI and then paste that in AI to ask for user stories. Without reading through those, they literally copy paste those user stories into JIRA, development picks it up and it eventually goes into UAT with the business responding " this isn’t really what I asked for " :stuck_out_tongue:

A new meeting occurs and we get into the loop again.

1 Like