AI and Technical Debt

[rosie (Rosie Sherry)]

What are you doing to prevent AI technical debt from a quality and testing perspective?

When tests can be churned out so quickly, how have you seen this actually cause problems?

@christinepinto wrote about this recently:

AI-generated tests are creating a new kind of 𝘁𝗲𝗰𝗵𝗻𝗶𝗰𝗮𝗹 𝗱𝗲𝗯𝘁. And we’re about to drown in it.
LLMs can now generate 1,000 tests in minutes. Sounds great, right?
Wrong.

They’re making the same mistakes every time:
→ Writing trivial tests that only fail when you intentionally change things
→ Setting up 20 lines of mocks, then skipping the one line that actually matters
→ Testing how code works instead of what it does

The result? A 𝗺𝗮𝗶𝗻𝘁𝗲𝗻𝗮𝗻𝗰𝗲 𝗻𝗶𝗴𝗵𝘁𝗺𝗮𝗿𝗲 disguised as safety.

After 18 years in QA, I’ve watched teams struggle with too little test coverage. But we’re about to flip to a new problem: drowning in low-value tests that make every refactor expensive.

Douwe Osinga on the Block Engineering blog nailed it: “Writing tests has become 𝗳𝗿𝗲𝗲. Maintaining them hasn’t.”

This is the hidden cost everyone’s missing.

The same thinking that values test coverage percentages over bug prevention. The same thinking that makes teams afraid to change working code because the test suite will explode.

Not all tests have positive value. Some actively make your codebase harder to evolve.

The fix? Prompt LLMs with reasoning about test value, not just test coverage. Make them ask: “If this test fails, what did we just prevent?”
Because the goal isn’t having tests. It’s catching real bugs.

[ghawkes (Gary Hawkes)]

Well to be honest, we’re not using AI to create tests.

We may use AI to evaluate requirements/stories to come up with ambiguities, shortfalls and test scenarios as an input to our test creation, but ultimately the quality of our test cases is our responsibility.

It sure isn’t the LLM’s responsibility. If you live in the illusion that it is, then thats a scenario that tech debt is made for…but its the worst type of tech debt as you need to investigate what it created as you don’t quite know what it created - especially if your talking about 1000+ tests. So people then turn back to AI to come up with the solution to solve the tech debt problem they’ve just created with AI, still not taking responsibility for the quality problem and thus commences the cycle of AI insanity.

[larsthomsen]

It doesn’t matter who writes the tests, humans or AI, it doesn’t matter how the tests are run, manually or automated. If there are too many test cases, too many doubles, triples, no clear tracing, no structure in the test suites, no test of the actual requirement, you’ll get problems. In creating and of course in maintenance. And since AI can churn out giant amounts of tests in no time, it is bound to fill an existing problem (no clear tracing, no on point tests) with even more drivel.