I’m a test automation engineer who’s been working with microservices for past few years.
From my personal point of view that one thing consistently frustrated me across different teams: E2E tests that span multiple microservices feel like they’re always one API change/new feature away from falling apart. The cost of maintaining them keeps creeping up until someone quietly decides they’re not worth keeping current.
So, at this time, I decided to build something real new for test automation engineers for better maintaining e2e testing for multi mciroservices — CratonAI — a tool that generates a runnable E2E test framework automatically based on how your services and business flows are defined.
Before this move, I have three questions I’d genuinely love your perspective on:
What’s the part of E2E test maintenance that eats up the most time — and is it something your team has ever tried to fix?
If a tool generated your test framework automatically, what would need to be true for you to actually trust it enough to run it in CI/CD?
Has your team ever scrapped an entire E2E test suite and started over? What led to that decision?
Critical feedback is just as welcome as positive — probably more useful at this stage.
Data is generally the most complex and important thing when it comes to e2e testing I find. Whatever solution you choose for handling data will likely have the biggest impact on test flakiness and / or performance.
I find this a difficult one to answer. I’m unsure how an AI building a test framework fixes the problem you presented. You say that the reason you are building CratonAI is because of maintenance, but the solution to this problem is an entirely new test framework? Wouldn’t the problem of changes requiring maintenance still exist?
Yes. Our old framework was overengineered and due to a reduction in QA resource the decision was made to streamline. Best decision I’ve made honestly.
On point 1: data management is something we’ve thought a lot about. In CratonAI, the test data logic is defined by the team themselves as part of the input — so the data setup is tied directly to the business flow definition rather than hardcoded into the framework. By now, the ideal process is to let user to identify the data structure and we create fake data.
On point 2: you’ve put your finger on exactly the right tension, and I want to be honest about it. You’re right that changes will always require some maintenance. What I am trying to shift is where that maintenance happens — instead of engineers manually updating test code when a service changes, the idea is that updating the service definition in CratonAI regenerates the framework automatically. So the maintenance burden moves from “fix broken code” to “update a configuration.” This is my current thought to resolve it.
On point 3: “overengineered” is something we heard a lot. Just curios what made you decide to streamline rather than refactor?
Always, be ready to start over. But be sure you understand why you are starting all over again.
Most often the reason I start over is sheer weight of technical debt, and that’s where the temptation to auto-generate tests that do not actually deliver direct customer value start to slow things down. Customers do not buy or use your framework, they consume the experience. And that is harder to test when you are stuck thinking about architecture. Architecture really needs it’s own tests, and that’s where generators are great, but E2E really has to be architecture agnostic and very small, so that architecture, can be big. All of which makes it easier to scrap your E2E and start over, if you separate it from your architecture testing.
Thanks @conrad.braam . That’s really a sharp point of view from the true E2E testing.
There’s a fundamental difference between testing architecture and testing customer experience. What I am trying to do with CratonAI is closer to the latter — the input isn’t service endpoints or technical contracts, it’s business flows defined by the people who understand what the customer should experience. The idea is that the generated tests follow the user journey, not the architecture underneath it.
But your point about keeping E2E small and architecture-agnostic is something I want to think about more carefully. Thanks again!
I have always been a fan of testing at the user level. Even though I am a nuts and bolts coder at heart, the customer is always king. It’s a test surface that is tricky because some applications are embedded/native client and some are web-hybrid-mobile clients. So E2E needs to be able to move between the client types quickly, and even between UI frameworks. Nothing stands still for long. For example teams who suddenly dump web-client apps and port their C# apps to Linux with the new .NET 10 , might force you to move your E2E test tooling with them to Linux and MacOS in a hurry. That’s been my driver behind using more than one tool for any high-level test system. Hedging your bets between high level visual testing tools and lower level tools that inspect and trigger button click events via hooks.
But that does mean not testing being able to test “all-of-the-things” at every level. I’ve moved away from end-user product coding, but am encouraged to see new contract frameworks for systems evolving. That may be the big area that software engineering moves into, where our imagination and AI work together to write code. Writing code is still harder than it needs to be, shipping working and tested code has not sped up much in the last decade. AI will eventually change that, so test tools that move faster are needed, to get us through the code bottleneck as it opens up in the next few years.
1- Depends on your framework that you are using, if you truly separate the methods for testing your Microservice(s), then the change is focussed
2- No, We generate test scenarios based on User Journey, and add the ‘random diruptive tester’ into the mix… (In Ways only QA know how!)
3 - Many times we have scrapped and started agian - this is due to finding better / more effective approaches to testing. Technically we do no tscrap tho - we juste rewrite coverage.
As for issues with updates to your microservices - I would consider looking at the ‘contract’ for use and have it ‘consumer’ based, hence a break in contract will identify where it will break your E2E basically (PACT). ALWAYS test the Microservice in isolation with scenarios based on the consumer.
“overengineered” is something we heard a lot. Just curios what made you decide to streamline rather than refactor?
Refactoring would have taken a lot more time. The complications in the framework were embedded in every test which would mean updating every test (or altering the methods used to be less complex and still functionable with their current implementation / parameters).
It was genuinely faster in my opinion to create a new simplistic framework which did what we needed it to do, and then replicate the coverage we had before.