I’m currently working on developing a test automation framework for a complex, multi-layered web application, and I could use some advice. The application I’m working on has a lot of moving parts – a frontend built in React, a backend API layer, and several microservices that interact with external systems. We aim to establish a framework that can cover end-to-end testing, API testing, and some UI testing. I want this framework to be reliable, scalable, and capable of providing actionable insights to our development team.
Here are a few of the questions I’m wrestling with, and I’d appreciate your input:
Framework Recommendations: Based on your experience, what testing frameworks would be best suited for a multi-layered application like this? I’ve considered Selenium for the UI layer, but I’m curious if there’s something more recent or better suited.
Best Practices for Structuring Tests: How do you typically organize your test cases and scripts to ensure modularity and reusability? Any specific patterns or folder structures you’d recommend for ease of maintenance?
Testing Microservices in Isolation: We’re looking at testing individual microservices both independently and as part of the full application workflow. Do you have any recommendations on tools or strategies that make it easier to isolate and mock these services for testing?
Integrating with CI/CD: I’d love advice on the best way to integrate this framework into our CI/CD pipeline. Are there any best practices you’ve found for ensuring tests run smoothly in CI, especially for a mix of end-to-end, API, and UI tests?
Monitoring and Reporting: Finally, how do you structure your reporting to ensure the results are easy for developers to interpret and act upon? Any tools you’d suggest for generating clear, actionable reports?
Thanks in advance for your insights. I’m excited to hear how others have approached similar challenges and to learn from your experiences!
Hey,
I’m not that sure you’re asking the right questions at the right time.
I’d ask things like these before moving to the actual tool:
who will be the PM/architect going to manage it? Are they having enough knowledge of solution design, frameworks, programming languages and tools, CI/CD in order to lead this project?
what resources do you have for creating, leading, coding, maintaining this new project? timeline, milestones, people, knowledge, servers, repository? will the devs. be actively involved in coding in this project?
what testability is there for the application at the moment? have you made a few scenarios/analyzing how you might do the interaction from the automation framework to the actual product?
what environment are you running from/to? embedded stuff? on a mobile device, locally, on a virtual environment, server, production, pre-prod, test environments?
what will you manage the environment build/destroy, data generation?
what’s the risk/priority/urgency of this framework compared to others in the context of quality related practices in the team/product/company?
who do you make this for? or for what kind of problems do you think this is a solution in your current project?
is the product highly/actively changing that testers also can barely keep up, is it generally stable, or in maintenance mode?
I used several frameworks/tools across different types of products, and most can achieve what you want. I think planning well, putting yourself in the boots of an architect, project manager, lead developer and balancing a budget of time and resources to create something relevant for the long term is way more important.
The questions here are broad and deep enough that answer to each one could easily fill a book. I’m afraid that’s one of the reasons you didn’t get too many answers so far. You might get better response rate if you are able to reduce your problems to few specific choices, or more focused questions that can be answered in couple of paragraphs.
Separate frontend and backend are pretty standard architecture these days. I am curious about these microservices - are they deployed independently from backend? How does backend communicate with them? or is communication coming directly from frontend? What are these external systems - are they external to your team, but still owned by your company, or are they external to both team and company (things like Slack or GDrive integration)?
I’m now going to give a short answer to each of your question, based on my experience, but while reading it, keep in mind that I know nothing about your company, project, team and problems you are trying to solve. My experience might come from completely different place than yours and give you more harm than good if you try to apply it blindly. Caveat emptor.
I’m sure there are frameworks that can cover everything, like Robot framework, but I don’t have experience with these. Teams I’ve been part of always picked specific tools for specific interactions and created a custom framework to integrate these tools and provide things that are commonly needed. So it would be more about programming language choice than anything else. Python, Java, C# and JavaScript/TypeScript are all good enough.
For UI layer, I see only two reasons to pick Selenium over Playwright:
philosophical - Selenium implements actual W3C standard, it operates with real browser that can be installed through standard means, it is open source project with large community
it’s easier to find people with Selenium experience than with anything else, simply because it was around longer. That is probably changing fast.
First, tests should be separate from actual framework. Tests should use the framework to interact with the product, but your goal should be that you can change parts of framework, or switch one library for another, with minimal impact on any test code.
It doesn’t mean that tests should be in separate repository than the framework, they can be both in single place. And I would start there.
The internal structure of the framework / library / helpers should make sense for your project, make it easy to decide where changes should be implemented, and provide API that makes sense from test writing perspective. Yes, this is pretty generic answer, but I don’t think anyone could give better one without first learning a lot about your project.
You might consider splitting tests from library into separate repositories. This way tests could exist along with the SUT, and would pull in library as a test dependency. However, this introduces complexity as now you need to care about external library API and correct versioning.
If you have trouble testing microservice in isolation, then it is not a microservice. You need to solve that problem first.
Microservice should have good unit / integration / e2e coverage for all the internal stuff. It needs clearly defined API for communication with outside world. For communication with software that your company owns, consider contract testing.
Take a step back and consider what do you want to actually test and how the entire product is going to be deployed. For e2e tests you want environment that is close to production environment, and for sufficiently complex product that will mean long installation times and very high hardware costs. Look for ways to avoid these costs.
The main question is your worker target (bare metal / VM / containers) and orchestration (it’s harder if you need couple of VMs than when you can do with one). All CI systems worth their salt allow you to bring your own workers if you have to, so specific choice does not matter here as much.
Green checkmark right in PR page means no action is needed, red cross means you need to go into report and try to understand what is going on.
It’s generally good to try to give each test one reason to fail, and use a framework that has nice assertion failures reporting, but in my experience people will need to get the actual experience in running the framework and debugging the failures. Try to give as much information as possible (store the logs from backend, save HAR of client requests etc.), but don’t try to be smart and predict what is the actual reason for failure. If you already knew all the ways a software may fail, you would protect against them directly in code and you would not need these tests at all.
,
I’ve considered Selenium for the UI layer, but I’m curious if there’s something more recent or better suited.