I’m wanting to hear some of your experiences, both positive and negative with regards to the various testing activities and your CI/CD pipeline.
My current project has now deployed 6 releases (one each sprint) to customer test environments, which has made us acutely aware of issues in our current process. We currently have implemented - a suite of unit tests with good coverage that run on merge to master. However, all of our manual and other automated testing (only a handful of Playwright tests, contract testing still to be implemented) happen after merge to master, and as a result of ‘untested’ code being merged to master we often have a broken pipeline which can impact our deployments.
We also currently lack feature branch testing, and are looking to implement this soon - so I’m also interested in any tips for how to get this up and running as quickly as possible.
I’ve just watched a recording from Testbash that suggested:
Unit tests before allowing merge to master (as we have currently)
Integration tests immediately after merge to master
I think it very much depends upon all of the variables involved - time to run, risk, deployment cadences, frequency of test failure, etc. There’s no one size fits all response and it can require a bit of experimenting. I’ve moved pipeline stages to pre-merge before and then moved it back again.
For E2E tests I would probably want to run them more often than just once a night.
Unit and Integration tests run on every check in to main, but they don’t gate the merge itself. We allow our builds to be broken (but I think either process can work).
We run all of our E2E tests (we call them System tests) each night, but we also run a subset multiple times during the day, typically our sanity tests, plus anything that is currently in flux or of special interest.
Ahhh interesting! This seems so obvious now that you’ve said it but is not something I had considered before. Running the high priority/most important e2e tests more than once a night to ensure they are passing but that the test run time is small is a great idea! Thanks for your responses
The statement “Integration tests immediately after merge to master” says that you potentially will have a software with integration problems and the statement “e2e tests overnight” say that you can spend 24 hours with software with e2e problems (whatever e2e means).
I would suggest reading the booklet CD Pipelines and these two videos by Dave Farley:
Thank you! I’d stumbled onto his youtube channel yesterday and have watched a few videos so far, so will add these to the list!
Something else that I’ve started wondering - does manual testing have a place in CI/CD and where would it fit in? Exploratory testing is always especially valuable, but as it takes more time than automated regression, could it hinder a products’ ability to be delivered/integrated continuously?
We always had the rule before a branch could be merged that all test passes or that the results are similar (master branch merged into feature) with the master branch . This prevents that CI/CD work flows are broken and that no new issues are introduced with the merge and or that other new development in the master branch will not influences the new feature.
An approach that helped us is what i am going to describe in similar setup.
Our unit and integration tests used to run while building the code and creating the artefacts
We use to run whole stack locally ( using containers ) in ci pipeline and used to run limited number of ( bread and butter scenarios ) tests before deploying/merging code.
We had extensive suite of other priority tests that used to run on regular intervals
This ensured that we don’t break the code for critical issues and other priority issues used to get caught as well on regular interval.
Our devs used to run all of these build verification tests on their local as well before pushing code.
In my previous company we had 2 separate businesses with 2 separate build processes:
Older stack: Unit tests ran in CI and returned results to the PR. After a change was merged, it would be auto-deployed to Staging, which would kick off API tests and UI tests. This was rough. We were often playing catch up, debugging UI and API test failures → more specifically who introduced a breaking change to the application. We then had to either patch the application or update tests.
Newer stack: application ran in containers, installed a minimal database, ran all migrations, built the application, ran the application and then ran unit, API and UI tests. If there was a test failure it blocked any changes from getting to Staging. You found out on the PR.
Honestly the newer stack was way better. Bugs still made their way to Staging but thats where we would explore (testers and product managers). The short feedback cycle of automated tests meant it was clear right away if someone broke something and who needed to fix it.
Not sure what TestBash video you watched but I disagree. I’m a big fan of making sure automated tests are run as part of the build process because of the tight feedback loops. It’s much more costly (time consuming, frustrating, expensive) to run tests afterwards.
Having said this, tests in a CI is an iterative process. You start with unit tests because you don’t need a working application and then expand out from there. Its also depends on the type of application and how complex it is to get running in a pipeline. Perhaps the talk had some specific reason / context for why they were doing it that way?
You said your company lacks feature branch testing. What does that mean? Exploring changes on a single feature branch? Or running automated tests in the CI on a given branch?
Thank you for this response, it’s really interesting how you have explored the 2 different processes.
By ‘no feature branch testing’, I meant that we do have feature branches used by developers and unit tests do run against them. They can then only merge to master if the unit tests pass and code linting is successful. All other tests (UI/integration/manual) are then completed in environments spun off master, and if there are failures these can then either break the master branch or bugs can end up being deployed to customer environments as they now exist in master.
We are currently working towards moving our UI tests onto these feature branches so that we know key existing journeys are functioning correctly BEFORE new code is merged to master.
The remaining question is whether manual testing activities should also shift left and be completed in environments spun off the feature branch. Personally I think they should, as we are seeing too many small bugs deployed to customer environments as a result of them existing in master. There are some concerns though that manual testing on the feature branch is not representative. For example, imagine the manual testing on the feature branch passes successfully, but then on the point of merge to master there is now a merge conflict due to something someone else has merged which needs resolving. There are concerns that depending on the conflict, the manual testing would then be either pointless or require repeating. It’s an interesting problem that I can’t really fathom how often would occur without first giving manual testing on the feature branch a try.
In re-reading this, I’m thinking that really what we should do is to try and ensure increments of our product that are being merged are as small as possible to avoid large merge conflicts requiring repetition in testing.
I’m not sure what you mean by shift left, but my preference is to test new features on their feature branches. This generally allows you to test a smaller set of changes at one time. It also isolates changes, you can often test them earlier and get feedback earlier. More props if you can do this locally on your own computer because it also means when you find small bugs (think text changes / typos), you can fix them yourself.
The way I’ve done this is you do feature testing on the feature branch. If it looks good, I merge the branch into the main branch, which deploys it to some kind of staging environment and then I do a wider set of exploration on staging to see how the functionality works with other changes and more data. The goal here is to not repeat our testing per se, but to expand the testing on staging to account for known differences.
… testing on the feature branch is not representative
This is a foundational problem in testing. How do we make our testing representative? How do we design good tests given the situation, the set of changes, and our understanding? It’s different on feature branches, but I find the exercise in understanding what is different only makes my testing better.
Merge conflicts usually arrive because you are making too large of a change or those changes are staying open (not merged) for too long. Those are things you can fix with some process changes. (You can also help this problem when you test a feature branch locally by making sure you pull in the latest changes from master into the feature branch. Then push up changes. ) In theory if you are doing CI (continuous integration) well, you should be integrating changes continuously and thus avoiding conflicts.
With any change like this, it makes sense to start small or go for the biggest pain points. If you can pick up feature branches right now and start testing them, do that.
