As a strategy, I’m a fan of minimising the number of end-to-end1 tests a team owns and runs. However, this advocacy for less testing can be a difficult sell. Software Engineers and testers express their reservations about adopting such a strategy.
I once remember working with a team and we grouped functionality into red, amber and green. Red functionality was absolutely critical to the success of the business. So the team reviewed our UI automation test goals for our full site regression pack and set out to build just the red ones. In a sense, we did less testing as we could’ve gone all in with both the orange and green areas of functionality.
Have a read of Anne-Marie’s article. What do you think? What “less testing” strategies have helped and in what contexts? How have you and your colleagues decided to remove end-to-end tests?
I definitely think figuring out how the team feels about the end to end tests is my preferred strategy in general.
In my current project, we know we’re going to have a good number, but that’s for a few reasons:
We have an incredibly complex project, so there’s a lot of tests at every level, and we’ve found that often times there are whole swathes of unit tests that don’t seem to do much.
Due to the first bullet point, we actually see the end to end tests as documentation for the application in some ways.
Also due to the first bullet point, we’ve found a good swath of bugs at the end to end level, even with solid unit and integration test suites!
I like the concept of the red, amber, green - I do something very similar but just 1-3 but going to adopt your system haha.
However, feels like when you say you’re doing less testing you’re not necessarily working smarter it seems (as in covering the various conditions in fewer tests and working efficiently), but literally choosing to do less testing. This is fine provided the risk is low and we get buy in from folk.
Do you use evidence to knock tests into amber, green (example, History of not failing?). Do you run the amber /green tests perhaps once ever N releases just to be certain etc. Who’s involved in the categorisation of the tests?
I’m an advocate for reducing the number of tests and have employed similar risk based techniques in the past with heavily reduced manual UI flows, bolstered with different levels of Automation.
However, my current project is all about data reporting. The end to end tests in this project are extremely complex as the data passes through various levels of infrastructure and transformations. Automating this would not be impossible, but would probably end up a heavier workload than manual (with tools) testing.
Additionally the risk level is hard to break down to a red, amber, green type grading as the purpose of the project is data reporting. The data needs to be 100% accurate. We can’t feel confident with this unless we do have a high coverage of manual data interrogation.
My worry is when upper management hear about these types of techniques and think they can be applied to everything. The reality is not everything can be reduced.
…. Well everything can be reduced, but how much do you want to wage your job on that risk?
Anne-Marie always talks sense, imo - this article in particular is great.
We are going through a process of putting our e2e tests into Github Actions - they run quicker and so give us quicker feedback - but there is a huge challenge of how these e2e tests are arranged- its interesting work.