This is a useful question as I reckon it helps us reflect on our relationship with automation and the context we are currently in.
Some replies so far on Twitter:
It doesn’t matter. Do whatever automation is appropriate at your context.
I am going to embrace the ambiguity in ‘ends’.
I’ve not really considered the ends the most important part, more the connections. There’s an entry and a goal, sure - but there has to be pieces working together in between.
How about your thoughts on this question: What are the ends of End-2-End tests?
That is mine
You can ask me for details.
Nailing down each end of the a string is pretty hard and time consuming, since for me, the start is totally “zero”, absolutely
clean sandbox, and the
far end is to verify that we got back to that totally
clean sandbox state again and left minimal traces. Both are often time consuming to assert.
- I’m usually far more interested in the bits in between because the longer or more circuitous you can make that route, the more coverage you get for that one journey in an end-to-end. It obviously helps to have any test only do one thing (aka smoke test), but when you want to release test, it makes sense to create a test in the suite that covers a full user journey and not just snapshots or segments in isolation. But the ends, are something I turn into fixtures.
But in a philosophical sense, the ends are really a goal. For example making sure that customers can pay you.
There are 2 schools of thought here:
- Keep tests as small as possible to run them in parallel as fast as possible
- Do the actually end-to-end to make sure that the full functionality actually works
I worked on a project a handful of years ago where new product data was entering an existing data processing stream. There were four or five basic systems in the processing stream and the business team was very focused on specific outcomes for each system.
My testing team created an “end to edge” strategy where we could demonstrate an outcome in a system, and then continue. That is, the product data was presented to the first system, then the first and second system, and so forth. In this manner, we could isolate errors to the most recent “edge”.
Only when the demonstration was accepted did we move on to the next system or “edge”. We advanced our way through the systems in this manner until we came to the last system or the “end”.
Echoing the “It doesn’t matter. Do whatever automation is appropriate at your context.”, I’d add that the important part is that the team you’re working with has a common understanding of what it means to them. The details after that don’t matter.
We started running into this a a few years ago, when a new dev wanted to add a task called “intTest” to our Gradle build scripts, without realizing that the term was already in use in our code, and meant something different than what they wanted. Coming up with a working definition for the team is what mattered, not whether what was called “intTest” in the code was what the individual believed was an integration test.
We actually ended up settling on the second Twitter response, to embrace the ambiguity in the terms, and instead of trying to build out an index of terms, we’d focus on being more expressive when describing things - i.e. this is an end-to-end test, where it goes from the API and related micros running in containers against a stubbed database and mocked dependent APIs". There are just too many permutations now a days of how you can run/test code and dependencies to be captured with just unit/integration/e2e tests.
These can be complimentary.
Take loads of option 1, check as much functionality as possible in a cheap, fast way.
Take as only what is needed of option 2. Say between 1 and 5 user journeys though the system.
Excellent point! You are correct, the combination would probably be the best approach.