Where do your ends start and end in E2E testing?

Test Automation
, ,
#1

I am interested to know where you consider the start and end of E2E or End-to-End tests?

I’ve seen many approaches to describe test Triangles, Pyramids, Cubes and also the concept Seems or Layers, where tests often cross system boundaries.

I consider myself a “Full Stack Tester”, in fact I think most testers Exploring or Automating systems cross many parts of a software stack with their testing.

From a product quality point of view, I like to think beyond the software stack with the real Start and End being the whole journey. Taking a user from their first interaction with an entity (company, open source community, individual etc) right though to when they use the product for the last time.

This wholistic view is useful form a Quality Process thought experiment, but still leaves me with questions from a Test Automation point of view.

So already in order to codify our testing we need to draw a boundary and define some scope that is smaller then the biggest possible end to end.

This gets smaller still when we start to consider what might be out of scope:

  • Interactions with third party systems
  • How the user experiences the system
  • How the system behaves offline
  • How the system works on physical hardware (e.g mobile phones)
  • The outputs of the system where they might cost money like shipping a real product

This is just a list of examples and is in no way complete. And yes, some of these have potential solutions, device farms might allow access to physical hardware to automate against for example.

So I come back to my original question and I am interested in your answers. Where do you draw the line in an End to End test, and how much of a users journey do you feel it covers?

2 Likes
#2

Hi Ben,

To me it’s a heuristic / rule-of-thumb thing.

Part based on the limits of your system landscape under test. Once I worked where we had 40+ named applications in the test environment. so E2E would be broadly across them.

Usually based on business flows. E2E would be from order entry to order implementation and billing.Or create a work item, process a work item… ship it. … etc. based on the specific business activities.

Again it matters if you test a public facing web app, a hospital system, public services case management, union/insurance claims, … it’s not necessarily a user journey. It’s more a person who has some thing that is processed… end 2 end.

/Jesper

4 Likes
#3

If it’s not including user account creating or whatever “onboarding” the app has, in the test, it’s not really end-to-end.
I do like to think of the E2E test design thing as a chance to look at the product offering from a distance. And understand how it all works when a user signs up using a web browser while a salesman on the other side is either giving them a license code, or whether they get the license code emailed to them. Just as an example, so E2E is a chance to step back and look at everything and then choose stripes or journeys as @jesper is describing. The most useful journeys are the ones a user will do every day, but the end of life journey and license cancellation/expiry journeys are financially valuable to the business too.

I recall when we worked together some time ago now @bencf1 and mobile experiences as part of a journey were brought in. Ever since that I’ve been a fan of understanding mobile as part of hybrid journeys which transition from desktop to mobile or the other way around. This for example helps me to design short mobile E2E tests that reflect customer use cases. I guess for me E2E is very very much about entire lifecycle, even to the exclusion of deep feature testing.

2 Likes
#4

I like to think of E2E in terms of User journey throughout the system, so looking at all the different types of users and ensuring that the full journey from first interaction to closing down of account or the complete end state.

I’m still working this through in my current role because the amount of paths which could be considered E2E is vast and there is low confidence between at the integration points between differing systems and components so there is a lot of push for a large amount of E2E testing which I hope to reduce…

4 Likes
#5

Hello @bencf1!

I agree with others that it is a journey through a system. It may be a user journey or it may be a data journey. I worked on a project similar to that described by @jesper.

In that project, data traveled through three or four subsystems and we were curious to learn if it traveled through with fidelity. Also, not all the changes required for each system were completed at the same time. So, we started with the first subsystem and verified the data. When the second subsystem was ready, we evaluated it. In this manner, it wasn’t so much an End to End test because we didn’t have a final End. We called it an End to Edge test. The End was the first subsystem and the Edge was the second sub system. When the third subsystem was ready, we moved the Edge.
The testing completed in earlier subsystems provided confidence in testing in the later subsystems (the later subsystems also did not have as much change).

I support @sjprior’s approach to reduce overall E2E testing. I think it is relevant for evaluating journeys that see high volume but I don’t believe it is generally useful. I prefer to use more unit and integration tests and fewer (much fewer) E2E tests.

Joe

3 Likes
#6

Just reminded by @devtotest that E2E can become quite evil if you don’t map it in terms of Edge-to-Edge testing. I recall a testing presentation that showed we would need as many as 4 billion testing paths through the product, to cover all components well. And it became clear that testing every environment E2E is prohibitively expensive and that defining behaviours of the “Edges” around the interfaces is far more useful. I suppose this idea of using “Edges” to reduce testing lends itself to “model based testing” tools. I still remain an E2E person, and prefer the developers to maintain their own unit tests with my encouragement and assistance, since unit tests are a valuable left-shift.

I’ve just used a dirty word there, so I’ll stop now.

3 Likes
#7

I’m interested to know what you think of giant automated end-to-end tests vs many smaller, granular tests?
E2E tests are the easiest things for business people (and Managers) to focus on but with so many points of failure they can be a headache to work with.
We struggle with mapping requirements to automated test cases, since expectation is often that 1 manual test case should equal 1 automated test case. Not often the case!

3 Likes
#8

Hello @danuk!

I believe giant automated E2E tests are a waste because long workflows change too often to make the tests useful, long workflows in non-production environments encounter errors (as you pointed out) which prevent valid and timely information about them, and automated E2E tests are more challenging to maintain.

I also believe that collaborating with business people (and Managers) to share challenges and understand what information is needed is a better test strategy than designing/implementing/executing/maintaining giant automated E2E tests. Those collaborations help explore concerns on both sides, and designs a way forward that gets people information they need.

Are they simple conversations? Nope. Can we complete them in a day? Nope. These conversations are the start of a working relationship to create respect, trust, comfort, and confidence in your products. I recommend starting with a small pain point, collaborating with a small group (one developer, one tester, one business person) and explore alternatives. Repeat and grow.

Joe

4 Likes
#9

Working with an API company, e2e has a narrower scope than those of you working with full web apps. e2e tests for me are that you can move through the workflow for a given API, which generally means the CRUD operations exposed via the API.

Up until a few years back, e2e meant we’d deploy to our staging environment, which had instances of SQL, mongo, third-party services, etc, and test the workflow there. More recently, we’ve moved to containerization, so you can effectively spin up the world anywhere. There’s some limitations here, in that the data stores, service dependencies, etc are also containers that are usually thin mocks (i.e. the SQL container usually starts with empty tables, the auth server returns yes for most tokens, etc), but there’s also advantages that it’s much easier to put the system into known states - i.e. if you need an account in a certain state, a particular downstream service to return a 404 instead of a 401, etc.

1 Like
#10

Thanks all, I’m learning some great things here so keep em coming!

The main thing I’ve learned:

  • End to end testing is typically a system test that follows a user or data journey through the system
  • The scope of the test highly depends on the system under test
  • There will be a vast number of possible paths through the system
  • Judicious selection of as few critical paths as possible is important to avoid high maintenance cost
  • Where possible it is preferable to have multiple smaller tests that cover sub systems and system integration over more E2E tests
  • Context is king and no one size fits all definition of E2E testing covers all usages

I’m looking forward to hearing more about what E2E tests people have successfully written and any cautionary tales!

My key action: Identify key user journeys though my SUTs and choose what what E2E means in context for each. Then work out what paths if any can’t be better covered by smaller integration tests.

2 Likes
#11

Summed it up well Ben!

1 Like