Where do your ends start and end in E2E testing?

Questions
, ,
#1

I am interested to know where you consider the start and end of E2E or End-to-End tests?

I’ve seen many approaches to describe test Triangles, Pyramids, Cubes and also the concept Seems or Layers, where tests often cross system boundaries.

I consider myself a “Full Stack Tester”, in fact I think most testers Exploring or Automating systems cross many parts of a software stack with their testing.

From a product quality point of view, I like to think beyond the software stack with the real Start and End being the whole journey. Taking a user from their first interaction with an entity (company, open source community, individual etc) right though to when they use the product for the last time.

This wholistic view is useful form a Quality Process thought experiment, but still leaves me with questions from a Test Automation point of view.

So already in order to codify our testing we need to draw a boundary and define some scope that is smaller then the biggest possible end to end.

This gets smaller still when we start to consider what might be out of scope:

  • Interactions with third party systems
  • How the user experiences the system
  • How the system behaves offline
  • How the system works on physical hardware (e.g mobile phones)
  • The outputs of the system where they might cost money like shipping a real product

This is just a list of examples and is in no way complete. And yes, some of these have potential solutions, device farms might allow access to physical hardware to automate against for example.

So I come back to my original question and I am interested in your answers. Where do you draw the line in an End to End test, and how much of a users journey do you feel it covers?

2 Likes
Where to trigger E2E test suite in microservice land?
#2

Hi Ben,

To me it’s a heuristic / rule-of-thumb thing.

Part based on the limits of your system landscape under test. Once I worked where we had 40+ named applications in the test environment. so E2E would be broadly across them.

Usually based on business flows. E2E would be from order entry to order implementation and billing.Or create a work item, process a work item… ship it. … etc. based on the specific business activities.

Again it matters if you test a public facing web app, a hospital system, public services case management, union/insurance claims, … it’s not necessarily a user journey. It’s more a person who has some thing that is processed… end 2 end.

/Jesper

5 Likes
#3

If it’s not including user account creating or whatever “onboarding” the app has, in the test, it’s not really end-to-end.
I do like to think of the E2E test design thing as a chance to look at the product offering from a distance. And understand how it all works when a user signs up using a web browser while a salesman on the other side is either giving them a license code, or whether they get the license code emailed to them. Just as an example, so E2E is a chance to step back and look at everything and then choose stripes or journeys as @jesper is describing. The most useful journeys are the ones a user will do every day, but the end of life journey and license cancellation/expiry journeys are financially valuable to the business too.

I recall when we worked together some time ago now @bencf1 and mobile experiences as part of a journey were brought in. Ever since that I’ve been a fan of understanding mobile as part of hybrid journeys which transition from desktop to mobile or the other way around. This for example helps me to design short mobile E2E tests that reflect customer use cases. I guess for me E2E is very very much about entire lifecycle, even to the exclusion of deep feature testing.

3 Likes
#4

I like to think of E2E in terms of User journey throughout the system, so looking at all the different types of users and ensuring that the full journey from first interaction to closing down of account or the complete end state.

I’m still working this through in my current role because the amount of paths which could be considered E2E is vast and there is low confidence between at the integration points between differing systems and components so there is a lot of push for a large amount of E2E testing which I hope to reduce…

6 Likes
#5

Hello @bencf1!

I agree with others that it is a journey through a system. It may be a user journey or it may be a data journey. I worked on a project similar to that described by @jesper.

In that project, data traveled through three or four subsystems and we were curious to learn if it traveled through with fidelity. Also, not all the changes required for each system were completed at the same time. So, we started with the first subsystem and verified the data. When the second subsystem was ready, we evaluated it. In this manner, it wasn’t so much an End to End test because we didn’t have a final End. We called it an End to Edge test. The End was the first subsystem and the Edge was the second sub system. When the third subsystem was ready, we moved the Edge.
The testing completed in earlier subsystems provided confidence in testing in the later subsystems (the later subsystems also did not have as much change).

I support @sjprior’s approach to reduce overall E2E testing. I think it is relevant for evaluating journeys that see high volume but I don’t believe it is generally useful. I prefer to use more unit and integration tests and fewer (much fewer) E2E tests.

Joe

3 Likes
#6

Just reminded by @devtotest that E2E can become quite evil if you don’t map it in terms of Edge-to-Edge testing. I recall a testing presentation that showed we would need as many as 4 billion testing paths through the product, to cover all components well. And it became clear that testing every environment E2E is prohibitively expensive and that defining behaviours of the “Edges” around the interfaces is far more useful. I suppose this idea of using “Edges” to reduce testing lends itself to “model based testing” tools. I still remain an E2E person, and prefer the developers to maintain their own unit tests with my encouragement and assistance, since unit tests are a valuable left-shift.

I’ve just used a dirty word there, so I’ll stop now.

3 Likes
#7

I’m interested to know what you think of giant automated end-to-end tests vs many smaller, granular tests?
E2E tests are the easiest things for business people (and Managers) to focus on but with so many points of failure they can be a headache to work with.
We struggle with mapping requirements to automated test cases, since expectation is often that 1 manual test case should equal 1 automated test case. Not often the case!

3 Likes
#8

Hello @danuk!

I believe giant automated E2E tests are a waste because long workflows change too often to make the tests useful, long workflows in non-production environments encounter errors (as you pointed out) which prevent valid and timely information about them, and automated E2E tests are more challenging to maintain.

I also believe that collaborating with business people (and Managers) to share challenges and understand what information is needed is a better test strategy than designing/implementing/executing/maintaining giant automated E2E tests. Those collaborations help explore concerns on both sides, and designs a way forward that gets people information they need.

Are they simple conversations? Nope. Can we complete them in a day? Nope. These conversations are the start of a working relationship to create respect, trust, comfort, and confidence in your products. I recommend starting with a small pain point, collaborating with a small group (one developer, one tester, one business person) and explore alternatives. Repeat and grow.

Joe

4 Likes
#9

Working with an API company, e2e has a narrower scope than those of you working with full web apps. e2e tests for me are that you can move through the workflow for a given API, which generally means the CRUD operations exposed via the API.

Up until a few years back, e2e meant we’d deploy to our staging environment, which had instances of SQL, mongo, third-party services, etc, and test the workflow there. More recently, we’ve moved to containerization, so you can effectively spin up the world anywhere. There’s some limitations here, in that the data stores, service dependencies, etc are also containers that are usually thin mocks (i.e. the SQL container usually starts with empty tables, the auth server returns yes for most tokens, etc), but there’s also advantages that it’s much easier to put the system into known states - i.e. if you need an account in a certain state, a particular downstream service to return a 404 instead of a 401, etc.

1 Like
#10

Thanks all, I’m learning some great things here so keep em coming!

The main thing I’ve learned:

  • End to end testing is typically a system test that follows a user or data journey through the system
  • The scope of the test highly depends on the system under test
  • There will be a vast number of possible paths through the system
  • Judicious selection of as few critical paths as possible is important to avoid high maintenance cost
  • Where possible it is preferable to have multiple smaller tests that cover sub systems and system integration over more E2E tests
  • Context is king and no one size fits all definition of E2E testing covers all usages

I’m looking forward to hearing more about what E2E tests people have successfully written and any cautionary tales!

My key action: Identify key user journeys though my SUTs and choose what what E2E means in context for each. Then work out what paths if any can’t be better covered by smaller integration tests.

3 Likes
#11

Summed it up well Ben!

1 Like
#12

Loved reading this thread! I have been avoiding big scary icky automated E2E tests for eons but have found myself thrown in by some new execs. I’ve given up on talking people out of it. They essentially worked around me by hiring contractors, and made the genius move of, “Eric, make this a success and oh, do it our way…big icky automated E2E tests that take 4 hours to run and catch all the bugs…we don’t like bugs…but we will clap if you show us a ghost clicking around a web page…we love that shit.”

If you can’t beat em, join em. That’s where I am. So…I’m also thinking about where E starts and ends.

2 Likes
#13

So let’s get back into the weeds…

Do you think it’s better to have a looooooong E2E test, checking (let’s say) 10 components, or two shorter E2E tests, each checking 5 components?

It seems to me, we want one longer one. Even with E2E tests, it seems to me, we still want to follow the Xunit test pattern of making each test mutually exclusive. And in that case, two distinct E2E tests lose data fidelity.

2 Likes
#14

If you can shortcut to start a test part way though by setting the system state in a direct way, use that!

Noddy example, if login can be done via the UI, but all it does is generate a cookie or token.

Do a login test or four (valid, wrong password, no password, wrong user).

But all other tests could do a http request directly to auth endpoint (e.g not Selenium), and use the cookie.

A similar pattern can be done to make other “end to end” tests mutually exclusive.

In this way you build and end to end test suite, rather then a single test. This means if something fails, you have a better idea where the bug is.

Does that make sense?

2 Likes
#16

Very good question, breaking up into separate tests? Does that still make it a smoke test? That depends on if you are happy to ship if one of the 2 tests fail. If not, keep it in one test - I’m hopeing the answer is one, if it was not, then the other test/check you care less about or has lower risk is probably not belonging in the E2E Smoke. There is nothing wrong with triggering a second batch/suite of tests, but having one simple headline test is easier to maintain when deadlines approach. Do not add anything that does not block your “mission”, to the smoke/E2E. But for example if something really stupid happens like, example the company name does not appear in the title, would you still ship it? If even a small detail is mission critical, you probably want to fail, as a way of making sure we never make that mistake again. Fail early, fail often. (I belong to a sect of testers that believe a continuously green board is lying.)

I had a long look for James Bach references to end-to-end theory, and actually cannot find any explicitly. I wonder why that is.

I second the suggestion by Ben, cut out some repetitive stages like the test setup fixtures, or break up into a suite of tests, and experiment a lot. Remember the goal of automation is to unblock the coding pipeline, and use automation to do it so you don’t have to stress. You will be trying to find a line between the stress of fixing flakey tests, and simply bypassing/working around any “untestable” things that are flakey in your E2E. A suite of tests that approach different parts can make it easier for people to know what to do when just one test in a small suite fails. It’s easier to mentally triage one small test in a suite, than one huge test. Then, if for example your testing framework does not handle reboots well, work around it by not rebooting and fix the rebooting problem later on. Don’t be afraid to test everything, but also don’t be afraid to look at alternatives that buy you time if they come with very low risk.

Finally make sure people know about any workarounds you use in automation, speak about the gotcha often. You might be surprised someone comes to help fix it if you can show value with the rest of the system.

#17

Thanks for looking for JB E2E posts, @conrad.braam. Sounds like you and @bencf1 agree, keep your E2E tests mutually exclusive. My instincts say the same. But lately I’ve encountered seasoned automators that are building dependent tests and can’t think outside that box.

Anyone use the term Steel Thread? I am finding this useful when trying to get others to think about what may or may not be E2E. And to further this concept, I created a notation to help with E2E brainstorms: a table with two columns labeled “One End”, “The Other End”. Each row represents a “test”. The implication is, don’t over think the middle. Get from this end to that end as simply as you can.

2 Likes
#18

Thanks I’ll take a read of this. I’ve never heard the term before!

#19

Will also have to read that about Steel Thread. I am pretty sure context is important in any strategy decision. I’m more often than not a fan of being happy with your strategy than being sure it’s the right one.

It’s possible for teams to succeed using tools and techniques they believe in, rather than for teams to succeed using the correct tool but not liking the tool or the process. You will have to find that balance, and for me the balance is along the lines of covering as much of the critical path as makes sense versus time cost and maintenance cost. Steel threads IMHO might work when you have multiple testers, but if it’s just you, a test that covers a feature properly can be more useful. Here is an example:

  • Suite A: Install app, set it up and run a basic test. [10 minutes]
  • Suite B: Install app, set it up and run a basic test, stop the app and uninstall. [11 minutes]
  • Suite C: Install app, don’t run it, uninstall the app. [3 minutes]

If you never expect users to be able to uninstall, the correct answer is “A”, but for an extra confidence that an upgrade will actually stand a chance, “B” is much more useful. It’s a slightly longer “thread”, and easier to maintain if done right. It’s value is more complicated to prove though. Whichever route you choose, your choice will be correct for your context though.

1 Like
#20

In my last role, we didn’t do E2E testing; indeed, a CIO moaned at me for looking at systems that were outside my immediate remit (“test this project and this project only” vs. “but these updates to the company website don’t work”).

When the new app was deployed to the network, it took a while to realise that when the spec had been drawn up, no-one had looked at the API it was supposed to hand account data off to. (All those people were either contractors, consultants, or had left the company.) As a result, there was about £3 million’s worth of business stuck between the app and invoicing.

Oh, what fun we had.

Ever since, I’ve always insisted on hands-on E2E testing following the data, from initial user to final accounts payable. As someone once said, “Follow the money.”

2 Likes
#21

E2E / End to End testing done by qa testing services, is carried out with the intent to simulate how end-user interact with a product.
When to start E2E testing:

  • Test environment should be available
  • Correct test data should be available with QA team
  • Test cases should be cover end-user scenarios
  • The testing tool should be installed and ready
    When to end E2E testing:
  • All test plans should have run and passed
  • No Critical or High severity defect should be Open
  • High-risk areas should be tested completely
  • Requirement coverage should be met
1 Like