How do you debug automated e2e tests?

If you have end-to-end tests integrated into your CI/CD, how do you debug them?

I know Cypress has video playback, but it seems like there’s not much support for video playback in Playwright or Selenium.

Or, is everyone using Sauce Labs or Browserstack which has those sorta features?


Great question, I am really curious about how others debug e2e tests?


Playwright does have support to record videos. I recently set up a GitHub Actions pipeline that ran Playwright end-to-end tests. It recorded videos for all of the test cases, and retain the ones for test failures as artifacts that we could review. This is how I typically debug these tests on CI.

When working on the tests locally, most end-to-end frameworks have decent debugging tools that can help you deal with most issues. With Playwright I heavily use the VS Code extension which works really well. The inspector tool is also great. Most of the other E2E tools I’ve used (TestCafe, Cypress, etc.) have similar debugging tools. I’ve found that I don’t need much else.


This is going to sound harsh, and yes debugging tests on any platform is hard if the developers don’t help you to help them. I’ve had 5 or 6 tester jobs, and much of my day job has been about easily capturing traces, not about writing tests.

3 things you want as artefacts in every test.

  1. Does your app have any logging? I’m assuming it’s a browser app here, but it still has tons of logging, talk you your devs and get the JS hooks you need written so you have logging, you win, they win.
  2. Screenshot I assume you have a screenshot to go with the stacktrace?
  3. DOM snapshot. A snapshot of the DOM immediately after the test starts dying.

Backend logging to show state of the session/activity and loads of timestamps also super helpful. Everyone will now squeal and say our app has no logs. Well in that case you app has no bugs, ergo, your app does have logs, you just don’t know how to find them. I’ve released software before without any logging, it had a very low bug count, and almost no functionality. The biggest feature in the follow up release was a silent feature, you guessed it, logging and a way to pipe the logs right into the automated test system. Learn about Log4J (dont use it it a security minefield) we use a variation on that same pattern and it’s awesome. Make sure that any exceptions your test throws are not just ‘ElementNotFoundException’ that’s pure laziness. A good stacktrace with “context” to go with it, beats a video hands down, hardly ever found videos useful except for slick sales demos.


I run our Playwright tests locally. I run the tests locally when I write them and when I fix them. Playwright also has Trace Viewer which I use locally and in CI to debug issues: Trace viewer | Playwright.

1 Like

And I think I should add that Trace Viewer is brilliant! It gives me so much information. It is better than a video because I can see logs and inspect the page at each step.


Nice, I didn’t know about inspector tool, thanks for sharing that - does it support breakpoints?

No, it is for use when the code has run.

Is this post just about getting feedback from others on the topic?

Or is this also a discussion about something lacking regarding automated e2e test execution, within the context of CI, when there is no video recording? If the latter, would like to hear about the lacking aspects.

As far as I’m aware, you can always manually trigger rerun of (failed) CI/CD tests and debug from there, hoping that failure reproduces. The pain for this scenario is when you have flaky tests or or flaky bugs that don’t always reproduce across test runs consistently.

And when you manually rerun CI/CD tests, depending on the setup, you should be able to visually see the automation in action unless you use an entirely headless test setup, but I assume such a headless test setup won’t get you much by way of video recording and screenshots as well?

I think I recall some test frameworks have a screenshot on failure option, that should help compensate for lack of video recordings. I don’t recall the specific framework or tools with the feature though.

Selenium (Grid) also has video recording support, but not natively, you have to install/setup some plugins or use the feature that is part of some test framework. But it is there should you really want it to use it, just have to research around.

And perhaps an expansion of this discussion here, is how do you debug automated e2e mobile app tests, for case of simulator runs or running on devices in a device farm/lab, etc.

Or even automated e2e tests of desktop GUI apps, is that even a thing anymore?

I agree with Comrad. I have to debug our Selenium E2E tests on the CI/CD pipeline and I need:

  • Logging. Know what your tests are doing
  • Screenshots. Get the last “status” of the tested page when it failed.
1 Like

The debugging support was the biggest reason we picked Cypress ~4 years ago (versus Protractor at the time…given the latter is now deprecated I’m even more glad we went that route!)

In addition to the video playback and screenshots captured automatically on test failure (though I see Cypress 13 made it a little harder to keep video if you’re not using their “Dashboard” cloud-based feature), it’s also possible to:

  • Use cy.pause() to have the test pause anywhere in your test code, and optionally single-step from there.
  • Use cy.log() to log additional information in Cypress’s log panel on the left side of the test runner
  • Use console.log to print to the browser’s Dev Tools console (make sure you have it open first!)
  • Use the Browser’s dev tools to inspect the state of the page, either at the end of the tests or while paused with cy.pause–super useful for figuring out selectors, etc.
  • After the tests finish running, view “snapshots” of the page state by clicking on the actions in the Cypress command log.
1 Like

I find using Playwright inspector & VSCode breakpoints to work quite well. The inspector apparently stops automatically at each first point where it calls the Playwright api (goto method for us in particular), which can be a little annoying but once you get over that it’s fine… And then when you reach a breakpoint in VSCode it stops & you just go to that window in your apps.

In a CI pipeline I’ve used WebDriverIO.

Never had a chance to get video working, although that would be preferred. Always relied on logging and screenshots. That was usually enough for me but not for the other developers. That’s where CypressIO has really spent their time making things smooth.

Generally speaking, if I’m debugging tests locally, I use Visual Studio Code and then configure my setup so I can set break points in my code. I’m sure other frameworks have a similar process.

1 Like

A few aspects to consider here that can quickly enhance your debugging process.

  1. Framework language choice - With languages like Ruby and Python, you won’t have the asynchronous challenges that occur with JavaScript for example. This means that you can run and effectively pause and then continue your tests mid-flow. You can use the interactive console in your IDE to ask questions and test values of variables and constants in the code as well as page object presence and state in the UI for example.

  2. Adopt a reporting framework that gives you some kind of historical test result pattern analysis. It’s all good and well to know that a test has failed but how many times in succession has it failed and has it failed for exactly the same reason every time. These are the ‘burning’ tests that want your attention first.

  3. IMO, video playback doesn’t add that much more significant value to screenshot capture. If you have screenshot, stack trace and analytics for all failing tests as per point 2 above, you should be able to easily and quickly debug your tests.

  4. Sauce Labs and others etc are great for runtime coverage and raising non-functional bugs but not so much for quick debugging. That needs to be done locally. You’ll want to integrate with these other tools only once you have a nice clean set of well performing tests in your own local pipeline.


Welcome to the Club James.
Yes good wrap-up of the high level triage, I’m quite keen to know how @ianjennings started to overcome the problem of not being able to run in a local environment. It’s a common issue even for non-public-cloudy test setups, for me the right thing to do is look at what point in time and other environmental change the fault occurred, by having that good history dashboard.

Playwright’s trace view is provides snapshots of all the actions it went through.
Thus, allowing you to see in what state the DOM was when the tests failed.

Currently, I’m trying to implement an email reporter in Playwright. Lets see how it goes.

Welcome and thank you James!

I appreciate you comment about recordings and screenshots. Its one tool in the shed. But often really good logging will uncover the issue. But I also want to call out your comment on “What and When” a test fails. Because the temptation is often to dive in and thrash about in order to quickly solve the problem. Stepping back and observing the result over time can provide a lot of information to direct the next steps.

Is the test suddenly failing every time when it worked before? (Could be a product code change thats not being accounted for? run the last known good AUT branch is a task to consider

Is the test occasionally failing over several branches? Could be an infrastructure issue, timing, or something unrelated to the code itself? also known as: internal screaming when everything works locally but fails in the CI/CD

Stepping back, observing all of the circumstances and then making notes of variables between “works” and “a pile of chaos” can go a long way to abbreviating the thrashing.

Oh and log the heck out of everything. then once youve solved the problem, clean that up. (devs who create logging chaff while solving a problem and then dont clean up after are a pet-peeve of mine. Makes my eye twitch)