If you have end-to-end tests integrated into your CI/CD, how do you debug them?
I know Cypress has video playback, but it seems like there’s not much support for video playback in Playwright or Selenium.
Or, is everyone using Sauce Labs or Browserstack which has those sorta features?
Great question, I am really curious about how others debug e2e tests?
Playwright does have support to record videos. I recently set up a GitHub Actions pipeline that ran Playwright end-to-end tests. It recorded videos for all of the test cases, and retain the ones for test failures as artifacts that we could review. This is how I typically debug these tests on CI.
When working on the tests locally, most end-to-end frameworks have decent debugging tools that can help you deal with most issues. With Playwright I heavily use the VS Code extension which works really well. The inspector tool is also great. Most of the other E2E tools I’ve used (TestCafe, Cypress, etc.) have similar debugging tools. I’ve found that I don’t need much else.
This is going to sound harsh, and yes debugging tests on any platform is hard if the developers don’t help you to help them. I’ve had 5 or 6 tester jobs, and much of my day job has been about easily capturing traces, not about writing tests.
3 things you want as artefacts in every test.
- Does your app have any logging? I’m assuming it’s a browser app here, but it still has tons of logging, talk you your devs and get the JS hooks you need written so you have logging, you win, they win.
- Screenshot I assume you have a screenshot to go with the stacktrace?
- DOM snapshot. A snapshot of the DOM immediately after the test starts dying.
Backend logging to show state of the session/activity and loads of timestamps also super helpful. Everyone will now squeal and say our app has no logs. Well in that case you app has no bugs, ergo, your app does have logs, you just don’t know how to find them. I’ve released software before without any logging, it had a very low bug count, and almost no functionality. The biggest feature in the follow up release was a silent feature, you guessed it, logging and a way to pipe the logs right into the automated test system. Learn about Log4J (dont use it it a security minefield) we use a variation on that same pattern and it’s awesome. Make sure that any exceptions your test throws are not just ‘ElementNotFoundException’ that’s pure laziness. A good stacktrace with “context” to go with it, beats a video hands down, hardly ever found videos useful except for slick sales demos.
I run our Playwright tests locally. I run the tests locally when I write them and when I fix them. Playwright also has Trace Viewer which I use locally and in CI to debug issues: Trace viewer | Playwright.
And I think I should add that Trace Viewer is brilliant! It gives me so much information. It is better than a video because I can see logs and inspect the page at each step.
Nice, I didn’t know about inspector tool, thanks for sharing that - does it support breakpoints?
No, it is for use when the code has run.
Is this post just about getting feedback from others on the topic?
Or is this also a discussion about something lacking regarding automated e2e test execution, within the context of CI, when there is no video recording? If the latter, would like to hear about the lacking aspects.
As far as I’m aware, you can always manually trigger rerun of (failed) CI/CD tests and debug from there, hoping that failure reproduces. The pain for this scenario is when you have flaky tests or or flaky bugs that don’t always reproduce across test runs consistently.
And when you manually rerun CI/CD tests, depending on the setup, you should be able to visually see the automation in action unless you use an entirely headless test setup, but I assume such a headless test setup won’t get you much by way of video recording and screenshots as well?
I think I recall some test frameworks have a screenshot on failure option, that should help compensate for lack of video recordings. I don’t recall the specific framework or tools with the feature though.
Selenium (Grid) also has video recording support, but not natively, you have to install/setup some plugins or use the feature that is part of some test framework. But it is there should you really want it to use it, just have to research around.
And perhaps an expansion of this discussion here, is how do you debug automated e2e mobile app tests, for case of simulator runs or running on devices in a device farm/lab, etc.
Or even automated e2e tests of desktop GUI apps, is that even a thing anymore?
I agree with Comrad. I have to debug our Selenium E2E tests on the CI/CD pipeline and I need:
- Logging. Know what your tests are doing
- Screenshots. Get the last “status” of the tested page when it failed.
The debugging support was the biggest reason we picked Cypress ~4 years ago (versus Protractor at the time…given the latter is now deprecated I’m even more glad we went that route!)
In addition to the video playback and screenshots captured automatically on test failure (though I see Cypress 13 made it a little harder to keep video if you’re not using their “Dashboard” cloud-based feature), it’s also possible to:
cy.pause() to have the test pause anywhere in your test code, and optionally single-step from there.
cy.log() to log additional information in Cypress’s log panel on the left side of the test runner
console.log to print to the browser’s Dev Tools console (make sure you have it open first!)
- Use the Browser’s dev tools to inspect the state of the page, either at the end of the tests or while paused with
cy.pause–super useful for figuring out selectors, etc.
- After the tests finish running, view “snapshots” of the page state by clicking on the actions in the Cypress command log.
I find using Playwright inspector & VSCode breakpoints to work quite well. The inspector apparently stops automatically at each first point where it calls the Playwright api (goto method for us in particular), which can be a little annoying but once you get over that it’s fine… And then when you reach a breakpoint in VSCode it stops & you just go to that window in your apps.
In a CI pipeline I’ve used WebDriverIO.
Never had a chance to get video working, although that would be preferred. Always relied on logging and screenshots. That was usually enough for me but not for the other developers. That’s where CypressIO has really spent their time making things smooth.
Generally speaking, if I’m debugging tests locally, I use Visual Studio Code and then configure my setup so I can set break points in my code. I’m sure other frameworks have a similar process.