If you have end-to-end tests integrated into your CI/CD, how do you debug them?
I know Cypress has video playback, but it seems like thereās not much support for video playback in Playwright or Selenium.
Or, is everyone using Sauce Labs or Browserstack which has those sorta features?
Great question, I am really curious about how others debug e2e tests?
Playwright does have support to record videos. I recently set up a GitHub Actions pipeline that ran Playwright end-to-end tests. It recorded videos for all of the test cases, and retain the ones for test failures as artifacts that we could review. This is how I typically debug these tests on CI.
When working on the tests locally, most end-to-end frameworks have decent debugging tools that can help you deal with most issues. With Playwright I heavily use the VS Code extension which works really well. The inspector tool is also great. Most of the other E2E tools Iāve used (TestCafe, Cypress, etc.) have similar debugging tools. Iāve found that I donāt need much else.
This is going to sound harsh, and yes debugging tests on any platform is hard if the developers donāt help you to help them. Iāve had 5 or 6 tester jobs, and much of my day job has been about easily capturing traces, not about writing tests.
3 things you want as artefacts in every test.
Backend logging to show state of the session/activity and loads of timestamps also super helpful. Everyone will now squeal and say our app has no logs. Well in that case you app has no bugs, ergo, your app does have logs, you just donāt know how to find them. Iāve released software before without any logging, it had a very low bug count, and almost no functionality. The biggest feature in the follow up release was a silent feature, you guessed it, logging and a way to pipe the logs right into the automated test system. Learn about Log4J (dont use it it a security minefield) we use a variation on that same pattern and itās awesome. Make sure that any exceptions your test throws are not just āElementNotFoundExceptionā thatās pure laziness. A good stacktrace with ācontextā to go with it, beats a video hands down, hardly ever found videos useful except for slick sales demos.
I run our Playwright tests locally. I run the tests locally when I write them and when I fix them. Playwright also has Trace Viewer which I use locally and in CI to debug issues: Trace viewer | Playwright.
And I think I should add that Trace Viewer is brilliant! It gives me so much information. It is better than a video because I can see logs and inspect the page at each step.
Nice, I didnāt know about inspector tool, thanks for sharing that - does it support breakpoints?
No, it is for use when the code has run.
Is this post just about getting feedback from others on the topic?
Or is this also a discussion about something lacking regarding automated e2e test execution, within the context of CI, when there is no video recording? If the latter, would like to hear about the lacking aspects.
As far as Iām aware, you can always manually trigger rerun of (failed) CI/CD tests and debug from there, hoping that failure reproduces. The pain for this scenario is when you have flaky tests or or flaky bugs that donāt always reproduce across test runs consistently.
And when you manually rerun CI/CD tests, depending on the setup, you should be able to visually see the automation in action unless you use an entirely headless test setup, but I assume such a headless test setup wonāt get you much by way of video recording and screenshots as well?
I think I recall some test frameworks have a screenshot on failure option, that should help compensate for lack of video recordings. I donāt recall the specific framework or tools with the feature though.
Selenium (Grid) also has video recording support, but not natively, you have to install/setup some plugins or use the feature that is part of some test framework. But it is there should you really want it to use it, just have to research around.
And perhaps an expansion of this discussion here, is how do you debug automated e2e mobile app tests, for case of simulator runs or running on devices in a device farm/lab, etc.
Or even automated e2e tests of desktop GUI apps, is that even a thing anymore?
I agree with Comrad. I have to debug our Selenium E2E tests on the CI/CD pipeline and I need:
The debugging support was the biggest reason we picked Cypress ~4 years ago (versus Protractor at the timeā¦given the latter is now deprecated Iām even more glad we went that route!)
In addition to the video playback and screenshots captured automatically on test failure (though I see Cypress 13 made it a little harder to keep video if youāre not using their āDashboardā cloud-based feature), itās also possible to:
cy.pause() to have the test pause anywhere in your test code, and optionally single-step from there.cy.log() to log additional information in Cypressās log panel on the left side of the test runnerconsole.log to print to the browserās Dev Tools console (make sure you have it open first!)cy.pauseāsuper useful for figuring out selectors, etc.I find using Playwright inspector & VSCode breakpoints to work quite well. The inspector apparently stops automatically at each first point where it calls the Playwright api (goto method for us in particular), which can be a little annoying but once you get over that itās fineā¦ And then when you reach a breakpoint in VSCode it stops & you just go to that window in your apps.
In a CI pipeline Iāve used WebDriverIO.
Never had a chance to get video working, although that would be preferred. Always relied on logging and screenshots. That was usually enough for me but not for the other developers. Thatās where CypressIO has really spent their time making things smooth.
Generally speaking, if Iām debugging tests locally, I use Visual Studio Code and then configure my setup so I can set break points in my code. Iām sure other frameworks have a similar process.