If you have end-to-end tests integrated into your CI/CD, how do you debug them?
I know Cypress has video playback, but it seems like thereās not much support for video playback in Playwright or Selenium.
Or, is everyone using Sauce Labs or Browserstack which has those sorta features?
Great question, I am really curious about how others debug e2e tests?
Playwright does have support to record videos. I recently set up a GitHub Actions pipeline that ran Playwright end-to-end tests. It recorded videos for all of the test cases, and retain the ones for test failures as artifacts that we could review. This is how I typically debug these tests on CI.
When working on the tests locally, most end-to-end frameworks have decent debugging tools that can help you deal with most issues. With Playwright I heavily use the VS Code extension which works really well. The inspector tool is also great. Most of the other E2E tools Iāve used (TestCafe, Cypress, etc.) have similar debugging tools. Iāve found that I donāt need much else.
This is going to sound harsh, and yes debugging tests on any platform is hard if the developers donāt help you to help them. Iāve had 5 or 6 tester jobs, and much of my day job has been about easily capturing traces, not about writing tests.
3 things you want as artefacts in every test.
Backend logging to show state of the session/activity and loads of timestamps also super helpful. Everyone will now squeal and say our app has no logs. Well in that case you app has no bugs, ergo, your app does have logs, you just donāt know how to find them. Iāve released software before without any logging, it had a very low bug count, and almost no functionality. The biggest feature in the follow up release was a silent feature, you guessed it, logging and a way to pipe the logs right into the automated test system. Learn about Log4J (dont use it it a security minefield) we use a variation on that same pattern and itās awesome. Make sure that any exceptions your test throws are not just āElementNotFoundExceptionā thatās pure laziness. A good stacktrace with ācontextā to go with it, beats a video hands down, hardly ever found videos useful except for slick sales demos.
I run our Playwright tests locally. I run the tests locally when I write them and when I fix them. Playwright also has Trace Viewer which I use locally and in CI to debug issues: Trace viewer | Playwright.
And I think I should add that Trace Viewer is brilliant! It gives me so much information. It is better than a video because I can see logs and inspect the page at each step.
Nice, I didnāt know about inspector tool, thanks for sharing that - does it support breakpoints?
No, it is for use when the code has run.
Is this post just about getting feedback from others on the topic?
Or is this also a discussion about something lacking regarding automated e2e test execution, within the context of CI, when there is no video recording? If the latter, would like to hear about the lacking aspects.
As far as Iām aware, you can always manually trigger rerun of (failed) CI/CD tests and debug from there, hoping that failure reproduces. The pain for this scenario is when you have flaky tests or or flaky bugs that donāt always reproduce across test runs consistently.
And when you manually rerun CI/CD tests, depending on the setup, you should be able to visually see the automation in action unless you use an entirely headless test setup, but I assume such a headless test setup wonāt get you much by way of video recording and screenshots as well?
I think I recall some test frameworks have a screenshot on failure option, that should help compensate for lack of video recordings. I donāt recall the specific framework or tools with the feature though.
Selenium (Grid) also has video recording support, but not natively, you have to install/setup some plugins or use the feature that is part of some test framework. But it is there should you really want it to use it, just have to research around.
And perhaps an expansion of this discussion here, is how do you debug automated e2e mobile app tests, for case of simulator runs or running on devices in a device farm/lab, etc.
Or even automated e2e tests of desktop GUI apps, is that even a thing anymore?
I agree with Comrad. I have to debug our Selenium E2E tests on the CI/CD pipeline and I need:
The debugging support was the biggest reason we picked Cypress ~4 years ago (versus Protractor at the timeā¦given the latter is now deprecated Iām even more glad we went that route!)
In addition to the video playback and screenshots captured automatically on test failure (though I see Cypress 13 made it a little harder to keep video if youāre not using their āDashboardā cloud-based feature), itās also possible to:
cy.pause() to have the test pause anywhere in your test code, and optionally single-step from there.cy.log() to log additional information in Cypressās log panel on the left side of the test runnerconsole.log to print to the browserās Dev Tools console (make sure you have it open first!)cy.pauseāsuper useful for figuring out selectors, etc.I find using Playwright inspector & VSCode breakpoints to work quite well. The inspector apparently stops automatically at each first point where it calls the Playwright api (goto method for us in particular), which can be a little annoying but once you get over that itās fineā¦ And then when you reach a breakpoint in VSCode it stops & you just go to that window in your apps.
In a CI pipeline Iāve used WebDriverIO.
Never had a chance to get video working, although that would be preferred. Always relied on logging and screenshots. That was usually enough for me but not for the other developers. Thatās where CypressIO has really spent their time making things smooth.
Generally speaking, if Iām debugging tests locally, I use Visual Studio Code and then configure my setup so I can set break points in my code. Iām sure other frameworks have a similar process.
A few aspects to consider here that can quickly enhance your debugging process.
Framework language choice - With languages like Ruby and Python, you wonāt have the asynchronous challenges that occur with JavaScript for example. This means that you can run and effectively pause and then continue your tests mid-flow. You can use the interactive console in your IDE to ask questions and test values of variables and constants in the code as well as page object presence and state in the UI for example.
Adopt a reporting framework that gives you some kind of historical test result pattern analysis. Itās all good and well to know that a test has failed but how many times in succession has it failed and has it failed for exactly the same reason every time. These are the āburningā tests that want your attention first.
IMO, video playback doesnāt add that much more significant value to screenshot capture. If you have screenshot, stack trace and analytics for all failing tests as per point 2 above, you should be able to easily and quickly debug your tests.
Sauce Labs and others etc are great for runtime coverage and raising non-functional bugs but not so much for quick debugging. That needs to be done locally. Youāll want to integrate with these other tools only once you have a nice clean set of well performing tests in your own local pipeline.
Welcome to the Club James.
Yes good wrap-up of the high level triage, Iām quite keen to know how @ianjennings started to overcome the problem of not being able to run in a local environment. Itās a common issue even for non-public-cloudy test setups, for me the right thing to do is look at what point in time and other environmental change the fault occurred, by having that good history dashboard.
Playwrightās trace view is provides snapshots of all the actions it went through.
Thus, allowing you to see in what state the DOM was when the tests failed.
Currently, Iām trying to implement an email reporter in Playwright. Lets see how it goes.
Welcome and thank you James!
I appreciate you comment about recordings and screenshots. Its one tool in the shed. But often really good logging will uncover the issue. But I also want to call out your comment on āWhat and Whenā a test fails. Because the temptation is often to dive in and thrash about in order to quickly solve the problem. Stepping back and observing the result over time can provide a lot of information to direct the next steps.
Is the test suddenly failing every time when it worked before? (Could be a product code change thats not being accounted for? run the last known good AUT branch is a task to consider
Is the test occasionally failing over several branches? Could be an infrastructure issue, timing, or something unrelated to the code itself? also known as: internal screaming when everything works locally but fails in the CI/CD
Stepping back, observing all of the circumstances and then making notes of variables between āworksā and āa pile of chaosā can go a long way to abbreviating the thrashing.
Oh and log the heck out of everything. then once youve solved the problem, clean that up. (devs who create logging chaff while solving a problem and then dont clean up after are a pet-peeve of mine. Makes my eye twitch)