So, in my organization, we are evaluating Playwright as a replacement for our framework created with WebdriverIO.
While I know that there is no silver bullet, no one software/tool can fit all needs, that’s why I want to know from folks who have been using Playwright for quite some time now, what are the problems/limitations you have faced while using Playwright that you only came to know after you started using it?
EDIT
What works well for us with WebdriverIO?
Regular updates/Community Support
Documentation
Features - webdriver/devtools, component testing
What doesn’t work well with WebdriverIO?
Buggy
Auto-waiting - This is good, but does not work every time.
Reporting - Allure Reporter/@wdio/allure-reporter - It has limitations of its own and debugging is really difficult with it.
Flakiness of the tests - I know this is also dependent on test design. But I believe having webdriver as a middleware between the browser may cause problems as compared to DevTools that works directly with Chrome.
I’ve been using it at work for almost 2 years with no major issues.
Depending on the problems you are having with webdriver.io you will more than likely have those problems with Playwright. Maybe it’s more useful to talk about the problems you are facing with webdriver.io and look for solutions there first?
Auto waiting for Playwright can be helpful but it doesn’t solve all your problems depending on how the UI renders. With the ability to monitor/route/take action on underlying network requests you can use this in Playwright to solve any edge case problems, from my experience.
If you use TS/JS and the Test runner bundled with Playwright, there is a report that is output but you will find it limiting if you are used to robust reports. The good thing is you can write your own reporter or integrate Allure as well if the default isn’t good enough for your teams needs. If the main use of the report is for debugging, I can say it’s really really nice. Here is an example - Playwright Test Report I don’t have any failures (if I did there would be a screenshot of the failure on the test fail and a video) but you can click into one of the UI tests, and click on the trace at the bottom, and open up the trace viewer/ui mode that gives you access to a lot of information of what happened within the test (very similar to cypress).
For flakiness, I haven’t run into any issues where I would call the browser interactions flaky, I’ve written bad tests that caused flaky interactions and I’ve run my tests against flaky infrastructure that would return 502/503 because of load balance/container issues.
One huge plus for Playwright is you can write UI/API tests within the same framework
If you have specific questions feel free to ask or you can check out https://playwrightsolutions.com/ where I’ve been posting articles once a week (you can also subscribe to get them to your email).
It’s a bit flaky running browsers in headed mode in a container over VNC. To be honest, the fact I can do that and it mostly just works is still pretty awesome.
Thanks for sharing your website! Will have a great time reading it
I’ve been working with playwright for 2 years and I can’t complain at all. I moved from Cypress when it was not easy to work with multiple tabs/origins etc (now they changed it afaik) but I don’t regret and actually prefer it.
I’ve been facing one issue tho, it might not be a problem for the OP but it would help a lot my project.
It seems pw doesn’t support global hooks (running something before/after the whole spec file, for all of them - without the need to write it in every spec). GlobalSetup and Teardown are not what I am looking for…not sure if I can change it tho.
I posted a question in the pw discord to see wether someone has the answer for this issue: Discord
Maybe you faced this problem already? If yes I’d love to hear how you fixed it!
@lufis I haven’t had to solve that problem yet in the last 2 years, but seems like a valid one. I added some code that shows how to solve the problem in the discord thread. I’ll also make a post about it in the upcoming weeks on playwright solutions.
Over the years have looked at numerous solutions.
Selenium - Days may be numbered for new integrations with the latest offerings.
Robot - did not get on with it
Cypress - I loved it, but it was missing Webpack for iOS
Reluctantly went to Playwright and was sold from Day-1. The more we use it, the more we find. Better still, the support and integrations for it are growing rapidly.
I liked the UI test approach of Cypress - well that is now in Playwright as Playwright UI. I like the variety of project setups and can integrate this with real device testing - this can be flakey, but is down to the device as opposed to Playwright.
Reporting - so many tools out there. We are about to integrate with TestMo, and it looks great!
Issues - only issue really is my knowledge and the team’s knowledge of JS. Playwright are updating fairly regularly with new features. We have adopted a POM model for our UI and not seeing a lot of issues.
