I’m a full-stack dev, and I struggled with finding the right tools for UI tests too. I’ll be upfront, I am one of the creators of UI-licious, which is a paid UI test automation tool for web applications. I’ll try to answer your questions as far as I can, and tell you what I think objectively of TestCafe and Cypress. I’ll also share about about UI-licious, for you to consider.
1. Which JS-based setup worked well for you?
I used to use Codecept.js for a very long time in my prior projects. It’s free and open-source. Loved its simple commands, super easy to learn and recall, and I could write tests that were readable and easier to maintain. However, it’s not as elegant as I hoped it be, from time to time, I had to fallback to hard coding CSS selectors and using magic waits to work around async events. As far as I know, there isn’t a way for me to split my tests to run in parallel for faster test suite completion. It’s still the best free open-source testing framework I’d recommend.
2. Is mocha + WDIO is still something reliable and modern enough to pick it for a new long term project?
My efforts to wraggle Codecept.js into something to my liking led me to this actually… Webdriver.js is the underlying implementation of Codecept.js, which I used for handling edge scenarios to test, where Codecept.js falls short. Webdriver.js provides a nice API to use the webdriver protocols to do nitty gritty stuff, but it’s not something I’d recommend for every project to use all the time, because you probably don’t need to go to that level of control.
3. I heard WDIO core codebase is sort of flaky.
I don’t blame them. The webdriver specifications are still not fully implemented by all browser vendors. It’s a full time job trying to keep abreast of the changes in every browser driver, and trying to make sure that you still support testing for the older version of the browsers.
4. What’s your opinion about more modern tools like TestCafe and Cypress?
Cypress: Really powerful stuff. Conceptually I like that they are building testing from ground up instead of using Selenium as a base like most testing library, which is the reason why tests are flaky in the first place. The reports look great. Syntax is easy enough to learn, and is like Chai, which is very familiar. Pricing is expensive after beta, from $99/month, but if you don’t value your time saved, then look somewhere else. And they only support Chrome testing as of now, testing other browsers doesn’t not look like something they plan to support in the near future.
Now that we are here, let me share with you my perspective on testing, and why I created UI-licious.
User is king/queen. Focus on the user’s journey. - I don’t think UI testing is only about filling in forms, and pressing buttons. That’s why I dislike the PageObject syntax, it’s not readable nor dev-friendly to me. I love BDD-styled tests, as they describe the journey of the users, and becomes living documentation for your app. It’s much easier to author tests this way.
Testing should be independent from UI implementation - Hard-coded CSS selectors and magic waits are a no-no. Hard-coding is a code smell, and guess what, your tests are code. Hard coding your tests to your UI code is what’s making your test flaky. And in large web applications, ID-ing all the elements to interact with for testing is hard, and your design is going to change (and no doubt IDs will change). With UI-licious, using keywords like “Email” is encouraged. UI-licious’s powerful test engine does the hard work for you of identifying the element to interact through static code analysis and contextual analysis. As long as the user journey remains the same, tests can be used for multiple designs and UI implementations.
Here’s what logging in to Github with UI-licious looks like:
I.fill("Username or email address", "brucewayne");
Try it for yourself. I’d be happy to hear any feedback (criticism included).
Back to your remaining questions:
Any pitfalls to be aware of?
Don’t get the same guy to write tests for the feature they’ve build, get a different person to do the testing. Everyone has tunnel vision and believes they write perfect code, but I’ve consulted several development teams, and even senior developers make these mistakes (admittedly even I do). And this is even more so when the developers are not used to writing tests because they are taught to “Move fast, break things”.
Would you suggest any other programming language but JS that is front-end engineers friendly?
I think you made the right choice to choose JS for testing front-end, because this is the language of the modern web, and the JS ecosystem is considerably mature and robust.
Hope this helps, thanks for sharing your questions!