[Advice Needed] Front-End Engineers Friendly UI Automation

(Alexander) #1

Hey! I’m about to start a new web-based project and I’m almost certain about picking JS for UI tests. Previously I’ve been working with Java (just saying for the record). The main ideas behind picking JS is to make developers involved:

  • ability to read tests codebase;
  • ability to run tests locally before pushing code to branch (for some really weird reason this point didn’t work well with Java on the previous project, and most of them admitted they’d use tests more frequently if that would be a code they understand)
  • able to update tests along with pushing new code to feature-branch
  • able to code new tests before merging feature-branch to master

I have a huge headache rolling through all these JS frameworks as I feel none of them perfectly fits any of my actual and potential requirements out of the box.
Before I’ll list the questions, here are some of my crucial requirements which should be supported out of the box or could be implemented with minimum efforts and high stability level:

  1. Parallel tests execution in concurrent threads. Say, there’s a hundred-tests suite which is needed to be run in Chrome only. The goal is to execute tests in 5 concurrent threads, so 5 tests would be executed simultaneously.
  2. HTML reports with screenshots.
  3. Ability to easily extend base set of framework’s features.
  4. Ability to run code synchronously (asynchrony makes tests flaky and somewhat hard to read imo)
  5. Supports PageObject.

So, the questions I’m curious about are:

  1. Which JS-based setup worked well for you? A short description of what you’ve been doing with your framework is appreciated.
  2. Is mocha + WDIO is still something reliable and modern enough to pick it for a new long term project?
  3. I heard WDIO core codebase is sort of flaky, so its maintainers more likely wouldn’t accept a PR because of uncertainty about possible problems caused by the changes. From your expirience did you have any problems with WDIO stability?
  4. What’s your opinion about more modern tools like TestCafe and Cypress?

Also, In case if you had experience with involving front-end engineers into UI testing process:

  1. Any pitfalls to be aware of?
  2. Would you suggest any other programming language but JS that is front-end engineers friendly?

(Igor) #2

Checkout this doc Perfecto_Webframework_ebook_F
Could be quite useful in your case

(Sriram) #3

Based on your needs, I presume BDD - Cucumber - Protractor would be a good option to look into.

We currently use this in few front end development projects and has given some good takeaways such as,

  1. Testing of e2e scenarios before a code promote ( practically makes it easy to adopt CI approach)
  2. Supports page objects
  3. Reduces repetitive work as the people involved mature in using a standard pattern
  4. Reduces rework caused due to application changes as the feature files, page objects and step definitions are separated without any one to one mapping

(Alexander) #4

BDD is out - i’m 99% sure that’s not something front-end engineers will be happy with, tbh, so am I. As for Protractor - Google designed it specifically to support Angular. We aren’t gonna use Angular (it’s already a made decision), so I believe there should be a better option. Not that I imply that it’s not gonna work with a non-Angular app - I just think there should be a better option for such case.

@igla Thanks for the reference, I’ll check it out.

(Shi Ling) #5

Hey Alexander,

I’m a full-stack dev, and I struggled with finding the right tools for UI tests too. I’ll be upfront, I am one of the creators of UI-licious, which is a paid UI test automation tool for web applications. I’ll try to answer your questions as far as I can, and tell you what I think objectively of TestCafe and Cypress. I’ll also share about about UI-licious, for you to consider.

1. Which JS-based setup worked well for you?
I used to use Codecept.js for a very long time in my prior projects. It’s free and open-source. Loved its simple commands, super easy to learn and recall, and I could write tests that were readable and easier to maintain. However, it’s not as elegant as I hoped it be, from time to time, I had to fallback to hard coding CSS selectors and using magic waits to work around async events. As far as I know, there isn’t a way for me to split my tests to run in parallel for faster test suite completion. It’s still the best free open-source testing framework I’d recommend.

2. Is mocha + WDIO is still something reliable and modern enough to pick it for a new long term project?
My efforts to wraggle Codecept.js into something to my liking led me to this actually… Webdriver.js is the underlying implementation of Codecept.js, which I used for handling edge scenarios to test, where Codecept.js falls short. Webdriver.js provides a nice API to use the webdriver protocols to do nitty gritty stuff, but it’s not something I’d recommend for every project to use all the time, because you probably don’t need to go to that level of control.

3. I heard WDIO core codebase is sort of flaky.
I don’t blame them. The webdriver specifications are still not fully implemented by all browser vendors. It’s a full time job trying to keep abreast of the changes in every browser driver, and trying to make sure that you still support testing for the older version of the browsers.

4. What’s your opinion about more modern tools like TestCafe and Cypress?
TestCafe: Haven’t tried it. But what I like is - parallel test execution, uses modern Javascript (ES6). Not sure about failing on JS errors, I don’t really see a need for this. Not a fan of PageObjects style syntax.
Cypress: Really powerful stuff. Conceptually I like that they are building testing from ground up instead of using Selenium as a base like most testing library, which is the reason why tests are flaky in the first place. The reports look great. Syntax is easy enough to learn, and is like Chai, which is very familiar. Pricing is expensive after beta, from $99/month, but if you don’t value your time saved, then look somewhere else. And they only support Chrome testing as of now, testing other browsers doesn’t not look like something they plan to support in the near future.

Now that we are here, let me share with you my perspective on testing, and why I created UI-licious.

  1. User is king/queen. Focus on the user’s journey. - I don’t think UI testing is only about filling in forms, and pressing buttons. That’s why I dislike the PageObject syntax, it’s not readable nor dev-friendly to me. I love BDD-styled tests, as they describe the journey of the users, and becomes living documentation for your app. It’s much easier to author tests this way.

  2. Testing should be independent from UI implementation - Hard-coded CSS selectors and magic waits are a no-no. Hard-coding is a code smell, and guess what, your tests are code. Hard coding your tests to your UI code is what’s making your test flaky. And in large web applications, ID-ing all the elements to interact with for testing is hard, and your design is going to change (and no doubt IDs will change). With UI-licious, using keywords like “Email” is encouraged. UI-licious’s powerful test engine does the hard work for you of identifying the element to interact through static code analysis and contextual analysis. As long as the user journey remains the same, tests can be used for multiple designs and UI implementations.

Here’s what logging in to Github with UI-licious looks like:

I.click("Sign in");
I.fill("Username or email address", "brucewayne");
I.fill("Password", "iloveselina");
I.click("Sign in");

Try it for yourself. I’d be happy to hear any feedback (criticism included).

Back to your remaining questions:

  1. Any pitfalls to be aware of?
    Don’t get the same guy to write tests for the feature they’ve build, get a different person to do the testing. Everyone has tunnel vision and believes they write perfect code, but I’ve consulted several development teams, and even senior developers make these mistakes (admittedly even I do). And this is even more so when the developers are not used to writing tests because they are taught to “Move fast, break things”.

  2. Would you suggest any other programming language but JS that is front-end engineers friendly?
    I think you made the right choice to choose JS for testing front-end, because this is the language of the modern web, and the JS ecosystem is considerably mature and robust.

Hope this helps, thanks for sharing your questions!

(Alexander) #6

So I’ve spend about a week diving into testing tools in JS. I’ll share a couple of useful links below along with the stack I’ve picked up.


  • WDIO + Mocha + Chai, with WDIO runner. It has native concurrent tests execution, while in Mocha you would need to use a 3rd party library since Mocha doesn’t share the ideology of running tests in parallel.
  • Allure for reports. Probably, the best html reports out there, though, wdio plugin lacks some features. But it’s being developed.
  • Selenoid for enhanced parallel tests execution. Basically, it’s a GRID router written in Go (consumes much less resources compared to the one written in Java) that runs browsers in docker containers. This tool is programming language agnostic, however it’s a great tool and I guess it worth mentioning that wdio can be paired with it easily (e.g. codeceptjs can’t)

I’d recommend checking them in exactly this order:

  1. This one is not quite about UI testing, however its first part worth reading in order to get the initial idea of how to approach the process of choosing tools in JS world.

  2. 10 Test Automation Frameworks for Cross-Browser Testing. Kudos to @igla for this one! A good starting point to understand ‘which is what’ in JS UI testing world. However, this overview doesn’t have TestCafe and Cypress covered.

  3. Quite deep analysis of all the most popular and actively developed JS UI testing frameworks, with graphics, numbers and code snippets. To me, this one is the most valuable in terms of information.

(Shi Ling) #7

I vote for Selenoid too - makes setting up Selenium environments fuss-free, and it’s very stable in large grid deployments. However as far as I know now, that Selenoid does not offer IE and Safari environments to test on, something to note if you need to support testing for these browsers.

Thanks for sharing the articles.

The third one is a really good, it really gets into the nitty gritty of using each library and talks about the developer friendly-ness. I’m most intrigued by the comparison of API styles, it’s one of the things people rarely consider upfront when choosing a tool, but it ends up mattering a lot depending on your programming philosophy - in this case, whether or not you think using async-await with promises is a good idea here (generally I don’t think it is, and I’d prefer coroutine-sync libraries for the readability and ease of maintain the tests).

(Alexander) #8

Yeah, me too - that’s one of the points I didn’t like about TestCafe. CodecptJS, WDIO and UI-licious definitely win here, at least for me.

As for Selenoid, both IE and Safari are coming up soon. Guys who develop it, shared 2 screenshots last week, showing IE and Safari running in the docker container under Ubuntu. IE session starts in 12 seconds, container size is a couple Gbs. Safari session starts in 20 seconds. So, basically it’s already possible, and once they’ll polish things, containers will be published. This is currently WIP and they got some things to solve with licensing (according to “beloved” Apple policies, you’re allowed to run MacOS only on Apple’s hardware, but Selenoid is somehow able to do it on any server). However, I’m pretty much sure it will be possible quite soon.
They got a Telegram support chat, which is very active - that’s a right place to ask for details.