I’m looking for an up to date, impartial, in-depth comparison between these tools for executing Selenium tests on real mobile devices based on recent experience.
Can anyone help or point me in the direction of one?
Many Thanks.
What are your requirements and is the app native, Web based or hybrid?
I have compared browsers and saucelabs for web based app to be tested on different os, devices or browsers
after using we found:
- both are easy to use and integrate in framework
- both have competitive pricing
- both were quite slow to execute tests compared to all maintained dockerized / local execution
- both provide similar device support which is more than enough for most use cases
- I personally liked saucelabs documentation better
- since our app isn’t native but just a week based app that can be opened in browser. we find it less beneficial to run it against different combinations of os, browsers, devices (considering today’s browsers have really good support for latest js, react and other libraries)
ps: currently we used saucelabs with test Cafe (not selenium). testcafe itself provides us good cross browser testing capabilities and viewport testing (there are also other great tools like cypress , playwright, selenoid and zalenium.
sorry, wasn’t much of help but I find the to tools very similar for what they offer.
3 Likes
Thanks… I should have said this is for website testing only at this stage, not an a app. 
1 Like
aha, if it’s a web app, then given a chance I’d now think twice whether I really need to test on actual devices in cloud. It does depends on framework and libraries your app uses but because of standardization there are now less and less browser incompatibility issues compared to few years ago.
In 1 year (several builds per day) of using cloud testing I have found 1 or two incompatibility issue with edge which I could have also found equally efficiently and probably faster using local cross browser testing using open source tools. This could either mean that our cross browser tests are rubbish or that we have not faced many incompatibilities even after such a long time (I’d let readers guess that 
)
For screen sizes and resolutions, there are several tools that allow you to change viewport and test app. This testing is definitely important of your app is primarily used on mobile. There are free tools for visual regression and paid ones with some intelligent features like applitools.
For older browser versions, I have not worked on any app that ‘needs’ to support older, now insecure and off-support browsers (e.g. ie9)
1 Like
Thanks for your reply Yogesh and I agree with much of what you are saying… in theory.
I think it comes down to confidence in the tooling you suggest.
However a combination of a) having to maintain multiple website codebases… some of them brand new and other very legacy b) high levels of evidence and governance required and c) real life experience of bugs found with our newer tech stack on mobile devices using iOS / Safari browsers (particularly) that have not manifested themselves using say Cypress on various viewports for example, mean that testing on real devices in the cloud is something that I see us doing …certainly in the short to medium term.
Support for IE11 is also still required but no further back than that - thankfully! This browser also provides us with issues so using something like Browserstack OR Saucelabs OR Crossbrowsertesting.com …simply using our existing Selenium test suite, I felt would give us the capability to reuse what we already have on a realistic selection of devices until the confidence I mention was achieved …not only in our codebase implementation but also the test tooling.
Crossbrowsertesting.com is the on I have least knowledge of: Price wise it is considerably cheaper that the other two… but do you get what you pay for?
The previous comparisons I have read online be appear quite old.
Cheers.
3 Likes
Have you looked at AWS Device Farm? You can do website testing there with Selenium. Usually much cheaper than the other tools though the logging etc. isn’t as good.
1 Like
You need to decide whether an emulator is good enough or real devices. When I was looking, only browserstack had real devices. This may have changed. I would use google analytics to see what you need to test. If you don’t need devices you could probably just use cypress.
I kind of agree with the other commenter upon the amount of bugs you will find being low by testing devices. I agree you won’t find many and but it depends what you’re customers are prepared to put up with, before using a competitor?
1 Like