I’m wondering if I could have a little help from all of you experienced people.
Our company has shifted their priorities and wants to achieve parity in all platforms, this includes iOS/Android native app, iOS/Android browsers and web app browsers.
Currently they have an automation suite created and owned by developers with Cypress, however this doesn’t scale well when we want to also automate regression and smoke tests to the rest of the platforms.
My question to you all here is:
is there an automation language/app which could work for an all in one solution?
would there be any benefit in keeping one automation suite per platform and in that case do native automation?
what would you use if it where you and why? what’s the benefit/cost?
The end goal is to have the automation suite(s) owned by our QA team (they have development skills) and extend as much as possible for parallel runs. There might be the need to also have snapshot testing further down the line, so an automation who would allow us to grow on this area would be preferable.
What is your underlying testing stack built on? As in your test “runner”. It’s a common mistake as companies grow to use an off-the-shelf tool to batch test suites or just to build one that later on fails to scale. Every place I have worked at to date have built their own runner inside jenkins/teamcity triggers. Most tools don’t know how to manage resources nor your test nodes, and the tools that do manage nodes on the other hand won’t manage your domain problem. Teams that go with off-the-shelf can find they want to change the tool out after 10 years because their product rotated into a new environment that the tool does not deliver well on.
Like @jon_thompson proposes, raw appium might be a better fit with java or python engine. Java and python have very strong support for solving domain problems, distributing your tests, managing network resources and other business constraints that cypress or other off-the-shelf tools rarely can deal with.
We use selenium in Java for our web project, and we use a pytest which we heavily forked for Appium (ios+android) tests. We have a “runner” which manages our grid of resources which gets triggered from jenkins. But no it’s not cross-platform by any stretch, because even between ios and android the code sharing is not that high, it’s around 3/4 generic, but I still spend most of my time in the phone-specific bits. Our grid also handles native desktop, and once again that 3/4 of generic code is not the problem, it’s the time spent dealing with unusual macOS behavior or a timing bug low down in a test step. If you go with a good OO programming language you rarely hit walls. It depends on how much you want to invest.
My biggest pain about off-the-shelf is that when you find something like a customer has an issue on “phone model X” , and as you investigate you discover that some default setting on the phone that other phones don’t have was different. Or that some background app , like an email client that is related to your test on the phone is popping up more than once during a test. And [Cypress] (insert any popular vendor’s toolstack here) ends up not dealing with that. If you roll your own “runner” engine, then these become very easy to add pre-test and post-test sanity checks for these problems.
I fully second Conrad.
Engineering problems can not be solved (easily) with off-the-shelf products.
Automation is mostly a development task aka engineering. IN/FOR TESTING.
We also use a hybrid java + appium framework for doing this type of mobile testing. We run these tests on cloud devices in a mobile device farm. Test output includes appium logs, device logs, screenshots, and videos taken during execution.
Cypress also works well with cloud providers. I am curious why you think that Cypress does not scale well… this is a new tool that is gaining more acceptance in many companies. If you want the QA team to switch to another test tool, you need to have a POC and a lot of good arguments supporting your motion to switch.
I assume, from the problem description, that you’re interested only in F/E testing - I’ve yet to see any reference to BrowserStack which provisions SeleniumGrid across 3000 browsers (incl. iOS) - (their claim, not mine ) - it costs, but as we all know “Quality is free, but only to those who are willing to pay heavily for it” (DeMarco & Lister). As far as orchestration goes, IMO, using Cucumber with the aforementioned BrowserStack provides a near panacea
Thank you for your feedback.
The reason why at the moment Cypress isn’t working for us because we also need to test mobiles with the same features files/scenarios that we have for web. That’s why we where looking to something else that doesn’t require duplicated code.
Thank you for your feedback.
We’re not expecting an off the shelf solution. We currently have a few tests in cypress and a few integration ones with pytest. What we are trying to achieve if possible is to come up with a solution which would incorporate differente browsers and different platforms (web, mobile) but also if we could add api tests in the same place would be amazing.
We’re all SDETs so not afraid of code, just trying to understand if anyone already had a challenge like this and what people from the community would recommend in this case.
We currently use BrowserStack from a manual perspective and we’re aware of the automated area, however we would still need to create the code for it to run in their grid. We are also aware of LambdaTest (BrowserStack competitor) which at the moment has a few other things we could be interested in as well. Unfortunately, none would create the code for us hence why we’re looking for ideas on best practices and languages that would work for all of these.
Probably worth a skunkworks Skunkworks project - Wikipedia project to see what test automation is possible at least for smoke. The issue is that it might be non-performant to expect one stack to test all platforms depending on how your test lab is set up and how much resources and how much security and environment realism you want to have. So I would scope things as “smoke only” to avoid disssapointment and make it cheap to abort entirely.
App uniformity is great thing to have but it hides details like the way different OS actually have different conventions “natively” in terms of how furniture works, so a test that uses things like OK or back and hamburger menus as well as the common mobile things like swipes will “fork” and add complexity once you go deeper. It’s a great way to find inconsistencies in platform look, but manual testing is far better at that IMHO. Try for a skunkworks project punt I say.
) - it costs, but as we all know “Quality is free, but only to those who are willing to pay heavily for it” (DeMarco & Lister). As far as orchestration goes, IMO, using Cucumber with the aforementioned BrowserStack provides a near panacea
hence why we’re looking for ideas on best practices and languages that would work for all of these.