Mobile testing: Startup to Scale Up

I’ve only ever done mobile testing on a small scale myself (startup level). The site was built in React. We had decent logging so we could see the devices our users were accessing our site on. We had no mobile automation set up, my testing was mostly done manually with a few physical devices and browserstack. Oh, and we were deploying continuously to production.

When I was there, we were in talks to diversify and expand the product. This, of course, would eventually mean a larger software development team and hopefully a larger software testing team to meet the diversification demand. The hope was that the product could become a progressive web app. This wasn’t set in stone though but it was in discussions.

So, to my question(s):

  1. When is a good time to start thinking about mobile test automation?
  2. Have you ever had to scale mobile testing efforts? Either from a situation similar to above or for some other reason. If so, what helped you/your team to achieve this?
  3. How did/does the release cycle affect the scaling? E.g. continuous deployment vs deploying once per week or once per fortnight.
  4. When it comes to something like a progressive web app, how different is your approach to that vs a native app?

I have some experience in this realm, and my cohorts in the QA brain trust of NYC, working at vimeo, facebook, stash, and 1stdibs have given me feedback as I have started working with mobile more in my job duties.

  1. The right time to start thinking about mobile test automation is as early in the process as possible. There may be ways to shift some of the testing burdens onto the developers themselves or to use their unit testing libraries as ways to expand early efforts at QA automation.

  2. I’ve had to take mobile to scale from 100 -100,000 users. Realistically most of the solidity of ensuring that the app scales were at the api and server level. Does the API allow for contiguous users requesting read and write access from the database? Do the transactors for graphql queries or cqrs endpoints handle fragments of queries well, and does the schema optimize for speed and caching of common use data? Doing this and load testing helps a lot. Load testing can tell you how the servers you are accessing are responding to many requests at once. Do the requests show that the requests make servers CPU bound or do the request show failures at the middleware level. This can help inform you infrastructure strategy that helps your app scalability strategies.
    From a purely QA perspective, as far as testing efforts go, I would bank on api automation test suites as iPhone test suites are painful to setup and maintain.

  3. If you plan on heavily investing in mobile strategies from a business perspective I would be advocating for additional headcount for an SDET or purely automation engineer. The build lifecycle, the risk assessment, the requirements documentation, and the backward compatibility of API’S at scale is a painful dance you will have to play regardless of your level of involvement. I am the release manager and QA engineer so luckily release management plays a role in my test planning. If you are not involved in release management you should attempt to get your feet wet there, its a valuable skill and will help guide your company gracefully forward into scalability mode. Mobile has a different development cadence and because of that some features that are ready on a web app may not be available on a mobile app. Understanding the feature release schedule is important in assessing risk and understanding how long duel code maintenance may be required or how much scope backward compatibility may add to a feature release schedule.

4.A progressive web app’s i don’t have much experience with.