Performance testing for mobile application

Hello ,

Can anyone help me with some tools/solutions available for mobile performance testing

Requirement:
Need to extract the performance parameters from the android bug report like (CPU Usage , Network performance ,App response time , Battery usage , Crash rate, User engagement)

Ask:
A solution which can help me compare two bug reports extracted at different times and give me a new report of the above parameters after comparison

7 Likes

We have used a tool called Apptim in the past, itโ€™s really great tool in terms of providing client side performance metrics visually inorder to make better decisions. It also has a feature that compares multiple test sessions and gives a benchmark sort of insight too, I think its what you were looking for.

You can find a bit more information here from a webinar I did on the topic: Teaser - STET Talk by Nithin SS on Mobile App Performance Testing - YouTube

3 Likes

If the mentioned video doesnโ€™t already cover this,

for the crash rate, user engagement, network side app/API response times, solutions or tools like Firebase, Crashlytics, sentry.io might offer you visibility/insight to that kind of data if you use them or instrument your app with them.

CPU, network performance, and battery usage, you might be able to collect from the application under test (under say a test mode feature) or a test helper application that you build to run on the Android device alongside the application under test. Because there may be Android APIs you can call on to get that information that you can then collect via test automation or having the app send that info over the internet via custom APIs, etc. to your server. But as such, youโ€™d have to build/instrument such functionality into your app (for test mode use) or in a helper app. I donโ€™t think such features are provide in a ready to run app or SDK you can utilize. If such exists, that would really be nice.

What you ask for is a whole lot easier on the desktop applications case. The scenario is also restrictive for mobile web applications (not native apps).

4 Likes

I wanted to share my exciting experience with performance testing for mobile applications. As a part of an Automation Testing Company, I have always strived to deliver the best user experience, and performance testing has been instrumental in achieving that goal.

When I started incorporating performance testing into my development process, I discovered its incredible benefits. It allowed me to identify bottlenecks, optimize resource utilization, and enhance the overall responsiveness of my app. By simulating various usage scenarios and stress-testing the app, I gained valuable insights into its limitations and areas for improvement.

Not only did performance testing help me optimize speed and responsiveness, but it also contributed to enhancing battery life and reducing memory consumption. By carefully monitoring and analyzing resource usage, I was able to identify and address any memory leaks or inefficient code that could negatively impact the appโ€™s performance.

One key aspect I enjoyed about performance testing was the ability to simulate real-world conditions. By replicating different network speeds, varying device capabilities, and even low battery scenarios, I could ensure my app performed seamlessly across a wide range of devices and situations.

I also discovered the importance of performance testing during app updates. By retesting the app after implementing changes or adding new features, I could proactively identify and address any potential performance regressions, ensuring a smooth user experience across different versions.

Moreover, the insights gained from performance testing helped me optimize the appโ€™s backend infrastructure, leading to improved server response times and better overall scalability.

1 Like

What do you mean by

From bug report?

So do you want to continuously monitor how your app behaves, or do you want to only gather this data from one test run?

1 Like

Hello ,

I want a solution which can monitor and provide the performance analysis of my application on each release . So that if there are any performance glitch in the system i can let the team know about it .

Since i work on a mobile application(Android) , the approach is to extract the data from the bug report which we get after each regression cycle .

hope this answers your question !

@anindita Iโ€™ll split my response into 3 parts :spiral_notepad:

:spiral_notepad: 1/3

The most reliable solution would probably be to use Firebase Performance Monitoring SDK.

Firebase Performance provides real-time insights into app performance. The SDK offers detailed performance metrics, such as network requests and app startup times, which can be used to optimize app performance and reduce load times. Furthermore, the SDK integrates seamlessly with other Firebase services, such as Crashlytics, to provide a comprehensive suite of tools for app development and monitoring.

What you can measure:

  • Network Requests: It automatically measures app startup time, HTTP/S network requests, and provides insights about the network performance of your app.
  • App Start Time: Firebase Performance Monitoring can track the time it takes for your app to start and become responsive to user input.
  • Screen Rendering: It can monitor the rendering performance of your app, helping you to identify and fix any issues that could be causing slow rendering times.
  • Custom Metrics: You can define your own custom metrics to measure specific aspects of your appโ€™s performance that are unique to your app.
  • Trace Metrics: You can create โ€˜tracesโ€™, which are a series of code that you want to monitor. For example, you could create a trace to monitor the time it takes for a certain function to execute.
  • Automatic and Manual Instrumentation: Firebase Performance Monitoring SDK provides automatic instrumentation for some common tasks, but also allows you to manually instrument specific parts of your app.
  • Real-time Results: The SDK provides real-time results, allowing you to quickly identify and address performance issues.
  • Cross-Platform Compatibility: Firebase Performance Monitoring SDK works with both Android and iOS, allowing you to monitor performance across different platforms.
  • Integration with Other Firebase Services: It can be integrated with other Firebase services like Firebase Crashlytics to provide more comprehensive insights about your appโ€™s performance.

:spiral_notepad: 2/3

But what if you want to use something without including a new SDK in the app?

Well, I would still strongly advise using Firebase Performance Monitoring SDK or something similar in the project. Or at least plan to add this SDK in the near future.

However, there are situations where itโ€™s not feasible or even possible, so there are two other ways:

  • Apptim, or something similar
  • Cloud device labs

Apptim - or something similar
You can use ready-to-go tools like Apptim to just hook up to your local mobile device and measure the performance of the app.

But you need to check if the app is behaving in the same way with and without this tool hook-up to your device. Iโ€™ve stopped using Apptim as it was slowing down my clientโ€™s apps way too much, most of the time it was close too double the speed of each screen loading and each animation. Which made it unreliable to use on day to day basis on during RC testing stage.

Cloud device labs
Various cloud device labs offer the ability to measure and record the performance of the app.

Itโ€™s up to you what you need to measure, how and when.

  • Manual tests?
  • Automated tests?
  • Regularly or not?

For more, you can check:


:spiral_notepad: 3/3

You can also check other tools that you can use. List of example tools from one of my presentations:

1 Like

Hello, @pwicherski Thank you for your very detailed explanation on my query .
It surely did open a few possibilities for me to try out.

2 Likes

I found the clip from QAFox - but itโ€™s 90 minutes long ๐Œ๐จ๐›๐ข๐ฅ๐ž ๐€๐ฉ๐ฉ ๐๐ž๐ซ๐Ÿ๐จ๐ซ๐ฆ๐š๐ง๐œ๐ž ๐“๐ž๐ฌ๐ญ๐ข๐ง๐  ๐ฎ๐ฌ๐ข๐ง๐  ๐€๐ฉ๐ฉ๐ญ๐ข๐ฆ (By NITHIN S.S) - YouTube skip to around 14:30 mark when talking about performance starts, unfortunately the audio quality is poor, so it really fails to teach anything useful and runs more like a product training tutorial than an example of what is possible. The big takeaway I did get is that Apptim is easy to use, and for example compare performance of different versions of your app for regression, and how your app behaves on different phones. It will help you find memory leaks and crashes too for example. But the biggest work will be in getting agreement with your developers and the product owner to agree what performance really looks like in terms of the nice graphs and reports Apptim produces. Apptim also produces a video of your session, which I felt was overkill.

I would like to see some open source tooling, but actually your best bet is to actually use the apple IDE and itโ€™s performance tools in interactive mode, because they are pretty good. As @pwicherski points out Android Studio is a pretty good starting point for one-off performance picture as well.
But only interactive in the apple case, and I think thatโ€™s important, because a once-off performance health check of your app is probably the biggest bang for your buck. Setting up a CI/CD performance tool is often a thankless task. Most teams set one up, and then have it rot away after a year or two as people loose interest. That alone explains why not a lot of talk about mobile performance testing actually goes on. Everyone is probably just embarrassed to talk about how theirs died. Android, lots of data can be retrieved using tools over the android debug bridge. Thatโ€™s an exercise worth looking into in more detail, be keen to learn of any quick wins there as these become really easy to add to any existing test especially if you have rolled your own test framework and built a local test farm.

1 Like