I am trying to performance test an application built using an event driven architecture and one of the problems we are having is identifying which data we use to generate number of users per second.
We are using artillery as the tool we use to run these tests, and our tests are opening a socket connection and directly targeting events on a page to generate load.
The cases that we are trying to test for are the user experience at low, medium and high usage as well as what happens to page speed when we have an influx of users.
The trouble we have is that there are conflicting opinions on what ‘usage’ is. For instance do we use the number of site sessions as an indicator of how many events to send per second, or do we use the number of page views per second etc etc.
Would appreciate any ideas from anyone who has done similar and what was used as the usage figure, thanks!
What I’ve done is used the number of site sessions to decide the concurrent usage, but to represent the page views by making each virtual user trigger events in the order of a typical flow. For my context that seemed to be the optimum of realism versus effort.
So I took some of the most crucial journeys through the application, defined page views and events for them with some wait times in between (that were random within a range, to try and reflect real users), then calculated concurrent usage (per second) based on Sessions Per Hour (for normal and peak usage and in your case I see you want low usage as well) x Average Visit Duration (seconds) and then divide the result by 3600, added some ramping, then I would spawn virtual users to the concurrent usage results I got and had those virtual users execute the journeys I defined.
That’s a really interesting approach. We have now decided to use page views as the measure for the numbers of users interacting with the site per second.
Our approach is:
Get the number of ‘active users’ (people interacting with the site based on these page views) for each of the scenarios we are doing (low, average, high load + ramp up)
Work this out as a per second figure
We are targeting several core pages so will split this load between the pages for each scenario: IE If low load is 10 active users per second and we are testing 5 pages, we will use 2 virtual users per second targeting the events triggered on each of these pages concurrently
We think this should give us an indication of load across the site and allow us to test the ‘user experience’ under these scenarios
We can measure the performance in cloudwatch but also intend to navigate the site as a ‘user’ whilst under the load and view the real time load times in a tool called datadog, measuring the time it takes for largest content on the page to load as our metric (LCP)
Wow, nice to see new club joiners jumping in to help gain confidence for decisions. Welcome to the club, keenly await some learnings myself as I’m a newbie to web QA, and hope we get our regular folk who have used Datadog and Gatling before to share too.
A thing to consider is, that a visitor is not using the most optimal journey. E.g. if I want to watch a movie, then I want to read about the stories of several movies and pick the most interesting movie. This behaviour should also be modeled in the load. After reading the first story there is a chance of 70 percent that the user want to read another story.
Another thing to consider is, that the focus of the load should be on actions using the most resources like CPU, RAM, or network. E.g. an inventarisation is done once a day, but it could have a major impact on the performance of the system.
Another thing to consider is to randomize your test data. If the same data is requested during a performance test, then this data will be placed into a cache. The next time the data is shown faster than using some service. E.g. if all visitors of a cinema always ask information about the same movie, than this information might be retrieved from the cache.
In case of a heavily used web site it is important to take load balancing into account. The load must represent the users. During the performance test, unfortunate situations are avoided. E.g. the American server handles the requests of American and European visitors. In the meantime the European server has no load.