Performance: which testing tool and why?

So, what would you recommend for performance testing with Citrix? Approach and Tool? Or reading ressources?

1 Like

For Citrix testing both Loadrunner and IBM Rational Performance Tester have a module for it , i’ve tried IBM RPT with Citrix years ago it was working but little annoying , you can try Load Runner community version as a proof of concept for your case , if it is working with you , you can decide if 50 users is sufficient or you need a license for more users.

LR Tips article :
https://admhelp.microfocus.com/lr/en/12.53/help/WebHelp/Content/VuGen/128100_c_citrix_recording_tips.htm

Hope you find this useful :slight_smile:

1 Like

Hi, thanks for that. So why do we need special tools for performance testing citrix, is it because of the special protocol?

1 Like

Yes , most of citix implementations are 2 parts the xenapp which is a web interface and the citrix apps themselves which is most of the times desktop apps , that’s why a special protocol is needed.

Thanks ,
Mohamed

1 Like

While considering all above comments, the below attached resources might be also helpful for some of the folks

2 Likes

Definitely a consideration, especially in highly regulated orgs such as financial sector.

2 Likes

Hi all,

great thread and collection and tools!

I think for the tool selection it’s also important to know the limits of the way of the load generation.

All tools that model virtual users through threads (i.e. JMeter) and multiple “real users” gets simulated by repeating steps on the same thread are in a feedback loop with the the system under test. So when the system reaches it’s capacity, the load generator gets throttled by the system under test because following requests (and users) have to wait before previous requests gets responded to (synchronous IO). By this it might get hard to overload the system (depending on the dominant bottleneck).
It’s not per-se a problem as there are ways to circumvent this problem (1 thread per user, distributed load) but all require a bit more upfront thinking and more HW resources.
Tools in this category are

  • JMeter
  • Grinder
  • wrk

Tools which do not generate the load by thread-user mapping suffer less from this problem. They don’t wait for a response in order to sent a new requests (asynchronous IO) and have no feedback loop. These allow to model load by arrival rate (users per second) instead of concurrent users (max users) - actually they support both ways.

Tools in this category are

  • Gatling (Akka models concurrency using the actor pattern instead of thread, underlying Netty is async IO)
  • Tsung (Erlang models concurrency using the actor pattern instead of threads)
  • k6 (I don’t know for sure, but as it’s written in Go, so I assume it uses async IO)
  • locust

I personally enjoy working with Gatling, it’s Scala DSL allows well readable Scripts with the full power and flexibility of a programming language (Scala), i.e. generate load profiles or request sequences using algorithms (math functions, dynamic sequences).

6 Likes

While you point out a potential problem with such threading models for load generation for workloads that do not include “think time” in the pacing of the virtual users, the limitations aren’t really critical for more real-world scenarios where the virtual user thread has pauses, sleeps or delays between requests.

The majority of tools that you point out (like JMeter or LoadRunner and others) where the main virtual user session is held by a single thread in a thread group are still loading an http stack or browser assembly that supports multiple, asynchronous, concurrent connections to the system under test.

In LoadRunner - you’d look for an output message:
“Maximum number of concurrent connections per server: 6 [MsgId: MMSG-26000]”

and functions like:
web_set_sockets_option(“MAX_CONNECTIONS_PER_HOST”,”10”);
web_set_sockets_option(“MAX_TOTAL_CONNECTIONS “,”60”);"

In Jmeter - you look for the settings:

When combined with ample think-time and pacing in the workload design, then your load generation will be less forgiving to any slow down in the system under test.

3 Likes

Reviving this thread as I’ve seen some other suggestions pop up on Slack

2 Likes

A nice concise blog post comparing Locust to Jmeter. It presents an easy to read table comparison and some considerations for each tool.

Also an interesting summary post here from @mintgreenmonkey

3 Likes

And a follow on from Rafaela with Jmeter vs Gatling

3 Likes

I’m currently being challenged by our company to investigate some possible perf testing options, so I just wanted to say thanks for keeping this thread going over time. It looks like it will be very helpful to me, especially your latest post as jmeter was definitely one I was thinking of.

3 Likes

Oh cool! Interested to hear what tool you choose when the time comes :grinning:

3 Likes

Another great insight blog. This time from @marie.drake about Taurus

3 Likes

Hello,

As a performance tester, I refer JMeter performance testing tool which is free to use and easy to use also. The tool is specifically used for web and mobile app performance testing.

Why I refer to JMeter Performance Testing Tool?

  • It supports multiple load injectors
  • Less scripting :slightly_smiling_face:
  • Easy to Understand statistics
  • Supports Integrated real-time

I just Overview the one article which is related to JMeter Performance Testing Tool: https://www.testrigtechnologies.com/performance-analysis-load-testing-with-jmeter/

1 Like

Just replying like I said I would having done some testing. I chose Gatling. I think my experience of it could be summarized by their strapline ‘load test as code’. That wee snippet makes perfect sense when you experience the difference between them and Jmeter, detailed differences for which are set out in the article Heather mentioned above, i.e. https://azevedorafaela.com/2020/06/22/load-tests-jmeter-vs-gatlin/.

My test mandate was essentially “we are in a time crunch and need to quickly establish if the new API-based architecture will be performant when the old architecture wasn’t”. Therefore, I had to do some, erm, exploratory performance testing, for which the power and ease of use of Gatling really helped me get some useful rough measurements to help justify a more thorough investigation of performance later!

The following 2 pages were very helpful in me getting set up and going with Gatling: https://gatling.io/docs/current/quickstart, https://www.james-willett.com/gatling-load-testing-complete-guide/#71-basic-load-simulation

Here’s a little code snippet that should be pretty intuitive and you can see the power Gatling gives you in just a few lines of code (the language is Scala by the way, which is pretty intuitive to use):

In the above we see on lines 31 and 32 the power of Gatling. On line 31 is a spike-type load test, i.e. we will hit the test server with 100 users all making the same call to the meetings API endpoint at one time. On line 32 we see how in just one line of code we can do a constant load test over 5 minutes during which we will have a total of 10000 users. Very powerful stuff … and very efficient to create the tests as well.

PS the Postman stuff is in there just because the very helpful Gatling recorder put it there when I got it to record my Postman API call. Basically by recording Postman I had a usable test case that I then tweaked to make a spike or a load test. Pretty neat! …And not that scary either :slight_smile: I had expected to lose a lot of time using Gatling based on a previous expeience of Jmeter, which because it has a lot more features it demands a lot more knowledge on part of the user, i.e. me in this case. I would say that to be vaguely productive in Jmeter would require a noob to watch a video/demo and spend some hours messing about whereas Gatling would be reading a bit of those couple of websites I mentioned above and you can get productive within maybe an hour!

I would agree with the conclusions of the Gatling vs Jmeter article that your requirements as a tester will heavily dictate which tool you choose. I definitely have a place in my heart for Jmeter and I am sure I’ll use it again in future, but for a quick blast of exploratory performance testing, Gatling was very helpful.

Hope this helps others :slight_smile:

3 Likes

Love this Gerard! Thanks a million for sharing :grin:

1 Like

Another excellent insight on tooling from @marie.drake

3 Likes

Saw a great blog post today from @constancehermit, the first in a series, which talks about Gatling.

https://theartfultester.com/blog/2021/01/05/load-testing-with-gatling-1/

2 Likes

This thread is amazing! Currently we use jMeter but I’m looking for an alternative when we soon need to replace our capacity planning suite.

My initial default choice is likely to be https://artillery.io/ because the team has some prior experience and it installs via npm so fits well with our standard process.

But will definitely checkout some of these alternatives. I’ve looked at locust.io a few times and I like it. Personally I’m a Python fan but there is less experience of it in my team.

3 Likes