Performance: which testing tool and why?

Hi all,

great thread and collection and tools!

I think for the tool selection it’s also important to know the limits of the way of the load generation.

All tools that model virtual users through threads (i.e. JMeter) and multiple “real users” gets simulated by repeating steps on the same thread are in a feedback loop with the the system under test. So when the system reaches it’s capacity, the load generator gets throttled by the system under test because following requests (and users) have to wait before previous requests gets responded to (synchronous IO). By this it might get hard to overload the system (depending on the dominant bottleneck).
It’s not per-se a problem as there are ways to circumvent this problem (1 thread per user, distributed load) but all require a bit more upfront thinking and more HW resources.
Tools in this category are

  • JMeter
  • Grinder
  • wrk

Tools which do not generate the load by thread-user mapping suffer less from this problem. They don’t wait for a response in order to sent a new requests (asynchronous IO) and have no feedback loop. These allow to model load by arrival rate (users per second) instead of concurrent users (max users) - actually they support both ways.

Tools in this category are

  • Gatling (Akka models concurrency using the actor pattern instead of thread, underlying Netty is async IO)
  • Tsung (Erlang models concurrency using the actor pattern instead of threads)
  • k6 (I don’t know for sure, but as it’s written in Go, so I assume it uses async IO)
  • locust

I personally enjoy working with Gatling, it’s Scala DSL allows well readable Scripts with the full power and flexibility of a programming language (Scala), i.e. generate load profiles or request sequences using algorithms (math functions, dynamic sequences).

4 Likes

While you point out a potential problem with such threading models for load generation for workloads that do not include “think time” in the pacing of the virtual users, the limitations aren’t really critical for more real-world scenarios where the virtual user thread has pauses, sleeps or delays between requests.

The majority of tools that you point out (like JMeter or LoadRunner and others) where the main virtual user session is held by a single thread in a thread group are still loading an http stack or browser assembly that supports multiple, asynchronous, concurrent connections to the system under test.

In LoadRunner - you’d look for an output message:
“Maximum number of concurrent connections per server: 6 [MsgId: MMSG-26000]”

and functions like:
web_set_sockets_option(“MAX_CONNECTIONS_PER_HOST”,”10”);
web_set_sockets_option(“MAX_TOTAL_CONNECTIONS “,”60”);"

In Jmeter - you look for the settings:

When combined with ample think-time and pacing in the workload design, then your load generation will be less forgiving to any slow down in the system under test.

2 Likes

Reviving this thread as I’ve seen some other suggestions pop up on Slack

A nice concise blog post comparing Locust to Jmeter. It presents an easy to read table comparison and some considerations for each tool.

Also an interesting summary post here from @mintgreenmonkey

1 Like

This post was flagged by the community and is temporarily hidden.

And a follow on from Rafaela with Jmeter vs Gatling

1 Like

I’m currently being challenged by our company to investigate some possible perf testing options, so I just wanted to say thanks for keeping this thread going over time. It looks like it will be very helpful to me, especially your latest post as jmeter was definitely one I was thinking of.

1 Like

Oh cool! Interested to hear what tool you choose when the time comes :grinning:

1 Like