Performance: which testing tool and why?

Hi all,

great thread and collection and tools!

I think for the tool selection it’s also important to know the limits of the way of the load generation.

All tools that model virtual users through threads (i.e. JMeter) and multiple “real users” gets simulated by repeating steps on the same thread are in a feedback loop with the the system under test. So when the system reaches it’s capacity, the load generator gets throttled by the system under test because following requests (and users) have to wait before previous requests gets responded to (synchronous IO). By this it might get hard to overload the system (depending on the dominant bottleneck).
It’s not per-se a problem as there are ways to circumvent this problem (1 thread per user, distributed load) but all require a bit more upfront thinking and more HW resources.
Tools in this category are

  • JMeter
  • Grinder
  • wrk

Tools which do not generate the load by thread-user mapping suffer less from this problem. They don’t wait for a response in order to sent a new requests (asynchronous IO) and have no feedback loop. These allow to model load by arrival rate (users per second) instead of concurrent users (max users) - actually they support both ways.

Tools in this category are

  • Gatling (Akka models concurrency using the actor pattern instead of thread, underlying Netty is async IO)
  • Tsung (Erlang models concurrency using the actor pattern instead of threads)
  • k6 (I don’t know for sure, but as it’s written in Go, so I assume it uses async IO)
  • locust

I personally enjoy working with Gatling, it’s Scala DSL allows well readable Scripts with the full power and flexibility of a programming language (Scala), i.e. generate load profiles or request sequences using algorithms (math functions, dynamic sequences).

4 Likes

While you point out a potential problem with such threading models for load generation for workloads that do not include “think time” in the pacing of the virtual users, the limitations aren’t really critical for more real-world scenarios where the virtual user thread has pauses, sleeps or delays between requests.

The majority of tools that you point out (like JMeter or LoadRunner and others) where the main virtual user session is held by a single thread in a thread group are still loading an http stack or browser assembly that supports multiple, asynchronous, concurrent connections to the system under test.

In LoadRunner - you’d look for an output message:
“Maximum number of concurrent connections per server: 6 [MsgId: MMSG-26000]”

and functions like:
web_set_sockets_option(“MAX_CONNECTIONS_PER_HOST”,”10”);
web_set_sockets_option(“MAX_TOTAL_CONNECTIONS “,”60”);"

In Jmeter - you look for the settings:

When combined with ample think-time and pacing in the workload design, then your load generation will be less forgiving to any slow down in the system under test.

2 Likes