Performance: which testing tool and why?

With my pro Dojo membership I have been watching the Performance Testing 101 series. I’ve found it quite useful as a getting started with performance testing but also an introduction to JMeter. I’ve yet to start using it on a product but so far it’s making sense.

More and more I’m seeing people ask which tool to use and why. @mwinteringham posted in a slack channel today “I would like to hear arguments for why other opensource tools might be a better option to JMeter”.

I’ve seen people recommend JMeter, Gatling, locust.io and Artillery. I’ve also probably missed some tools there but these are the most common I’m seeing. Do you use any of these tools? Do you have any arguments for or against any of them?

3 Likes

I would also add https://goreplay.org to that list if you want to test your systems with production-like traffic.

2 Likes

Is there any reason you’d pick it over one of the others? Or is it a personal preference thing? :slight_smile:

1 Like

After using so many different performance tools, there are a few concepts to help guide your choosing of a right fit:

  • financial fit: do you need free? is it really free (like puppies are free) or do you need to invest other resources to make the tool really fly?
  • scripting fit: do you know how to program inside the script language? are you an old C programmer, Javascript, .NET, Java, Ruby, Python?
  • accuracy fit: is the tool going to measure exactly the transactions you want to simulate, or just generate load generally with some overhead? what kind of accuracy do you need?
  • scalability fit: will the tool run enough load for your simulations? can you expand the tool on more load generators?
  • extensibility fit: do you need to customize the tool to fit with other tools in your ecosystem? how difficult is it to do that, if you need?
  • collaboration fit: when you need help from a developer or database engineer, will they understand the tool you are using? Developers hate editing JMeter .xml
  • reporting fit: is the tool equipped to give robust reporting, or not so much? can you DIY in other tools to get your visualizations and reporting needs met?

For me, my current gig has been JMeter for about 5 years and it’s met my needs. Moving forward, we’re looking at locust.io - for 3 reasons:
- the programming is quite straight-forward code, even if we don’t have python skills on the team - it’s way better than JMeter XML in the IDE
- the platform fit works well because our workloads seem to be moving towards the Google cloud and GCP + Docker and K8
- it’s free to us, we’re already investing in the infrastructure for Docker, K8 and Teraform …so this would just be another pod on top of that architecture

HTH,

-mt

11 Likes

This is a topic for a 1hr talk at least in my opinion :slight_smile: By design GoReplay takes traffic from production servers, whereas the other tools use synthetic traffic generated from test environments or by the tester/developer.

So you need to answer the question mtomlins suggested:

accuracy fit: is the tool going to measure exactly the transactions you want to simulate, or just generate load generally with some overhead? what kind of accuracy do you need?

To help address your question without talking for hours, we need to unpack your question. Why did you want to learn about these tools? Why do you need to learn about performance testing?

3 Likes

My colleagues were using IBM Rational Performance testing for a while. When trying to do recently a performance testing evaluation for a client they noticed a lot of issues with it. They went to Neoload and got the Trial version which ran smoothly. They are thinking of getting a license of it.
HP LoadRunner is supposed to be the most used performance tool(around 60% of the market I’ve read in a charter).
I’ve personally only played with tools like Jmeter, Soap./Smartbear Load Testing, browser development tools assesments and Gatling. The thing is, with free tools, you have to know to code in various languages and make configurations and updates,add constantly libraries and scripts…and it might still not be satisfiable enough.
Not sure which one is a better approach…paid -> click and run or free -> configure…and eventually run.

1 Like

Thank you Mark :slight_smile: I hope some of this is what you were looking for @mwinteringham?

1 Like

Haven’t heard Taurus mentioned here yet - it’s a free tool written by Blazemeter (you can run tests through their site for money, but you can also just run them using Taurus on your own machine) that I’ve been using recently. It’s basically a wrapper for other tools such as JMeter, Gatling, Locust, Grinder, etc.

I’ve found it useful as it meant I could set up config in a YAML file (also supports JSON), and make it easier for devs using my tests to tweak values for throughput, duration etc. Also, I liked using Blazemeter’s reporting, which you can use even if you don’t have an account (they just won’t save the data for you).

As a newbie, I found it really useful, and I liked that it was easy for me to switch executors from e.g. JMeter to Gatling when I couldn’t figure out how to get JMeter to do what I wanted.

4 Likes

A post from Joe Colantonio with a brief why you’d use this tool for each

2 Likes

As per @mtomlins comments,

I think Mark summed up the list of criteria pretty well :grin:. I would only add one-

  • organisational fit: will your organisation’s security/IT team permit use of the tool?

Our organisation is now starting to use the performance test tools within Visual Studio to replace StresStimulus, which we’d piloted earlier. Why? because we used pretty much the criteria Mark outlined to determine our tool. It is (to us) free as we all have VS licenses. Our developers know it inside out, and our testers are learning to use it, with loads of help from the developers to pick up coding. The tool generates enough load for our purposes, and is scalable enough for what we need. It ties in with VS and TFS reporting, so no additional overhead generating reports, and, best of all, our developers understand the tool as it is their tool.

2 Likes

Loadster tool is a cloud based solution for distributed load testing of web applications. It records the test scripts in a browser and execute them in the graphical script editor. Then, it automatically build test scenarios to simulate thousands of concurrent virtual users executing the test scripts. Software Tester run the test and watch the dashboard to see how the application is behaving. Once execution finished, you can export the results to an HTML report with graphs and tables for analysis. Software testing companies uses it to watch the application’s performance, find issues, and validate the application before production.

Why Loadster:

  • Loadster records the test scripts with any available browser.
  • Loadster provides capability to edit and customize the test scripts.
  • Loadster gets the dynamic values from HTTP responses.
  • Loadster provides capability to import the data from system.
  • Loadster builds the test scenarios to simulate the large number of users.
  • Software tester is able to watch the live performance of tool on dashboard with information of CPU utilization, Thread count, Memory utilization and Thread count.
  • It provides feature to to display live graphs showing the details of virtual group.
  • Loadster has a repository called project repository for each project scripts in which test results are saved after completing the load test, which software test is able to export the results.
  • Loadster provides capability to record the scripts.
  • A test report is saved in the repository when the load test is completed.

Hope this information is clear and you can get back to us in case need more information.

3 Likes

I think that you should choose the performance testing tool according to the usage.

*JMeter is the best for a web apps performance testing which is not required any special handlings or protocols (like Citrix , SAP and Oracle suites)
*Visual Studio is useful for .Net technologies and sharepoint users
*Loadrunner is useful for special protocols as mentioned earlier but it is expensive and for the common web technologies as the othe tools.

For sure when needing a huge number of vusers we should use a cloud based testing and all the mentioned tools has a cloud testing service

But a personal preference i like JMeter more than any other Performance testing tools :slight_smile:

3 Likes

Is there a reason that you personally prefer JMeter? :smiley:

2 Likes

Yes :smiley: , the level of customization , the 3rd party plugins , it is free and last thing is you can debug more deeply than other tools.

Enough ? :smiley:

3 Likes

So, what would you recommend for performance testing with Citrix? Approach and Tool? Or reading ressources?

1 Like

For Citrix testing both Loadrunner and IBM Rational Performance Tester have a module for it , i’ve tried IBM RPT with Citrix years ago it was working but little annoying , you can try Load Runner community version as a proof of concept for your case , if it is working with you , you can decide if 50 users is sufficient or you need a license for more users.

LR Tips article :
https://admhelp.microfocus.com/lr/en/12.53/help/WebHelp/Content/VuGen/128100_c_citrix_recording_tips.htm

Hope you find this useful :slight_smile:

1 Like

Hi, thanks for that. So why do we need special tools for performance testing citrix, is it because of the special protocol?

1 Like

Yes , most of citix implementations are 2 parts the xenapp which is a web interface and the citrix apps themselves which is most of the times desktop apps , that’s why a special protocol is needed.

Thanks ,
Mohamed

1 Like

While considering all above comments, the below attached resources might be also helpful for some of the folks

2 Likes

Definitely a consideration, especially in highly regulated orgs such as financial sector.

2 Likes