With my pro Dojo membership I have been watching the Performance Testing 101 series. Iāve found it quite useful as a getting started with performance testing but also an introduction to JMeter. Iāve yet to start using it on a product but so far itās making sense.
More and more Iām seeing people ask which tool to use and why. @mwinteringham posted in a slack channel today āI would like to hear arguments for why other opensource tools might be a better option to JMeterā.
Iāve seen people recommend JMeter, Gatling, locust.io and Artillery. Iāve also probably missed some tools there but these are the most common Iām seeing. Do you use any of these tools? Do you have any arguments for or against any of them?
After using so many different performance tools, there are a few concepts to help guide your choosing of a right fit:
financial fit: do you need free? is it really free (like puppies are free) or do you need to invest other resources to make the tool really fly?
scripting fit: do you know how to program inside the script language? are you an old C programmer, Javascript, .NET, Java, Ruby, Python?
accuracy fit: is the tool going to measure exactly the transactions you want to simulate, or just generate load generally with some overhead? what kind of accuracy do you need?
scalability fit: will the tool run enough load for your simulations? can you expand the tool on more load generators?
extensibility fit: do you need to customize the tool to fit with other tools in your ecosystem? how difficult is it to do that, if you need?
collaboration fit: when you need help from a developer or database engineer, will they understand the tool you are using? Developers hate editing JMeter .xml
reporting fit: is the tool equipped to give robust reporting, or not so much? can you DIY in other tools to get your visualizations and reporting needs met?
For me, my current gig has been JMeter for about 5 years and itās met my needs. Moving forward, weāre looking at locust.io - for 3 reasons:
- the programming is quite straight-forward code, even if we donāt have python skills on the team - itās way better than JMeter XML in the IDE
- the platform fit works well because our workloads seem to be moving towards the Google cloud and GCP + Docker and K8
- itās free to us, weāre already investing in the infrastructure for Docker, K8 and Teraform ā¦so this would just be another pod on top of that architecture
This is a topic for a 1hr talk at least in my opinion By design GoReplay takes traffic from production servers, whereas the other tools use synthetic traffic generated from test environments or by the tester/developer.
So you need to answer the question mtomlins suggested:
accuracy fit: is the tool going to measure exactly the transactions you want to simulate, or just generate load generally with some overhead? what kind of accuracy do you need?
To help address your question without talking for hours, we need to unpack your question. Why did you want to learn about these tools? Why do you need to learn about performance testing?
My colleagues were using IBM Rational Performance testing for a while. When trying to do recently a performance testing evaluation for a client they noticed a lot of issues with it. They went to Neoload and got the Trial version which ran smoothly. They are thinking of getting a license of it.
HP LoadRunner is supposed to be the most used performance tool(around 60% of the market Iāve read in a charter).
Iāve personally only played with tools like Jmeter, Soap./Smartbear Load Testing, browser development tools assesments and Gatling. The thing is, with free tools, you have to know to code in various languages and make configurations and updates,add constantly libraries and scriptsā¦and it might still not be satisfiable enough.
Not sure which one is a better approachā¦paid -> click and run or free -> configureā¦and eventually run.
Havenāt heard Taurus mentioned here yet - itās a free tool written by Blazemeter (you can run tests through their site for money, but you can also just run them using Taurus on your own machine) that Iāve been using recently. Itās basically a wrapper for other tools such as JMeter, Gatling, Locust, Grinder, etc.
Iāve found it useful as it meant I could set up config in a YAML file (also supports JSON), and make it easier for devs using my tests to tweak values for throughput, duration etc. Also, I liked using Blazemeterās reporting, which you can use even if you donāt have an account (they just wonāt save the data for you).
As a newbie, I found it really useful, and I liked that it was easy for me to switch executors from e.g. JMeter to Gatling when I couldnāt figure out how to get JMeter to do what I wanted.
I think Mark summed up the list of criteria pretty well . I would only add one-
organisational fit: will your organisationās security/IT team permit use of the tool?
Our organisation is now starting to use the performance test tools within Visual Studio to replace StresStimulus, which weād piloted earlier. Why? because we used pretty much the criteria Mark outlined to determine our tool. It is (to us) free as we all have VS licenses. Our developers know it inside out, and our testers are learning to use it, with loads of help from the developers to pick up coding. The tool generates enough load for our purposes, and is scalable enough for what we need. It ties in with VS and TFS reporting, so no additional overhead generating reports, and, best of all, our developers understand the tool as it is their tool.
Loadster tool is a cloud based solution for distributed load testing of web applications. It records the test scripts in a browser and execute them in the graphical script editor. Then, it automatically build test scenarios to simulate thousands of concurrent virtual users executing the test scripts. Software Tester run the test and watch the dashboard to see how the application is behaving. Once execution finished, you can export the results to an HTML report with graphs and tables for analysis. Software testing companies uses it to watch the applicationās performance, find issues, and validate the application before production.
Why Loadster:
Loadster records the test scripts with any available browser.
Loadster provides capability to edit and customize the test scripts.
Loadster gets the dynamic values from HTTP responses.
Loadster provides capability to import the data from system.
Loadster builds the test scenarios to simulate the large number of users.
Software tester is able to watch the live performance of tool on dashboard with information of CPU utilization, Thread count, Memory utilization and Thread count.
It provides feature to to display live graphs showing the details of virtual group.
Loadster has a repository called project repository for each project scripts in which test results are saved after completing the load test, which software test is able to export the results.
Loadster provides capability to record the scripts.
A test report is saved in the repository when the load test is completed.
Hope this information is clear and you can get back to us in case need more information.
I think that you should choose the performance testing tool according to the usage.
*JMeter is the best for a web apps performance testing which is not required any special handlings or protocols (like Citrix , SAP and Oracle suites)
*Visual Studio is useful for .Net technologies and sharepoint users
*Loadrunner is useful for special protocols as mentioned earlier but it is expensive and for the common web technologies as the othe tools.
For sure when needing a huge number of vusers we should use a cloud based testing and all the mentioned tools has a cloud testing service
But a personal preference i like JMeter more than any other Performance testing tools
For Citrix testing both Loadrunner and IBM Rational Performance Tester have a module for it , iāve tried IBM RPT with Citrix years ago it was working but little annoying , you can try Load Runner community version as a proof of concept for your case , if it is working with you , you can decide if 50 users is sufficient or you need a license for more users.
Yes , most of citix implementations are 2 parts the xenapp which is a web interface and the citrix apps themselves which is most of the times desktop apps , thatās why a special protocol is needed.