Test.Bash(); 2021 - What’s Your Approach to Comparing Tools with Each Other? - Matthew Churcher

Matthew Churcher has shared his thoughts around how he compares tools against one another.

We’ll use this Club thread to share resources mentioned during the session and answer any questions we don’t get to during the live session.


Questions answered during the talk:

  1. Matthew, does the available test environments factor into your choice of test tool at all? Only local env vs functional test env vs dedicated perf env vs cloud?

  2. Do you have an explanation for the big difference between wrk and the other tools?

  3. So, k6 is the winner! would you recommend using it in all cases?

  4. Your knowledge is amazing - do you ever need to convince other folks at your company what tools to use?

  5. I know its not opensource but whats your opinion on flood.io?

Questions we didn’t get to:

  1. Any opinion on Blazemeter Taurus?

Resources mentioned:


So Taurus is fine thing. As I understand it is more an automation tool. It abstracts you away from Jmeter and lets you specify simple tests as configuration files for easy use in your CI/CD pipelines. It hasn’t had much adoption away from the originating vendor though and it’s not a problem I have needed to solve.

My tests are usually complex, I work for a cloud telephony provider, and previously worked on remote desktops. I usually have complex API calls and client side logic. Websocket support is usually a must. We also mix in other tools like SiPP. So custom scripting is pretty much a given.
Being able to swap out the engine of the test isn’t something we have needed. I’m happy finding one tool that can do the job, as long as it’s not locking us into a specific vendor.

To add to my previous answers….
Commercial vendors and cloud offerings have their place. I’m lucky enough to be able to spin up AWS, GKE or Azure resources when I need them and we have the expertise in teams to do that a low over heard. That saves money on egress costs as well as licenses, and gives more consistent results. That means I have limited personal experience with any of the vendors as a customer.

I did work for a cloud performance testing company and consultancy for a while though. Where they really added value was big events like black friday and site launches. Other providers also do a great job serving small companies who just want their public site tested and don’t have their own infrastructure or in house testers.

If you have locally hosted infrastructure, something in your datacenter or something behind a firewall. Most cloud providers will have a way to test that. But Yeah a public sites are always going to be easiest from a cloud testing provider.


p.s. Anyone want more on k6. I’m speaking at grafana observabilitycon next month. Tickets are free.