When I go looking for a new tool it’s because I have a new problem (or a problem has gotten too big to ignore any more!), so my first step is going to be thinking about what the problem I really want to solve is (looking at root-causes and deciding where I need need a tool to gather more information or intervene in a process). Then I’ll look around for solutions and my first pass will probably include:
Does it sound like it has the features to help me?
Does anyone I know say it’s good?
Is it used by a lot of people?
Does it get good online reviews?
Relative costs vs features of available solutions.
Bring this info back to my team for feedback.
Then I’ll give it a go and see if it works for me/us. Will start on a small scale and go from there.
I usually find it doesn’t take long to decide if there are any showstoppers to using a new tool. A day or two maybe, and after that I guess I’m going to keep using it until it starts not delivering what I need (as our needs change) or something better comes along.
The biggest turn-off when evaluating a new tool would be negative feedback from people I know. But also, no easy access to a free trial is also a big hurdle.
(Oh, and it better be pretty intuitive, 'cause let’s be honest, there’s no way I’m reading more than a couple of paragraphs to get started! Although tips along the way are welcome. )
How do you discover all the things you like/dislike about a new tool?
What do you rely on to make a reasonable evaluation of a new tool?
How do you know you’ve found “the one”?
I like our process for tool decisions. We spin up a Notion page detailing the problem we’re trying to solve, requirements, definition of success, and timelines.
Then we spike out a few options.
For example, in the case of an E2E test framework, we wrote a single test using each of WebDriver.IO, Cypress, Playwright and Katalon Studio.
We update the Notion page with pros and cons of each. Things like learning curve, documentation and supportability, cost, feature set, and so on.
Then, based on our findings, we pick something!
We then share the Notion page with a wider audience for feedback.
This is to ensure that anyone can object or raise concerns that haven’t been thought of.
It might sound strange to share findings after a decision has been made, but, we’ve found it saves on friction. Our team has a bit of history of ‘deadlocking’ on tool decisions when parties can’t agree.
What is the biggest turn-off when evaluating a new tool?
My biggest turn off is poor documentation / support. A tool might be great, but if I can’t find examples of how to do the things I need, I get frustrated pretty quickly.
Another turn off is not being able to find pricing easily. Makes it really hard to state any sort of case for investing time or energy.
I love to explore new tools, like exploratory testing I ask for feedback from relevant communities, read documentation, search online…
Can the tool help me? How much is it? Is it working well?
It is never “the one” there is still pros and cons
Time is my biggest problem. But I do agree poor documentation/support=>NO GO!
How do you discover all the things you like/dislike about a new tool?
I usually start by checking the docs (if any), reading the review and then just trying out the tool for a while. Trying it out in a real-world scenario is the most useful way of discovering if the tool is right for my current needs, or not. But, trying out a bunch of tools can be time-consuming so sometimes I’d just ask for tool recommendations from people in the testing community, especially the people I know well and I respect their expertise.
What do you rely on to make a reasonable evaluation of a new tool?
Ease of use, and how much value do I get out a tool, if it’s a paid tool try to figure what’s the ROI
How do you know you’ve found “the one”?
If the tool offers a good user experience, sort of speaking, and if using it makes certain kinds of tasks a lot easier to do.
What is the biggest turn-off when evaluating a new tool?
False advertisement, such as restrictive terms and conditions in a very ting font-size!
Reading articles with tables comparing tools doesn’t do it for me. I need to take a small problem I care about solving and apply it as a proof of concept. Doing this I’m able to evaluate not just how easy I can use the tool, but also how easy it would be for someone who’s never used similar tools before. During my evaluation I try to read at least 75% of the documentation, if it’s a large tool I will focus my efforts to focussing on the certain features I’m using.
How do you discover all the things you like/dislike about a new tool?
Normally, I look at the goal the tool I’m using is expected to fulfill, then I read documentation, reviews in some cases, and do my own experimentation to determine if it fits
What do you rely on to make a reasonable evaluation of a new tool?
Does it have documentation, is it actively maintained (or in-house maintenance is supportable and relatively trivial), and does it adequately fit the task it is intended for (particularly important if it’s a replacement for an existing tool)
How do you know you’ve found “the one”?
I’ve probably run out of alternatives to search for, and teh one I’ve picked does what I want well enough. I don’t think there is ever “the one” though, just “the good enough”.
What is the biggest turn-off when evaluating a new tool?
If it’s a paid-for tool, not giving sufficient evaluation periods to not just evaluate the tool myself, but to then take it up the chain to get approval for purchase. If you’re only giving me a week to decide I’m happy with a tool, it’s going to be very difficult to get me to sell it to my department.
Oh, and poor or non-existant documentation. Your tool may be perfect for what I need, but if I can’t find out how to use it, I’m probably not going to.
I don’t just look at 1 tool, I look at many and then compare all the tools on all aspects (easy to use, coverage, …)
Depends what the tool needs to do
If it’s a testing tool then assertions, reporting, coverage, friendliness, easy to use, can my team work with this? Does anyone on the team have experience with these tools?
When you got those little butterflies in your belly!
When all of your needs are covered
If it’s a GUI tool: UI/UX, it’s 2021 please fix it.
If nobody understands how it works and there is poor documentation.
How do you discover all the things you like/dislike about a new tool?
What do you rely on to make a reasonable evaluation of a new tool?
Feels like these two can answered at the same time.
Getting some hand-on time with it is the only path to truth. Doing a quick and dirty POC with a potential tool is going to give way more insights than an equivalent time spent reading documentation or evaluation articles somebody else wrote.
How do you know you’ve found “the one”?
Easy, or relatively low-friction acceptance from the team. You validate your findings, account for bias and if everybody else as hyped about it then there is something there.
What is the biggest turn-off when evaluating a new tool?
Some of these questions
Tooling red flags:
low or no community engagement, i.e. last contribution is 3+ months old, tons of GitHub issues with no answer;
outdated, non-existing or bad documentation - not sure which is worse;
pre-1.0 release (somewhat tolerable if there is a clear roadmap and vision);
How do you discover all the things you like/dislike about a new tool?
I use it in anger. I don’t generally just try new tools randomly these days. If there’s a real reason to try a new tool, it’s usually that there’s some need I have that I can’t fulfil with my current tooling. Working in embedded there is a real dearth of testing tooling available, especially in anything approaching a modern language. So a lot of my tooling is actually home rolled duct tape between different utilities.
What do you rely on to make a reasonable evaluation of a new tool?
Does it fulfil my need as well or better than my current tooling? How hard is it to make the switch? Can I just plug it into CI?
How do you know you’ve found “the one”?
I never have, some perennials like vim stay around, but for the most part, I just use whatever’s needed to get the job done.
What is the biggest turn-off when evaluating a new tool?
The time and energy needed to learn how it works and adapt any existing testing to it.
I’ve evaluated tools using a pugh matrix. I found it handy to list down what requirements we had and also what the tools provided along with support and costs involved. This let us evaluate and assess which tool scored the most but we also out in the priority of our needs to make sure we didn’t lose our focus on the needs
How do you discover all the things you like/dislike about a new tool?
Using the tool with some basic testing examples to find out if it’s fit for purpose. I may also look at customer reviews to see what others think as well as looking at any forums/docs for ideas on tool usage.
What do you rely on to make a reasonable evaluation of a new tool?
Does it do what is required for me to complete my task
Does it help me successfully complete my task faster
Easy to use
Easy to integrate / install - Chrome Extension, Website etc
How do you know you’ve found “the one”?
I am usually looking for tools for a specific task. It may be that I can use the tool for other similar things however I may have to look for an alternative depending on what is required.
What is the biggest turn-off when evaluating a new tool?
Doesn’t do what it says on the tin, Nothing worse than investing time in something that doesn’t do what you need.
I try and put it through its paces as best I can. For example, with Playwright, I tried to set up a small POC that tries to accomplish one of my tests at work. Something that might tests as many of the features as I can out in the new tool.
It is similar with non-automation tools as well. I have tried many note taking apps out, and keep coming back to Joplin. It did all the things I needed it to do including syntax highlighted code blocks.
Enough time to really use the tool, which is hard sometimes.
When it can do everything I need of it without much wiggle room required. Sometimes I am surprised that it can do more for me than expected.
Short trial periods and subscriptions. I prefer one time payments.
How do you discover all the things you like/dislike about a new tool?
The main thing I like to look at is how the tool is useful to ME. Not everyone will find the tool useful or easy to use and will have varying opinions regarding its usefulness. A lot of times, we’re told what tools to use but our line managers should still be willing to listen to feedback. If an individual doesn’t personally find the tool useful, and believe they could do the same job better without the tool or with a different tool, then they shouldn’t be forced to use it.
What do you rely on to make a reasonable evaluation of a new tool?
Activity/Task – What specific activity or task does this tool help me complete?
Improvement – What are the actual benefits of using the tool? This should be a quantifiable value like time saved. We’d also want to assess if the task could be completed without the tool.
Cost – How much does the tool cost? Is there a cheaper tool that could produce the similar outcome?
How do you know you’ve found “the one”?
Many are not given the luxury of choosing the tools they use in their jobs. It might be that the tool selection is decided by someone else, in which case all they can do is make recommendations. Often people start a new job to find that all decisions regarding tools have already been made and the tools paid for, so are forced to make do with what they are given.
If able to choose which tools to use, I would simply consider the task I am trying to complete and how that tool helps me complete the task.
What is the biggest turn-off when evaluating a new tool?
Poor UX!
Even if the tool is fit for purpose, when the user experience is so poor then any benefit to using the tool is not likely to be worth the cost.
How do you discover all the things you like/dislike about a new tool?
Learning by doing. As much as I want to believe recommendations or documentation the only way for me to figure out everything around a tool is to use it. I still also watch videos and read articles about tools to get a general understanding before I start using it
What do you rely on to make a reasonable evaluation of a new tool?
As strange as that may sound, I rely a lot on pro and contra lists, how big is the community behind the tool and what are my experiences on this topic, with which the tools are supposed to help me.
How do you know you’ve found “the one”?
Not sure if there is “the one”. All tools have their advantages and disadvantages. I think “the one” is the tool that helps me the most with the current issue I’m experiencing. But you should always keep your eyes open and evaluate from time to time if “the one” is still the best solution.
What is the biggest turn-off when evaluating a new tool?
Missing documentation📝. I think everybody knows what I mean by that.
How do you discover all the things you like/dislike about a new tool?
There’s only one thing you can do, is there. Use the [redacted] tool! You’ll use a tool and if you like it you’ll keep using it. And then at one point you start noticing how you have to try really hard to keep using this tool. That’s the point you discover what you dislike about a tool.
What do you rely on to make a reasonable evaluation of a new tool?
Experience. Others and my own.
How do you know you’ve found “the one”?
If they make your heart beat faster, but in a good way. <3
What is the biggest turn-off when evaluating a new tool?
UI can be a real turn-off sometimes. And some tools just don’t yield results fast enough. Sometimes you need to power through that, but I know I’ll complain about it.
Although tips along the way are welcome. )
I ask for feedback from relevant communities, read documentation, search online…
