What is best test case management tool you have used?

Hi all,

I would like to start open discussion on what you think is the best test case management tool you have used and why?

I’ve used Testrail & Tuskr for many project recently but as QA Manager they didn’t meet my requirements as a manager nor project as whole for the following reasons. below & was forced to come up with own solutions exporting raw data into excel:

  • No Testcase numbers or reporting of test results per feature, only per suite (folder of many tests)
  • No reporting on type of testing/coverage conducted/ reporting using any custom fields
  • No Test case stage pipeline or filtering - Draft > sign off > Testable > automated
  • With regression testing (Daily BVT or smoke regression packs), there is alot of repeated tests. Report to tell you how many Test cases executed just once (how many coverage) not including repeats.

Does a testcase management tool exist that functionality to meet these requirements?

Let me know your thoughts on this topic. Thanks,

Chris

5 Likes

In my previous job we used TestLink.

You can manage requirements and test cases, both can be organised in folders or test suites.
TestLink automatically assigns a unique intern number to test cases, which can be used for cross references and for the requirement coverage. It has great options for reporting, but I’m not sure about generating reports based on custom fields.
Test cases can have a status, like draft, or final.
You can have a separate test plan for the regression testing and then create reports from this test plan.

The reports didn’t include the fact, that some test cases were executed more than once. That was great for the overall coverage, but sometimes, we wanted an overview, how many testcase executions were done alltogether for a project, and it was really difficult to get that number.

TestLink is opensource, but it seems that it’s no longer developed and maintained. :frowning:

We’ve been looking for alternatives, and among others we came across Klaros and TestMonitor, which seemed quite useful. But except from a few experiments with their free trials, I don’t have any practical experiences.

2 Likes

We are about to, once the system team can get it installed, trail a jira plugin by a popular vendor. And this question is worrying me a lot too @captainjonesy2 . I suspect these tools suffer from the same problem all software suffers, it solves only one perspective of the problem. Often making managers happy becomes the goal, and things like trending the pass/fail rate of a single “feature” becomes impossible unless you organize your test suites accordingly. Basically you have to have good control over how features map or how functionality/architecture maps into your test suites, you can never have both, one or the other is what I have found. I personally prefer a component mapping, but a “feature” mapping is appealing to me more and more, since that makes deprecating a feature easier to just kill off later. But does mean I will need to plan a lot better.

Being able to run one test many times in different environments is a test report vector I am keen to get working to help me find those flakey environments. So that’s a requirement for me too @monija . I was also disappointed that Testlink eventually went out of support.

I’m dreading the trial start day, because I’ll need to temper my expectations, the demos they showed us look great, but until you try things yourself, it’s really hard to know if it will help you as much as you want. Perhaps the solution is to have lower expectations. My end goal is to be able to find flakey tests quickly and easily by being able to run historical queries. So sometimes just having a decent graphing front-end is what some of us really want.

1 Like

@conrad.connected - Yes you’re right, you can’t have everything. Most TC tools I’ve seen are built for audience of the testers for TC creation & execution, not for QA Managers. I’ve surprised that no one out has designed for tool for both type of users/use cases for execution & reporting.

@monija - Thanks for suggestions, very helpful. I will certainly check out TestLink, Klaros & TestMonitor.

Another requirement I have which has just occurred to me; the ability to design tests without creating dupes. My projects I’ve been on, I’ve had to create a process to ensure testers are not creating dupes leading to a waste of effect. Is a tool with a feature that would help with this?

I’m currently at looking at Xray & Zephyr which seem so far meet my earlier requirements.

@conrad.connected @monija - How do you want to decision on the right TC tool for your project? Do you have requirement checklist and see which tools meets the most of your requirements?

2 Likes

I’ve been prompted to write up my requirements on our internal wiki, I liked some things that Testlink did give us, but overall it was a very “expensive” maintenance-wise tool and usability was just terrible. Will update when I have a moment on what requirements goals I came up with. Busy pushing a release to the apple store today, so a tad busy.

3 Likes

In my old job, TestLink was already there, when I joined them. So I wasn’t participating in the decision process.

In my new job, there is no tool at all. Requirements are not explicitely documented, which means I search through dozens of tickets to find the customer’s extension requests, bugs and issues. Like a hunter and gatherer. :bow_and_arrow: :chipmunk:
But I’d like to have a place where I can store and organise my harvest and haul and I need to make sure that I don’t miss anything.

So I made a list with must haves and nice to haves based on the things I knew. But in the meantime i realised, that my list was too much oriented on the workflows of my previous job. Of course in my new team everything is handled differently, so I gave myself some more time to learn and find out what will be needed here.

This was my original minimum must haves list:

  • Manage product requirements
  • Manage test cases (with steps)
  • Assign test cases to requirements
  • Organise test cases in test suites
  • Document test executions
  • Have a field for an issue-tracker-ID

Maybe later there would be more features needed:

  • Assign test cases to test environments
  • Assign test cases (or even test case versions) to product versions
  • Have a list of automated test cases and somehow assign them to the product requirements to make the coverage visible.

My current view on this is: maybe I don’t even need very elaborated test cases with all those test steps and expectations, but focus on a proper documentation of requirements in the first place. :woman_shrugging:

2 Likes

Feel free to give our test management tool Testmo a try. We have built-in support for things like workflow states (e.g. Draft → Active → Retired etc.), automation linking (and filtering), coverage status columns (so you can see the latest status of all your test cases across all runs) and similar that might fit what you are looking for.

Hey Chris.
If looking at Xray & Zephyr, I hope they meet your needs. The one reason we are moving from Xray is down to their licensing (Zephyr is the same). They license on the number of Atlassian / Jira users as opposed to the number of QA / Test users. We have a team of 4 QA, yet over 300 Jira users - so the price for Xray is extortionate! If they could address this, we would ‘stick with it’ and its peculiarities.

We were looking at “Test-Gear.io” as seen at TestBash recently - but they are no longer trading!

So, “Testmo” & “Qase” are currently being considered for suitability.

2 Likes

We are moving away from TestRail ourselves, but I can add two tools that we have been looking at that have not been mentioned. One was PractiTest, which seemed quite robust to me, and the other was Kualitee, which is quite affordable. Unfortunately, I can’t tell you if they meet all your requirements because I don’t know them that well.

2 Likes