Test management tool - what's crucial for you to love it?

Iā€™m running a very simple research - trying to understand what QA testers and engineers really care about when they use test management tools - whatā€™s really important, whatā€™s not so much, and what will become important in a few years.

The context: Iā€™m in the product team in one of the ALM tools and we want to understand how to make the tool loved by end users, not just companies. Thereā€™s no agenda here, all I want is to learn.

If you have some spare time and youā€™d like to help - please let me know in the comment or DM!

Thank you!

5 Likes

I mostly try to avoid test management tools as their approach is very restrictive to my work.
Especial this relying on a fixed structure like steps with action & result.

This type of tools I see only usable in cases of often repeated executions of the same actions. Which is to me a very limited perspective what testing can cover.
I currently work with a combination of Confluence and Jira (without a plugin) where Iā€™m very unrestricted. And with that I also can cover the often repeated executions.
My work flow changes often (in detail), sometimes individual per ticket.

Concrete issues I had with tmtā€™s:

  • I use such a list of step results as total summary. I seldom start a new run from step 1, but find bugs , workaround them and continue.
    • at some tools , e.g. Testlink, changing to another test case terminated the current run and I had to start a new run - with all result fields empty, no summary available.
  • Often I have something to test and/or report where a different formatting than just text would be helpful. Most prominent tables.

Even by their name (and its implicit meaning) I do not like test management tools.
I document and share my notes of my findings. And do this in collaboration.
This typical management, mostly relying on execution numbers, gives a false impression of safety.

2 Likes

Not every project does well in a test management tool - but yeah, when you have ALM, you can be sure EVERYONE will have a poke in there.

I used Quality Centre back in 2010, and TBH it feels like all management tools are trying to build a similar experience. In the same way Netflix vs Disney+ vs AmazonTV all feel the same and have similar features.

We now work with TestRail, which is okay.

Thank you for all the insights, appreciate it a lot!

1 Like

Thanks, Mike! What are top 2 things that are missing/should be changed in TestRail from your perspective?

Ah - itā€™s a bit clunky in places. Moving tests in/out of a suite is just plain nasty.

And as with most test tools, it sits ā€˜on itā€™s own as a tool for testersā€™, so it doesnā€™t feel as integrated into the big picture of whatā€™s going on.

I prefer adding notes into Jira on how a story was tested (for instance) over just a set of tests existing in TR in isoltion.

A big thing for TOOLS FOR THE WHOLE TEAM, not just testers.

5 Likes

Iā€™ve asked my test lead to look at test case tools again - we have a funny Jira setup which has made dome-ing or buying a plugin a bit heavyweight and costly until now. But probably next year we will be in the market, after using homegrown and tools like testlink and then looking at demo videos of some Jira plugins, Iā€™m thinking with Mike here, all far too heavyweight.

Testlink and our homegrown tools used in previous jobs are too heavyweight, they have all the nice api entry point things where people can push data into the tool, but at the end of the day, those are tools that developers will never use, because the apiā€™s are often too complicated for a 5 minute job. In their UIā€™s they have too many button pushes like creating a test cycle, then choosing tests, then assigning a tester, and so on for infrequent users to use. Sometimes you just want a test cycle that anyone can pick up and work on, go on holiday and then be completed by another person. ALM tools are a Process overhead pain. Too many ALM tools erode trust by being complicated.

Almost yearly now, we run a test cycle where we get everyone in engineering to run product tests, and if you give people a thing that requires navigating about in menus, and learning a new tool, you create work for yourself, because they just wonā€™t. So for them I just paste all the tests into a Excel sheet on 0365 with a column where you can type your name and add a column for any notes, and another for a pass/fail verdict. Beautiful part is people in multiple teams can work down the list grab and run one, and just add their name. Then they can add colors to tests when they fail, and even make a test case gray to indicate itā€™s not worth running. I have to date never seen a Jira plugin that lets anyone in the company see the test steps, and even update the test steps in realtime without requiring training to sue the tool first. Downside is you do have to create a new spreadsheet on every iteration, but since we only do it once a year, itā€™s not that painful. We just export the test DB into CSV ,and off we go.

Lightweight, suitable for ad-hoc testing cycles as well as for automated testing nightlies/weeklies in one tool.

3 Likes

One of my teams simply creates tasks against the stories on the sprint board in Azure Dev Ops which isnā€™t ideal and requires additional regression scripts to be maintained. The other uses zephyr squad within Jira and I find this works much better. Itā€™s taking a while to get the team used to writing full test cases (the original lead on that team had initially only requested that the ACs for each ticket be recorded in zephyr squad which led to gaps in test coverage). The team now use ACs as the final sign off and plan testing around a user journey with pauses to investigate edge cases. This gives a re-usable set of tests and also clarity of what is worthwhile to automate (and I can quickly find the stats for the tests run, defects found etc).

3 Likes

@chrissi Having those AC (Acceptance Criteria) roughly in one place is also such a big help at building out an oracle that describes what we want a system to do. Thatā€™s a great use of Jira (other workflow tools exist) tickets. Having this original idea to what the requirement looked like, right there in your testing activity, is something that every TCMS (Test Case Management system) system Iā€™ve used to date has failed to bring me joy with.

1 Like

Mike, thanks so much for the details, itā€™s already so insightful!!

If I may ask, ā€œdoesnā€™t feel as integratedā€ is really about how easy/seamless you can move across different pieces of your workflow that live in different places (Jira-TR), did I get it correctly?

1 Like

Conrad, itā€™s pure gold, thank you!!

Iā€™m probably not getting something here: I understood the UI/too much clicking part, but youā€™ve also mentioned that these tools are not gonna be used by developers, and then mentioned the API complexity (for the 5 min job) - can you please explain what you meant with regards to APIs specifically?

1 Like

Christina, thank you! It was really interesting to learn separate teams are free to adopt their own tools!

1 Like