Why are test case management tools so complex and/or expensive?

I currently work as a freelance developer with a few smaller startups and companies. When it comes to QA and testing, these companies are usually all over the place. I’ve made sure to boost our test coverage and automation, but I’m typically hired to quickly build some new feature or fix bugs left by other developers no longer at the company. I can’t really go as deep as I’d like, so there’s still a lot of manual and exploratory testing that needs to take place.

Since there’s no dedicated testers on board at these places, they have a very rough test plan, usually written down in a shared document or spreadsheet. Testing is done ad-hoc, with multiple people testing different things without any sense of who’s testing what. It’s just really unorganized and leads to missed coverage that’s noticed only after deploying to production.

These organizations aren’t in a spot to hire dedicated QA (or have a strong desire to hire, to be honest), so I thought a good stop-gap would be to organize their testing efforts a bit more. I’ve done some work for mid-sized companies in the past and many used test case management tools for QA and it helped a lot with organizing their testing efforts. I thought of introducing a similar tool for these smaller places, but I couldn’t really find anything suitable for small organizations.

Looking around, it seems like almost all test case management tools are built for medium and large companies only. They have tons of advanced features mainly focused at places with dedicated QA departments and large testing teams. Obviously, they’re priced for those companies, too, making them cost-prohibitive for a small startup. The open-source tools I found don’t look too user-friendly or intuitive, especially for people at startups with no QA experience.

While admittedly most startups could care less about organizing their software testing activities, I feel like there could be a space for a smaller test case management tool that doesn’t cost so much, doesn’t need extensive training to even get started, or have too many fancy bells and whistles that these small places won’t use any time soon.

Is there a reason for this? Am I thinking incorrectly that small companies can benefit from a tool that handles the basics like creating test plans, test cases, generate test runs, and get reports? Just something to keep people organized that goes beyond a shared spreadsheet, so they can see how testing is helping them out. It feels like a missed opportunity.

3 Likes

It’s probably something that hasn’t been considered as a problem, it ends up working in their adhoc messy way.

This will be a pointless anecdote. In the past there was a a free tool that I forget the name of and I found it to be perfect, it had the right amount of detail for test cases and to get an understanding of what’s going on. It was free though, and they couldn’t afford to host it.

Where I am elsewhere on a big project they use Jira and XRay, but they don’t seem overly happy with it. When I get allocated to something, I just use a SharePoint list - don’t judge me! - it’s in one place, it has tracking and task allocation, it works pretty well for small 3-4 person tasks.

Years ago we used Testuff Test Management | Software Test Management | Testuff as again, it had just the right amount of stuff we needed for a 2 person QA team.

2 Likes

I have used TestLinkOpenSourceTRMS · GitHub , it’s a real beast for what it does, it tracks everything. But I really wish simpler open-source tools did exist, since it’s too much work for a 2-5 person team.

Not big enough to justify getting Xray, especially since with the exception of releases which are done exploratory style, the tagging and so on that the tools require you to keep updating is just not worth it for small orgs. And I’m not talking price alone. My take is that if you have a test “manager” person who spends 50% of their day looking at charts/performance/coverage/planning and so on, sure, get a tool.

Hard agree.

I started out with a special suite of release tests in my management platform, but the size of the regression automation suite kind of made much of it unnecessary. I found I was taking longer messing about in the tool than necessary making sure old tests were removed and new tests actually had the correct environments in them. I often have far too many environment criteria I suspect, something that the basic management tools I’ve used in the past struggle a bit with. If you aim for 90% automation then suddenly your CI/CD is your entire reality (even if that might be a bad thing).

Now I have a release tests document, which is really a list of “non-functional” checks and a smoke test, which must be done for every release. And generally tests things and an environment that you don’t want to automate. I simply create a wiki page per release, with a test session recording of some sort from following that document, and add a link to the automated nightly test result run that was the released build under test. As well as exploratory testing notes. Mark the build job as “keep forever” so you have an artifact, and also note the build number, since as part of the checklist you verify the build numbers in integration match the one in the build job. If the numbers match, hit the “release to prod” button.

I’m not sure I agree with the premise that there’s a market for a smaller featured test case management tool?

I know folks gripe about spreadsheets not scaling, but I think most of the concerns around that have gone away with cloud based documents/collaborative editing, i.e. the major pain point to using spreadsheets for TCM was having multiple copies that were out of sync among a team.

I thought a good stop-gap would be to organize their testing efforts a bit more.

100% agree with this, I just don’t believe tooling is the solution, and spreadsheets are as good as anything else. The focus should be on the individuals and their practices, the tooling is way down the list.

2 Likes

Yeah I’m not sure it’s a problem we have or tool we need outside certain types of large consulting companies.

I’ve been in the industry 20yrs companies from 10 to 10k people. Never used one never needed one.

Previously I’d just have used word, Google docs, a spreadsheet or wikipage.

Now all testing is automated, except the occasion explatory test that produces a mindmap.

Pretty sure they don’t have one a Twitter, google, Microsoft or name a unicorn.

1 Like

One of the other tools I investigated was Airtable | Create apps that perfectly fit your team’s needs. It’s basically a customisation front end for a database, pretty easily it can be used to replicate/replace an offline spreadsheet with allocated tasks, it would be somewhat similar to the SharePoint lists I mentioned before.

It’s also free to a point. 1200 records or 2 gig of data per base, but you can have unlimited bases.

2 Likes