So I was wondering firstly if anyone has had experience setting up a tool from scratch, and if so if they have advice on either the right tool to use or tool selection process. I am not a fan of test cases in general, as I believe they are bureaucratic in nature and not something that actively improves testing. However the reason for adopting one is, well, bureaucratic in nature. We need to demonstrate to external stakeholders that we are testing and thus something with charts and graphs will help ease that process.
To add to this, it needs to pull information on automation runs, exploratory tests, possibly load test runs, and push this to Jira. This seems straightforward but am wary that it might be six months into a year long contract before the reality hits and a tool cannot do X or Y.
e.g. as default: update it (with all testers?) once per day and show it either per day or week to the customer.
In general I suggest you to test and bring concrete problems on the table (while also showing your coverage).
Actual problems on the table show that you test and will quickly draw the attention to the treatment of the problems instead of arbitrary reporting demands.
I had a look at the Low-Tech-Testing-Dashboard and I think it can be useful for me and my team. I’m already in the middle of preparing a little document for it.
I stumbled upon one question, though:
On slide 9 it says: “Areas should have roughly equal value.” - What exactly does this mean?
Is it the estimated amount of work until the area is ready to ship? Or is it the area’s value for the steakholders? Or something else?
At the moment, identifying the areas, or rather being able to draw a limit between them is the hardest part for me. But that’s not a bad thing at all! On the contrary, it shows me, that it may be worth thinking about some aspects of our product that we didn’t so far.
Let me turn it into a question:
What type of areas will help your team and the management to monitor the testing?
In a very general and broad understanding: What areas are important for your customers directly and indirectly? And should therefore be tested.
As you see in the example ‘install’ and ‘compatibility’ are not directly features of the product a end user will directly use but areas that should work and be tested.
Somehow I assume James answer to be similar to this lines. Probably more elaborated phrased and richer on details.
You said the primary constraint is external stakeholders, correct? What does that mean? External to the company or external to the team?
To me this is about understanding who your stakeholders are and what information they are looking to understand about your testing process. Testing is all about delivering timely information to our stakeholders. Automation runs usually don’t contain information useful to anyone outside the engineering team. That’s why they are often run as part of a CI process → when tests identify possible problems its up to us engineers to figure out what went wrong.
Stakeholders, especially external ones, have other ideas in mind about what information they need. Talk to them. Every time I’ve gotten this backwards I end up paying for a tool that doesn’t solve the underlying problem. Then I abandon the tool.
External to the company, basically enterprise clients. The grim reality is that they want evidence that the product they’re buying into has had a record of the testing done, to the extent that their documents include outlines of the expected test cases.
Fortunately they’re looking for evidence of testing, not that we have purchased a tool. So really a googlesheet with an execution summary would probably hold in the short term. Longer term they will want something else (hence looking into test management tools) and prior experience informs me that sometimes the squid strategy (cloud of ink) is often the easiest.
@thewellis
The big thing with a test management solution is finding one to suit your needs.
This past month I have literally been looking to find a ‘new solution’ to take us away from Jira X-ray due to the way the license works (Xray seat per Jira user, not good when you only have 4 QA).
I started by making a list of what we needed, then evaluated the products available. I am also not a fan of test cases - so ours are BDD Acceptance criteria so that the C-Level managers can see what is being covered in terms that they understand (same with external stakeholders.
Reporting is usually by chart and many products give you a URL such that anyone can see the coverage and test run history.
Integration - we integrate with both Postman and Playwright for our API and UI testing so need to be able to report test results directly to the Test management solution. Then of course there is Jira, being able to link test runs to a story or bug, and of course, raising bugs was also a necessity.
Other things like user management, roles, and access were also looked at.
The final thing then is cost. When looking at just over £5,000 a year to renew Jira - I stated I would rather use Excel!! But found a lot of excellent products that did what WE needed. The solution of choice is priced at $99 per month, and I am looking forward to signing the contract and moving forward.
Hope this helps - If you need more info, let me know directly (I am not on commission)
Dennis from the Testmo team here. As we are offering a test management tool ourselves that covers the things you’ve mentioned, and as we give advice to many teams reviewing and using a new tool, I wanted to share some feedback as well. IMO the most important thing when getting started with a new tool is to keep it simple. Start with the default workflow/config as intended by the designers first. Then, when you are familiar with everything, you can start making customizations (e.g. adding/changing fields) to adjust it to your workflow over time.
As an example: I think most people here have seen Jira setups that have been made complicated by the admins by introducing many unnecessary fields & customizations, and it’s easy to get excited about all the options/workflow settings/custom fields etc. But often times less is more, and focusing on tester productivity & keeping the workflow fast and simple is worth a lot.
I would (almost) never provide test cases to an external client, since that’s like providing ingredients rather than the finished product. It might derail how your organization builds and tests software.
It’s also impractical. What if there were 10k unit tests, 1k integration tests and 100 automated UI tests and 6 exploratory sessions? Would you send them x pages of tests? No that would be silly. Plus if you have automated tests in a CI, a single release might have thousands upon thousands of tests run with passes and failures mixed in. Not realistic.
Having said this, I would dive into the requirements, what their asking about and why. Why are they asking for evidence of testing? Probably because they’ve been burned in the past or they have some compliance reason. I would attempt to solve for that reason, rather than the ask. Usually by providing a customized report describing the testing.
I’m in a new role where part of our setting expectations to our clients is that we will test on their behalf. We do some acceptance testing so they don’t have to. We agree on what that acceptance testing will be and then do it at specified intervals. This however is different than testing done for our own internal feedback and improvement.
@m.zn
Hey, tool wise we were using ‘X-Ray’ for JIRA. This became expensive since it is based on the number of Jira users as opposed to the number of QA / Testers.
We then went on to see what was available, but more importantly, what suited our needs and integrations:
JIRA
Playwright
Postman
Gherkin
Reporting
So from a large list, we looked at these:
TestGear
QA Touch
QASE
Testmo
These had the right integration, worked well with what we needed, and were in our price bracket.
Took a few meetings and long trials - but we did decide on ‘TestMo’ and it is working very well for ‘our’ needs.
Remember tho, it has to meet your specific needs and requirements and that of the team!
Hi there, Gil from Testuff (testuff.com) here.
Our test management solution has integrations with practically any tool (bug trackers, automation, project management, etc.) and that includes a two-way integration with Jira with a Jira app we’ve developed to enable getting testing data directly within Jira (so Jira users don’t need a Testuff license). These integrations are just on top of a comprehensive test management solution, over 15 years in the market offered at a very competitive price.
Good luck with your testing projects!
Besides price and integrations, what else about Testmo made it work for you @clittlefair ? I’m wary to get too invested in one tool and later have to change too now. What did you do to port away from X-Ray?
To be honest, the look, feel and ease of use won the day.
To port from Xray was a pain, we ended up having to copy data from one field to another in order to export it. We never did find a way to export the actual XRay fields.
