We have been using Kualitee for our test & bug management needs for quite some time now. The tool supports both manual and Automation testing. It has integration with Selenium for test automation and Jenkins & Bitbucket pipelines for release management.
The tool is also free for a small team of 3 people or add up to 10 user for $45. You can create tester, developer and QA manager profiles as per your need.
The tool itself is a simpler version of HP ALM. You can create, manage and execute your test cases and optionally add any bug with it. The tool also has it’s own defect management module where you can log and track issues.
The tool also integrates with JIRA . Lets you sync all issue types and you can also map all your JIRA custom fields with this tool. It’s pretty hectic by luckily a one time effort though.
For me the dashboard is the best feature because it provides insights on your test cases, executions, trend analysis, and bug severity and status.
We also do our reporting through the tool as we can customize our test and bug reports in-app.
We used Testlink at my last company integrated with our bug tracking tool (Trac).
After the pain of moving projects from excel to Testlink the major pluses were:
Requirements capture
Stats for tests
Having change control on test cases
Being able to quickly produce new test plans for similar products
Being able to produce formal test reports fit for customer.
Pros:
Open source so easily modified
Cons:
Open source so don’t expect much from raising bug / enhancement requests
I just started using Testlink, being modifiable has never been a selling point, as a SDET, I have not really got time to spend reading someone else’s code when it’s not related to my day-job, but when it comes to being open, Excel is far more open in my opinion, Wayne.
I have always used in-house built management tools, because the source of truth today is my test scripts themselves that run once a day and once a week. An off-the-shelf package has never integrated with the real world well enough in any place I have worked to date. But, honestly, don’t use Excel.
Fortunately for us we had a dedicated test team so it was a project in its own right. It took at least about a month to integrate it with Trac (also modified) to our requirements so I’d not call this a “quick win”.
I just want a management tool, Testlink is free, but it’s looking like abandonware after 4 weeks of me trying to understand it’s “schema” or organization. Drawing Testlink reports is just bizarre or requires writing code just to do meaningfully. I’ve not got time to search for another tool, and am drifting back to Excel. We have 3+ testers, but developers and managers do testing too (everyone tests in concept), so a free tier just does not cut it, and an execution engine built into the tool is IMHO overkill and a distraction. Anyone else enjoying using Testlink http://forum.testlink.org/viewforum.php?f=11 who can give guidance, or offer low barrier alternative tools? Wary of SAAS or web hosted offerings that only want to sell you other features you don’t scale into.
Jira integration is nice, Automation integration is nice, but an import and export button with a web API behind them to to drive them fits the bill. For small companies like where I am now time and skills to be setting up a server is wasted.
Variety of test styles accommodated (not just stepped tests, like so many)
Applicability to our internal process
We shouldn’t have to change process to handle the tool (although the right tool can help us develop our processes, for example the positive feedback styles handled by TestBuddy). As such we’re currently looking to move from Zephyr to Xray, which serves points 2 and 3 better.
In what ways did Zephyr not met your team requirements Stu? I am keen to just go with the 10-users tier and link Zephyr into our Jira somehow in a way that fits our process. Trials are so expensive when they require infrastructure and integration - even the time to just import all of your data. How can a tester trial all the tools out there and still do your day job? How do you find out about those hidden gem tools that almost nobody knows about that do a pretty good job, cheaply. I found another one that sounds good, I think was featured in a podcast last year https://testcaselab.com/ .
Zephyr has seriously underwhelmed me over the last 4 years of use and this was as much a switch away from Zephyr as a switch towards something else. Zephyr have forced a very buggy “new” UI on users, only providing an opt-out weeks later, which meant we had to switch to a temporary solution (excel) mid-sprint for a period of weeks, about a year ago. This dazzling, unadvertised new UI failed to load all over the place, then did things like failing to save data, failed to record executions etc. It was a bloody nightmare for enterprise-level software, in use by a team building and testing enterprise-level software! Since then we’ve switched to the essentially deprecated and unsupported “legacy” UI (which still mostly works), although every time they push an upgrade it switches back to the new one, meaning more manual work for me to get to the functional one… our last year with Zephyr has been like pulling teeth!
As to the reasons for Xray, Zephyr (in this case Zephyr for Jira) is firmly rooted in the “stepped tests” mode of testing, whereas Xray (Xray Cloud) supports gherkin and exploratory styles natively. Zephyr is relatively expensive given we pay by Jira instance, for a company of 750+ Jira users… Xray bills the same but works out about half the price. Xray also stores all of it’s “nodes” as Jira issue types, meaning bulk operations, cloning etc work properly. Zephyr has a number of proprietary structures, most annoyingly test cycles. I have no way of cloning my regression pack without undertaking a day’s worth of donkey work - in Xray I can do this in 10 minutes.
May be too much detail… but I can’t warn you off Zephyr (for Jira) enough! But if you do go with it I hope you find it more suitable. We found Xray in a review, checked the Jira Marketplace reviews (top-rated TM tool) and have trialled it at https://sandbox.xpand-it.com/
This is very helpful , the cost of evaluating a tool is really high if you have not got 1 whole week to spend learning how to deploy a tool to start with. And that setup process can throw you off some of the more powerfull tools early on.
Ah my mistake, it was top when we began our plan to move though (about a year ago now!). Very surprised to see Zephyr rated so highly, perhaps it’s just us - although have spoken to testers in other orgs experiencing similar.
I’ve yet to see a test management tool that works for me.
What I use is whatever the place I’m working for uses. The previous job had excel spreadsheets, then moved to TestLink. Neither fitted all that well, and nobody had adjusted to TestLink when I moved on. My current place I’m using TFS/VSTS/whatever for anything large, and the tasking in User stories/defects in Rally/Agile Central for smaller pieces.
My test cases are generally more along the line of a broad outline than a detailed test plan, and I haven’t got close enough to even think of using the ability to link a test case to a code routine.
In my opinion you use what you’ve got. If you’re picking something up, aim for something as close to your internal processes as you can get without blowing your budget to pieces, then make whatever adjustments you need to. The only time I’ve ever found the need to write detailed test cases was when someone who knew nothing about the software was going to be testing and I didn’t have the ability/time to mentor them.
Not found a great off-the-shelf tool myself yet either
I hate having huge dis-organised lists too. Have found infinitely more value in exploratory testing using the specs as a map, but when many teams have their own tester, having a common test spec language and organizing technique helps remove duplication.
I think this is the place anyone in a company that has staff churn, or has just joined, or is even in a start-up will find themselves in. Like when I went through manual release testing last week, I had completely forgotten to re-test the Raspberry Pi !! So having some kind of structure to the specific checks that really do have to be done, on all platforms - without too much duplication across teams happening does require a shared management tool of some kind.
Hello, Stu. My name is Laurent, I’m the VP of product management here at SmartBear and I’m sorry to hear that you and your teams had a negative experience with Zephyr for Jira.
When we acquired Zephyr in 2018, the company was working through a number of the issues you described, and since that time, we’ve made significant investments into strengthening integrations, usability, and adding new functionality that didn’t exist one year ago. We have introduced a new way to navigate in the application, increased performance, and released [Cucumber for Jira] that extends Zephyr with unique BDD capabilities like native Git integration and Living documentation.
I wish we could’ve helped you sooner, and if you would be interested in sharing any additional feedback with myself or our product team, I’d love to set up a conversation where we can learn more about what most impacted your experience.
Hi Lauren.
I’m trying to stop using Testlink, since it’s just not powerfull enough without spending time coding, but Jira bug tracking plugins are expensive when you are already paying for Jira seats across a wide company who use it for things like marketing workflows etc. and where software engineering < half the seats. I can pay for my whole team pub lunches every Friday with the cost of most plugins when we look around. My other cost is that I would need to carve out a lot of time to import my data into Smartbear, and then run a whole release iteration - which will cost me 2 man-weeks, so there is significant commitment cost during the trial as well. Just to install the plugin and read the starting up guides I assume will take a day. So gentle start is a key factor for me. Still open to Smartbear if I can hear a testimonial from a happy tester in a <50 engineer company.
Personally I don’t care about integrations to BDD or TDD or whatever new fad is sweeping the industry. I want a test management tool that lets me mine my test specifications for relationships between, code (via well classified bugs and git commits) and a REAL USER workflow-based view of the world. Users don’t care about app architectures, and too many tools focus on testing the code behavior, instead of the how real customer use cases change over time. Too many models model the functionality, when we know that 80% of customers use 20% of your code, and that low attention span on some platforms mean traditional views of testing everything fail to quantify new risks.
Testlink does not let me easily export then just certain (filter) data, and then create mind-map kind of models and tag my specs. I want to group tests at will (let me add structured metadata) so that I can view for example all tests that contain the word “security”, or for all tests that happen to contain the word “mouse”, or “click”. Basically a tool that does one thing well, hold all of my data.
My pain with nice tool integrations, is that most of my career I have tested apps that require special frameworks, so although we use Appium here, that’s less than half of our test surface and a tenth of the risk area. Every job as a tester I’ve ever had did not favor off-the-shelf tooling, hence I’m looking at tools designed to be open. I spend enough time writing test framework code already to control environments which few of the frameworks do, well.
Sure. I ll be happy to make an introduction to one of customer in that specific segment.
Based on the description of your need and workflow I think Zephyr will be a good fit. The way to organise tests with metadata is similar to what you can achieve with Jira as we leverage the issue type test for all the testing activity.
Zephyr also provide comprehensive APIs for integration with other tools and frameworks. So far we have been able to fulfil the needs of large organizations like Amazon or Citi that have pretty complex stacks and integrations. I would need to know more about your integration needs to make sure there is a fit. So here is my suggestion:
1- I make the introduction to one of our customers having 50 or less engineers. Please could share your contact info at laurent.py@smartbear.com
2- I arrange a discussion with our product team to answer any questions you might have about the product and the integrations/APIs
3- you can start a trial if you have positive answers to point 1 & 2 so you don’t waste your time.
Hi Conrad,
let me jump in you’re right; open tools, that support a bunch of integrations out-of-the-box and that can easily be adapted to your team needs are one of the key things to have in mind.
So, for example, with Xray you have support for Gherkin (Cucumber, SpecFlow) for several years; no added cost. But, and as you’ve said, teams may use tons of different automation frameworks.
That’s why Xray supports of bunch of automation frameworks out-of-the-box (JUnit, TestNG, Nunit, Cucumber, etc, etc): https://confluence.xpand-it.com/display/XRAYCLOUD/Integrating+with+Testing+Frameworks
Under the following page you may find concrete examples/tutorials, around 50 of them, for all sorts of frameworks, for web or mobile (android, iOS) or other: https://confluence.xpand-it.com/display/XRAYCLOUD/TTT%3A+Automation
You mentioned about exporting data; well, in Xray you can easily search Tests by whatever criteria since they’re issues in Jira… but you also can search & export your Test Plans and your Test Executions (i.e. scheduled tasks for running a bunch of tests) and easily add labels and fields on them since they’re Jira issues, in opposition to what it happens with Zephyr. And you can export them to fully customizable PDF, Word or Excel documents (using Xporter app).
Concerning integrations, you also have a bunch of them… for free, and more to come soon: https://confluence.xpand-it.com/pages/viewpage.action?pageId=31622291
As you can see, it offers REST API but also the more flexible GraphQL.
Note: the previous links are related with Xray on Jira cloud; for on-premises, they’re slightly different.
Also looking forward to collect some feedback (notice: I work at Xray). @harshp any initial feedback you may want to share from your experience, if that’s the case?
