Comparison between Azure DevOps and Testrail

What are the major differences between Testrail and Azure DevOps for test case management?

So far, the only function that is not in azure that is in testrail appears to be the import function, that would be critical for us. However, it appears Azure’s shared test steps are a function that is not in testrail.

They appear to be pretty comparable in terms of reporting.

What’s been your experience?

I’d be curious about this myself - I just started on with a team with practically no QA process so am working on setting up all the things. They are heavily invested in Azure DevOps and would prefer to use those products but are ultimately willing to consider what I think is best. I’ve not used it in the past (Azure DevOps) and am not finding a lot about what people like or don’t about it. Any input to share thus far?

I can’t say anything about TestRail, but Azure DevOps = Team Foundation Server Online with extra goodies.

Probably the biggest benefit to Azure DevOps is the integration with All The Things. As long as it’s set up correctly, you can connect your test automation to test cases, your test cases to work items (regardless of the schema you’re using for the project - that affects whether it’s defects, user stories, bugs, or something else), and hook it all to an Azure Devops build/release pipeline.

Manual test cases can be as detailed or broad as you please - I tend to keep mine broad and use the test steps to cover different settings that a test case needs to cover. You can do anything from stepping through your test case using the web tool to just marking it pass/failed when you’re done testing.

There’s also a ton of plug-ins/extensions, many of them free, that you can use - including a plug-in for Firefox and Chrome that allows you to record exploratory sessions and log them to Azure DevOps, associated with the user story/bug/whatever that you need them linked to.

Honestly, there’s so much there it can be overwhelming at first - it took me months to get my first automated test pipeline working. Now I can set one up in a day or so and get emailed the results.

2 Likes

HA, re: overwhelming… YES. I’m SO there. I’ve not worked with it before and am definitely trying to sort out what the workflow should look like. Questions for you since it sounds like you’ve worked with it a fair amount… do you know if there’s a way to automatically add a group of test cases (a test suite) into the next iteration? For example, I have a set of tests I’d like run each sprint. If so, how can I accomplish this? Thanks!

Azure DevOps provides the services such as share code, track work, and ship software for teams.
This provides unlimited private Git hosting, cloud build for continuous integration, agile planning, and release management for continuous delivery to the cloud and on-premises. Broad IDE support also included.

On the other hand, Testrail is like efficiently manage, track and organize your Devops testing efforts. This helps you manage and track your software devops testing efforts and organize your QA department.
Its easy to create test cases, manage test runs and coordinate your entire testing process just because of its intuitive web-based user interface.

Azure DevOps belongs to “Project Management” category of the tech stack, while Testrail can be primarily categorized under “Test Management”.

Following are some of the features offered by Azure DevOps:

  1. Agile Tools: kanban boards, backlogs, scrum boards
  2. Reporting: dashboards, widgets, Power BI
  3. Git: free private repositories, pull requests

Whereas, Testrail provides the following key features:

  1. Efficiently manage test cases, plans and runs
  2. Boost testing productivity significantly
  3. Get real-time insights into your testing progress
1 Like