What API automation testing tool you use and Why?

(Musaffir) #1


I would like to hear from all of you about the API automation tool that you have expereinced and would love to hear why you have choosen it ?
Do you happen to expeirence automated API tests with in CI/CD pipeline of your software delivery process ?
What is your test workflow ? Please share all your API test autoamtion expereince

~ Cheers

30 Days of Automation Testing Day 9 - Find, use and share your thoughts on an API testing tool
(Hilary) #2

For API automation, I use C# for the projects I’m working on. It’s the language of the application I’m testing, and my tests live with the rest of the code. I chose it for that reason, as well as the fact that the devs don’t have to get out of their usual work environment to run my tests. It also encourages them to write API automation, which they have :smiley:
For CI/CD we use TFS/VSTS for our .NET framework projects. The process is basically: build code; run unit tests; if all pass, push to TEST env; run automated tests. Because other teams use our non-prod environments, I also do manual testing after things are pushed, then manually approve it to start the process again in BETA.

there’s a lot of great info here if you’re looking for more :smiley: : http://katrinatester.blogspot.com/2015/09/api-web-services-microservices-testing.html

(Josue) #3

I used REST Assured because it can be used with Java and run with Maven using unit tests, so it integrates nicely with any CI/CD pipeline.

I’ve been using it in many projects and it is so easy to setup and write tests on it. My reason to choose it is mainly because I’ve got previous experience with Java and Maven that for me that was the obvious tool to use when needed to introduce API testing. Another advantage of it is that it uses the structure given/when/then to write the test so it is very intuitive even for people with non advanced coding skills.

(Musaffir) #4

Thank you so much @g33klady for your valuable time and putting details here and I must say its a wonderful process :+1:

(Musaffir) #5

Thank you @stardust for sharing your experience as well here :star_struck:

(Musaffir) #6

I have some experience with rest-assured and I too found it easy with its fluent style apis and given/when/then structure.

But I guess you need to spend some time initially to set up the automation test framework using res-assured, and its full of coding (in java) … it could also be taken as an advantage for you as it has all the feasibility of taking ALL the benefits that Java gives you.
I suggested this at my company, but the management was not so impressed especially when they understood that it requires coding to develop tests ( though I personally likes it )

Currently I am looking at a tool which can speed up the process and we can develop tests more easily and start integrating it with our CI/CD pipeline

I have experience in SoapUI too which we didn’t take forward because it need to be installed/present in the machine where the test is running. So that would be an expensive one for us especially if we need to configure the all the machines where build is happening, if we go with pro verson… and the free version has its own issues

Recently I have been playing a lot with Postman.

I liked postman for its simplicity, ease of test development, ability to incorporate Javascripting etc and especially the power of automating the tests with Newman command line tool. So you can easily make the tests as part of your CI/CD pipeline.

One slight inconvenience I saw with Postman is , it is not so easy to integrate with version control system.
It has a real time sync option with Github, but not with Atlassians Bitbucket ( the one we use at my company :sweat_smile: ). So you would need to manually export the test files from Postman GUI and version control it . That would slow down the process and may bring some issues at later point of time especially when Team with multiple people start developing the tests.
So my journey continues in fixing a proper tool for achieving api test automation :grinning:


(Chris) #7

In terms of test applications, I use:-

  • SoapUI - The free version is good, but the licensed version is much better.
  • JMeter - Not strictly for API functional tests, but for Load and Performance
  • Postman - Used this to a lesser extent. But if you are using Postman, you really do need to take a look at Danny Dainton’s excellent Postman series on GitHub.

(Musaffir) #8

Thanks @chris.adams for your response. Very kind :heart_eyes:

(Philipp) #9

Hay @musaffir, what is your future test strategy / process in your company? Do you want to outsource the tests to the developers, so that is a collaborative Tool needed?


(Musaffir) #10

Hello @sobsi06

We have no intention to outsource the tests to developers.
(at least we want the test to be part of the application code base itself, so during development work if dev wanted to run tests (say regression tests) against their changes, that should be easy…)

and we want the all these things to happen in single pipeline as part of the build system ( we use Atlassians Bamboo)
So pipeline will be like
Dev committing code > Build system pull these changes > Building it > Running Unit tests > Deploying to test server > Running automated integration api tests …

So its important for my tests to be version controlled and maintained in a version control system ( we use Atlassians Bitbucket),

There I see inconvenience if I go with Postman for developing tests especially i our eco system.
I could have used Res-assured and resolved it, but for some other reason( lots of coding) we didnt take that as well.
Please let me know your thoughts


(Dan Caseley) #11

Hi Musaffir,

I blogged about how we chose an API tool on my blog, here: http://testernaut.blogspot.com/2018/05/choosing-api-tool.html - this speaks to how we chose, languages etc.

Shortly after this, we found that frisby.js had some flaws and we switched to IcedFrisby.
There was a feature we needed for a particularly rough API that required cookie authentication (ew), which required some changes to IcedFrisby to support. We could have implemented the bits we needed via some hacks our side (probably), but contributing to the repo seemed like the best way, and one of the devs was encouraging me to get involved in Hacktoberfest. So I did, and I improved some docs and other things whilst I was there (and earned my Hacktoberfest t-shirt!)

The company now has a very decent collection of API tests, and I’m a maintainer on the repo.

On your question about CI: we used a CI system to monitor our live environment using a subset of the tests (because regularly testing login and other business logic operations is way more effective and informative than pinging). We wanted to get to what I think you’re talking about though - builds being automatically deployed and tested, but we got scuppered on the “deploy” part - it was non-trivial, and wasn’t worth the investment whilst the company was on a path towards .NET Core and Docker.

Hope this helps!


(Musaffir) #12

Hello @dancaseley
Thanks you so much for replying and sharing your blog post, It was a wonderful read to me , am sure I will re-read it again :grinning:

I didn’t know about frisby.js, I will explore the same and try to learn it . I will get back here with my findings and comments

Warm Regards

(Musaffir) #13

Hello @dancaseley
I found frisby.js framework as really cool, but I guess it need coding efforts to develop the tests . And as this based on Node.js and java scripting, the team should have competencies on those technologies. I have been playing with Node.js, NPM etc earlier, so I feel a bit ok with this. But my team is not technically aware or used to javascripting etc. So I guess it will take a lot of time for Team to get used to it and start developing the tests .
This was the same reason we decided not to go ahead with JAVA based Rest-Assured framework.

we are looking for some tool which can help us in speeding up the test development activities. Like Postman but it has its own few inconveneince as mentioned in other replies.
I will see what tool best suites for us and find a balance

Thanks for your valuable time

Warm Regards

(Vishal Dutt) #14

A type of software testing which includes application programming interfaces (APIs) testing as a part of direst as well as integration testing in order to determine if that meet expectations for reliability, performance, functionality and security is known as API testing. API testing enable communication and data exchange among two software systems those are separate from each other. A software system putting an API contains of several functions/sub-routines those can be executed by another software system. API testing is nowhere similar to GUI testing as it works on business logic layer of the software architecture of an application.

There are several tools for API Testing like SoapUI, Runscope, Curl, Cfix, Postman with jetpacks, dotTEST and Eclipse SDK tool - Automated API testing etc.

Out of all these tools SoapUI tool is the best preferred tool considering its following listed features:

  1. SoapUI has been built specifically for API testing.

  2. It supports all the standard technologies and protocols to test all kind of API’s.

  3. Quick and Easy Test Creation - Drag and drop.

  4. Helps in performing Data driven testing.

  5. Provides Script Reusability.

  6. Provides multi environment switching.

  7. Integrates with several other tools like Maven, Hudson, JUnit etc.

  8. It also provide Vulnerability Testing.

  9. Can create a custom workflow /functionality using Groovy.

  10. It can also estimate a web application’s load balancing capacity.

  11. Security, Functional and load testing.

  12. It is protocol dependent instead of being Browser dependent.

Note: Some of the above listed features are only available in the SoapUI NGPro.

Hope this information is helpful for you.

(Tuan Tran) #15

I use the Postman for API automation testing. And I found that we can run all Postman collections in CI/CD by using the lib Newman - the cli for running postman on NodeJS env.
Hope this help.

(Natasha) #16

We have just started API-level testing using Curl and Postman/Postman Inspector/Newman, we decided initially on these tools as most of our developers have experience using Postman and it integrates quite easy into our CI process.
However, we are currently evaluating other tools to see if they’re a better fit.

We’re still currently finalising our workflow, but it goes along the lines of:
Build env -> run unit tests -> if passed, run test automated tests -> if passed, push to master branch -> release

(Felipe) #17

I’ve been using Postman and was able to put together a useful regression suite for a specific project at my company after playing with it for a while. Danny Dainton’s Github series, including his template for newman reports are life-savers!

I’ve also been trying to explore more flexible solutions, particularly javascript frameworks since that’s what our devs use the most. I’ve tried supertest but never got around to deep-diving into it. @dancaseley, IcedFrisby looks very neat and I will look more into it. Thanks for bringing it up

(Thao) #18

Did you heard about Katalon Studio? If not, I can share somthing about this cool tool which I very love.

Katalon Studio listed as a Top Automation Tool by Microsoft https://www.katalon.com/resources-center/blog/top-automation-tool-list-microsoft/.

Katalon Studio is an automation solution for API, Web, and mobile testing. The application integrates all necessary components with built-in keywords and project templates into a complete automation framework. The tool was developed to support various testing technologies (Data-Driven Testing, TDD/BDD, Page Object Model, etc.) and integrate with the modern ALM systems (JIRA, qTest, Slack, CI, Katalon Analytics, etc.).

Expecially, it is a free tool. And the Katalon support team is very helpfull too. So many companies have started working on it. I used it to get started with API testing too.

Hope this helps.

30 Days of API Testing Day 8: Explore the API thread on The Club and contribute to the conversation