Ask Me Anything: API Testing

Tonight we were joined by the fabulous @g33klady for an Ask Me Anything all about API Testing.

Iā€™ll add all the resources mentioned in the webinar to this thread along with any questions we donā€™t get to. If I miss any resources or you thought of a question youā€™d like to add, why not add it to this thread?

If you missed the live session, a recording will be available on the Ministry of Testing website for all Club level members once weā€™ve edited it and added captions.

4 Likes

Resources mentioned in the AMA

If you want to join or create API testing discussions here on The Club, head to the API testing category

Hilaryā€™s TestBash talk that Vernon mentioned
https://www.ministryoftesting.com/dojo/lessons/spying-on-your-application-how-to-be-a-super-sleuth-tester-hilary-weaver-robb?s_id=202407

Pop a unicode snowman into your API and see what happens http://unicodesnowmanforyou.com/

https://www.postman.com/

How to choose API testing tools Search results for 'api testing tools' - The Club

Restful booker from @mwinteringham https://restful-booker.herokuapp.com/

some exercises for getting started with pytest and the requests library in Python: GitHub - j19sch/building-an-api-testing-framework: Learn to build an API testing framework in Python

Danny Daintons Postman guide GitHub - DannyDainton/All-Things-Postman: A selection of examples using Postman REST Client

@ezagroba shared in case you havenā€™t memorized http status codes https://httpstatuses.com/

Questions we didnā€™t get to

  1. Is it wrong to make my suiteā€™s tests dependent on each other?
  2. what is the best way to structure api endpoint tests with ~300 endpoints in your repository? many of the calls are divided in get, post, patch, delete calls and there are also different endpoints for lists and single objects? Each should be tested against JSON Schema and data param-values for ex. thank you
  3. The automation pyramid is made up of the unit layer, api layer and the UI layer. I know the difference between each of these layers - how do you decide what tests scenarios should be covered in the GUI layer and what should be covered in the API layer?
  4. is soap ui used even now or been replaced by postman
  5. Any specific tool to check which APIs are used in app. Also, how to know which APIs are triggered upon app activity (eg: App Launch) and the time taken. I use Charles Proxy.
  6. how different is swagger tool, is it more popular than postman, difference between the two
  7. Have you used any tools that automatically create tests from a Swagger json? How useful have you found them?
  8. How do API tests of different types usually sit within the ā€œsoftware testing pyramidā€ for you? In particular, testing APIs/services that should integrate to give particular behaviour vs. testing an API/service in isolation.
  9. How to effectively test multiple interactions between different APIs? Which level of test automation does this fit in?
  10. What is the value of doing contract testing (with a tool like PACT) vs. functional testing of API responses?
  11. Hello, I am looking for an API testing framework with the following specifications: - tests are written in some scripting language using an IDE - the same functional tests can be used also for performance testing - extensible (ability to mock other interacting components) I am considering: https://github.com/intuit/karate/tree/master/karate-gatling https://gatling.io/ https://locust.io/ GitHub - naver/ngrinder: enterprise level performance testing solution Any suggestions?
    11.1 John Smith said: we use Karate for GraphQL it is awesome: https://www.codemotion.com/magazine/dev-hub/web-developer/graphql-testing-with-karate/
  12. What are your thoughts on codeless automation platforms such as cloudqa, endtest, etc. ?
  13. How do you decide which test cases should be part of the REST API suite and which should be part of the backend integration test suite? Should the tests (REST API and integration) complement each other or there could be some duplication?
  14. What approach is better to use: - Creating a custom framework for Feature testing through the API. Which will combine test cases into test scenarios and run them as test runs and will be testing all the layers of the app (HTTP, Logic layer, Data layer) OR use the tool and make an end-to-end testing directory and keep it in the same code repo with the project?
  15. what tool should i start with, soap ui or postman as a beginner
  16. Can you suggest any open source tools or library for time travel feature?
  17. When thinking of end to end testingā€¦may include different service types like Kafka, JMS, REST, etcā€¦what tools would help you build and maintain these tests?
  18. Can you add SQL db checks in postman API tests
  19. API testing for Non-technical teams/ beginners My dev team use postman with lot collections per project. I am from QA team where most of members are not that much technical. What you suggest how we can pick API testing in most efficient way? because developer always suggest their own way, which are most of the time too complex for test team
  20. When the develops a new endpoint, he creates an hardcoded endpoint in NGINX for other teams to develop against. Is there a better way? How to make a more multipurpose mock?
  21. there are new tables being populated for a change request in my project. can we use api testing for the same, is it not additional efforts since once the testing completes those apis are not going to be used anymore
  22. We as a team adopting API testing with a use of mocha and Mountebank. Iā€™m beginner API tester, what would be the best approach to tackle this. What are the pros and cons of such framework? What tool would you recommend for contract testing?
  23. We are moving from REST APIs to gRPC APIs. Is there a suggested best tool for testing gRPC?
  24. Hi (though I know itā€™s more commonly used as a performance testing tool) is jmeter a good tool to use for api testing compared to something like soapui?
  25. Any tips about mocking proto message?
  26. Is there a specific methodology you follow?
  27. Hi, which is the best tool to test APIs that are in sequence, Ex, API 1 out is the input of API 2 and API 3. It is better to include each API test into a pipeline by CI or there is a tool that could perform it all alone?
  28. What do you think of checking that a json (or xml) schema received is valid? Is it a good strategy to start with?
  29. Do you test REST specific post/get/delete/put for some uri?
  30. Do you check headers in api testing?
  31. Which of these tools do you like for API testing: retrofit2, Apache Http Client, RestAssure, Other?
  32. Can a well defined API-testsuite reduce or remove the need for unit tests in a distributed system with well-defined microservices.
  33. Can we rely 100% on API tests before releasing a backend application? If yes, how can we make sure that nothing will break in the frontend application?
  34. What are your thoughts on GraphQL and related tools e.g. GraphiQL etc? Is this likely to increase in popularity? The POSTMAN support is indeed improving
  35. can we do security or performance tests using API?
  36. Do you check error texts for 400/401/403/etc? And try to avoid 500 answers from the backend.
  37. Why different requests are there? Other than GET and POST.
  38. Do you have a pyramid ( kind of ) for testing API based on your personal experience?
  39. What does containerization mean for API Testing?
  40. What is a soap api?
  41. Beginnerā€™s question: key factors to prioritise when you have 200+ end points?
  42. Do you use a lot of mock data when automating to validate against? When is it good time to mock or not?
  43. Do you have any suggestions on where to go to get API training resources?

Questions Hilary wanted to come back to

  1. Could you please share if you have any ā€œspecialā€ API checks which were discovered while testing or after missing a bug? (ā€œspecialā€ means-> you did not think about it previously but after discovering the problem or missing the bug you have started to check )

For the person who had a Graph QL query, maybe @jamesespie can help

I have been through several API training courses and they all basically show how to structure API requests. I would like to know how to structure tests for a completed GraphQL API. Do you use user journeys (if so how can I confirm all API endpoints have been covered) OR do you create tests for each query and mutation??

For me, yes. Automated checks of all types should by atomic, meaning they depend only on themselves. If they depend on other checks/tests, you start to get a maintenance nightmare when something goes wrong

2 Likes

Questions that are unclear or need more context!
Iā€™ve grouped together questions I can kinda vaguely answer because I donā€™t have enough clarity or context on the question itself. Feel free to message me or reply to this with further details!

Iā€™m not sure what you mean by ā€œstructureā€ - do you mean like file structure of your automation, or the structure of the tests/checks themselves?
Thatā€™s a LOT of endpoints for one repository! I guess with file structure, Iā€™d set it up like the api controllers themselves (so files for each endpoint). If you have a lot of checks for each endpoint, maybe divide that up as well - smoke checks (those that donā€™t make any changes) and functional checks (those that make changes). Definitely have shared utility methods to do the checking for various things, and utilize parameterization where you can. Itā€™s hard to say more specifically without details!

I think they both have their merits. I think there are some details here Iā€™m not getting that would better inform an answer. I hate to say ā€œit dependsā€ but it really does! Feel free to provide more detail and Iā€™ll try to answer better :smiley:

I have no idea lol I wish I could time travelā€¦ not sure what youā€™re referring to here unfortunately

Iā€™m not super clear on the question here, but if youā€™re just looking to make sure tables are populated and thereā€™s no existing API endpoint, you can query the database (assuming you have access to do so)ā€¦

Iā€™m not sure what you mean by methodology here, like functional vs non-functional? All of the above really! An API can and should be tested like any other application.

Iā€™m not sure the question here - basic API testing would be testing each available method (GET/POST/PUT/DELETE/PATCH/etc) for a given endpoint, yes.

1 Like

In the API layer, Iā€™m more looking at business processes and logic, whereas on the GUI layer, Iā€™m looking at user experience as I go through the user processes and flows. These can cross-over a lot, but thatā€™s the general standpoint I start with.

2 Likes

I believe SoapUI is still used as Postman doesnā€™t support SOAP services and those are still out there!

1 Like

Questions that could use Community input!
These are questions I donā€™t know the answer to - API testing is a huge area, and thereā€™s lots I havenā€™t done yet :smiley: Hopefully the community can answer some of these questions!

Is this for mobile apps? I believe you can do that with Charles, as well as Fiddler - set up a proxy from your device so you can see the traffic on your computer that is going on the device. Iā€™m not a mobile tester, so would love some other ideas here if anyone has them!

Great question for the community! I donā€™t write my API tests for performance testing (we have a separate team that uses their own tools for that), so Iā€™m not sure which would be best. Folks, feel free to chime in on this one!

I havenā€™t used them, but in general I like to see the code Iā€™m working with. ā€œcodelessā€ to me just means configurations that creates code that you canā€™t see/touch. I prefer to have full control over the automation Iā€™m writing. Maybe other folks in the community have experience with these, though?

The same tools youā€™re using to write individual tests should work for end to end as well. I havenā€™t worked with Kafka so Iā€™m not sure of the nuances there

This isnā€™t an area that Iā€™m very experienced in - Iā€™ve used mocks in unit tests, but not for developing against. I know thereā€™s a feature in Postman but beyond a brief tutorial I havenā€™t used it.
I suggest googling or asking the community!

I have never tested gRPC but Iā€™m sure the community has some great ideas!

Unfortunately Iā€™ve never used jmeter so youā€™d need to ask the community this one!

Iā€™m not familiar with proto messages, so Iā€™m not sure how mocking them would differ from mocking otherwise. Maybe the community can help?

As I said in the AMA, I havenā€™t had the opportunity to work in the GraphQL world so Iā€™m not sure. The community could help on this one!

1 Like

Swagger or OpenAPI is souped-up API documentation that is generated by the code itself. Itā€™s a quick/starter tool for manually testing an API but itā€™s not customizable as a user. Itā€™s more for a single session at a time, and not saved
Postman/Insomnia/etc are customizable and you can save those settings and share with your team etc, build automated tests within them, etc

1 Like

I have not used any tools that do that. One thing to remember is the Swagger/OpenAPI documentation is generated from the code, and more often comments in the code (not the API code itself). This means itā€™s not always accurate or up to date.
They could be a good starting point, but I donā€™t have any experience.

1 Like

I think of it more like a cloud within a slice of the pyramid, if that makes sense. They all kind of mingle together on that section. Or maybe all side by side? Hope that makes sense!

1 Like

This is certainly still integration testing, in my opinion.
For how to effectively test them, I would think about it like user stories or flows. That way you can prioritize the testing as well based on priority of those flows/processes.

2 Likes

Both of these types of testing have their place. Contract testing is done when youā€™re consuming an API from a 3rd party generally (whether itā€™s another team, or company). These are good to help you both (provider and consumer) ensure that changes being made by the provider of the API donā€™t break things the consumer(s) need/use. This is down to things like datatypes as well as the structure of the responses and response code/response body pairings.
Functional tests actually make sure things still function the way they used to! A provider of an API may or may not make any changes, but we need functional tests to make sure we didnā€™t screw things up either.

1 Like

@lgibbs captured sketchnotes from the session too!

1 Like

I see those as one and the same, so Iā€™m not sure how youā€™re differentiating here.
I answered a question during the live AMA about separating tests between API and database, for instance, if thatā€™s what youā€™re getting at here?
For that, essentially ā€œit dependsā€ of course, but you need to see what your database is doing vs what the API is doing, and verify theyā€™re doing their things correctly.
For instance, if a DELETE from the API isnā€™t actually supposed to delete from the database (it sets a flag in the record, a type of ā€œsoft deleteā€), then youā€™d want to check both the API and database for that operation. But if itā€™s really supposed to remove from the database, then an API GET for that same item you deleted would suffice in my opinion.

I think they both have their merits. I think there are some details here Iā€™m not getting that would better inform an answer. I hate to say ā€œit dependsā€ but it really does! Feel free to provide more detail and Iā€™ll try to answer better :smiley:

This depends on the API youā€™re testing. It looks like postman does support SOAP (which I didnā€™t think it did), so that might be a good place to start. There are a lot of great tutorials out there, and itā€™s a pretty well used product so you can get support (on The Club, slack, or twitter) if youā€™re stuck.

I have no idea lol I wish I could time travelā€¦ not sure what youā€™re referring to here unfortunately

Thatā€™s something youā€™d need to Google, I really donā€™t know. I would assume not - you might need to use SQL Server Management Studio or similar to do that

First, I think that anyone that works in software is very technical, we just have different specializations!
I think if theyā€™re not familiar with how APIs work, Iā€™d start there. There are some great courses on The Dojo for that like https://www.ministryoftesting.com/dojo/courses/introduction-to-http and https://www.ministryoftesting.com/dojo/courses/the-building-blocks-of-the-internet-mark-winteringham

Once they have that familiarity, I think folks can do some tutorials with a tool like Postman to become familiar with that and how it interacts with the APIs.
Thereā€™s a lot of value in using the same tools as the dev team, so having one of the devs run folks through how things work with your APIs in Postman would help, too.

Iā€™m not super clear on the question here, but if youā€™re just looking to make sure tables are populated and thereā€™s no existing API endpoint, you can query the database (assuming you have access to do so)ā€¦