Tonight we were joined by the fabulous @g33klady for an Ask Me Anything all about API Testing.
Iāll add all the resources mentioned in the webinar to this thread along with any questions we donāt get to. If I miss any resources or you thought of a question youād like to add, why not add it to this thread?
If you missed the live session, a recording will be available on the Ministry of Testing website for all Club level members once weāve edited it and added captions.
Is it wrong to make my suiteās tests dependent on each other?
what is the best way to structure api endpoint tests with ~300 endpoints in your repository? many of the calls are divided in get, post, patch, delete calls and there are also different endpoints for lists and single objects? Each should be tested against JSON Schema and data param-values for ex. thank you
The automation pyramid is made up of the unit layer, api layer and the UI layer. I know the difference between each of these layers - how do you decide what tests scenarios should be covered in the GUI layer and what should be covered in the API layer?
is soap ui used even now or been replaced by postman
Any specific tool to check which APIs are used in app. Also, how to know which APIs are triggered upon app activity (eg: App Launch) and the time taken. I use Charles Proxy.
how different is swagger tool, is it more popular than postman, difference between the two
Have you used any tools that automatically create tests from a Swagger json? How useful have you found them?
How do API tests of different types usually sit within the āsoftware testing pyramidā for you? In particular, testing APIs/services that should integrate to give particular behaviour vs. testing an API/service in isolation.
How to effectively test multiple interactions between different APIs? Which level of test automation does this fit in?
What is the value of doing contract testing (with a tool like PACT) vs. functional testing of API responses?
What are your thoughts on codeless automation platforms such as cloudqa, endtest, etc. ?
How do you decide which test cases should be part of the REST API suite and which should be part of the backend integration test suite? Should the tests (REST API and integration) complement each other or there could be some duplication?
What approach is better to use: - Creating a custom framework for Feature testing through the API. Which will combine test cases into test scenarios and run them as test runs and will be testing all the layers of the app (HTTP, Logic layer, Data layer) OR use the tool and make an end-to-end testing directory and keep it in the same code repo with the project?
what tool should i start with, soap ui or postman as a beginner
Can you suggest any open source tools or library for time travel feature?
When thinking of end to end testingā¦may include different service types like Kafka, JMS, REST, etcā¦what tools would help you build and maintain these tests?
Can you add SQL db checks in postman API tests
API testing for Non-technical teams/ beginners My dev team use postman with lot collections per project. I am from QA team where most of members are not that much technical. What you suggest how we can pick API testing in most efficient way? because developer always suggest their own way, which are most of the time too complex for test team
When the develops a new endpoint, he creates an hardcoded endpoint in NGINX for other teams to develop against. Is there a better way? How to make a more multipurpose mock?
there are new tables being populated for a change request in my project. can we use api testing for the same, is it not additional efforts since once the testing completes those apis are not going to be used anymore
We as a team adopting API testing with a use of mocha and Mountebank. Iām beginner API tester, what would be the best approach to tackle this. What are the pros and cons of such framework? What tool would you recommend for contract testing?
We are moving from REST APIs to gRPC APIs. Is there a suggested best tool for testing gRPC?
Hi (though I know itās more commonly used as a performance testing tool) is jmeter a good tool to use for api testing compared to something like soapui?
Any tips about mocking proto message?
Is there a specific methodology you follow?
Hi, which is the best tool to test APIs that are in sequence, Ex, API 1 out is the input of API 2 and API 3. It is better to include each API test into a pipeline by CI or there is a tool that could perform it all alone?
What do you think of checking that a json (or xml) schema received is valid? Is it a good strategy to start with?
Do you test REST specific post/get/delete/put for some uri?
Do you check headers in api testing?
Which of these tools do you like for API testing: retrofit2, Apache Http Client, RestAssure, Other?
Can a well defined API-testsuite reduce or remove the need for unit tests in a distributed system with well-defined microservices.
Can we rely 100% on API tests before releasing a backend application? If yes, how can we make sure that nothing will break in the frontend application?
What are your thoughts on GraphQL and related tools e.g. GraphiQL etc? Is this likely to increase in popularity? The POSTMAN support is indeed improving
can we do security or performance tests using API?
Do you check error texts for 400/401/403/etc? And try to avoid 500 answers from the backend.
Why different requests are there? Other than GET and POST.
Do you have a pyramid ( kind of ) for testing API based on your personal experience?
What does containerization mean for API Testing?
What is a soap api?
Beginnerās question: key factors to prioritise when you have 200+ end points?
Do you use a lot of mock data when automating to validate against? When is it good time to mock or not?
Do you have any suggestions on where to go to get API training resources?
Questions Hilary wanted to come back to
Could you please share if you have any āspecialā API checks which were discovered while testing or after missing a bug? (āspecialā means-> you did not think about it previously but after discovering the problem or missing the bug you have started to check )
For the person who had a Graph QL query, maybe @jamesespie can help
I have been through several API training courses and they all basically show how to structure API requests. I would like to know how to structure tests for a completed GraphQL API. Do you use user journeys (if so how can I confirm all API endpoints have been covered) OR do you create tests for each query and mutation??
For me, yes. Automated checks of all types should by atomic, meaning they depend only on themselves. If they depend on other checks/tests, you start to get a maintenance nightmare when something goes wrong
Questions that are unclear or need more context!
Iāve grouped together questions I can kinda vaguely answer because I donāt have enough clarity or context on the question itself. Feel free to message me or reply to this with further details!
Iām not sure what you mean by āstructureā - do you mean like file structure of your automation, or the structure of the tests/checks themselves?
Thatās a LOT of endpoints for one repository! I guess with file structure, Iād set it up like the api controllers themselves (so files for each endpoint). If you have a lot of checks for each endpoint, maybe divide that up as well - smoke checks (those that donāt make any changes) and functional checks (those that make changes). Definitely have shared utility methods to do the checking for various things, and utilize parameterization where you can. Itās hard to say more specifically without details!
I think they both have their merits. I think there are some details here Iām not getting that would better inform an answer. I hate to say āit dependsā but it really does! Feel free to provide more detail and Iāll try to answer better
I have no idea lol I wish I could time travelā¦ not sure what youāre referring to here unfortunately
Iām not super clear on the question here, but if youāre just looking to make sure tables are populated and thereās no existing API endpoint, you can query the database (assuming you have access to do so)ā¦
Iām not sure what you mean by methodology here, like functional vs non-functional? All of the above really! An API can and should be tested like any other application.
Iām not sure the question here - basic API testing would be testing each available method (GET/POST/PUT/DELETE/PATCH/etc) for a given endpoint, yes.
In the API layer, Iām more looking at business processes and logic, whereas on the GUI layer, Iām looking at user experience as I go through the user processes and flows. These can cross-over a lot, but thatās the general standpoint I start with.
Questions that could use Community input!
These are questions I donāt know the answer to - API testing is a huge area, and thereās lots I havenāt done yet Hopefully the community can answer some of these questions!
Is this for mobile apps? I believe you can do that with Charles, as well as Fiddler - set up a proxy from your device so you can see the traffic on your computer that is going on the device. Iām not a mobile tester, so would love some other ideas here if anyone has them!
Great question for the community! I donāt write my API tests for performance testing (we have a separate team that uses their own tools for that), so Iām not sure which would be best. Folks, feel free to chime in on this one!
I havenāt used them, but in general I like to see the code Iām working with. ācodelessā to me just means configurations that creates code that you canāt see/touch. I prefer to have full control over the automation Iām writing. Maybe other folks in the community have experience with these, though?
The same tools youāre using to write individual tests should work for end to end as well. I havenāt worked with Kafka so Iām not sure of the nuances there
This isnāt an area that Iām very experienced in - Iāve used mocks in unit tests, but not for developing against. I know thereās a feature in Postman but beyond a brief tutorial I havenāt used it.
I suggest googling or asking the community!
I have never tested gRPC but Iām sure the community has some great ideas!
Unfortunately Iāve never used jmeter so youād need to ask the community this one!
Iām not familiar with proto messages, so Iām not sure how mocking them would differ from mocking otherwise. Maybe the community can help?
As I said in the AMA, I havenāt had the opportunity to work in the GraphQL world so Iām not sure. The community could help on this one!
Swagger or OpenAPI is souped-up API documentation that is generated by the code itself. Itās a quick/starter tool for manually testing an API but itās not customizable as a user. Itās more for a single session at a time, and not saved
Postman/Insomnia/etc are customizable and you can save those settings and share with your team etc, build automated tests within them, etc
I have not used any tools that do that. One thing to remember is the Swagger/OpenAPI documentation is generated from the code, and more often comments in the code (not the API code itself). This means itās not always accurate or up to date.
They could be a good starting point, but I donāt have any experience.
I think of it more like a cloud within a slice of the pyramid, if that makes sense. They all kind of mingle together on that section. Or maybe all side by side? Hope that makes sense!
This is certainly still integration testing, in my opinion.
For how to effectively test them, I would think about it like user stories or flows. That way you can prioritize the testing as well based on priority of those flows/processes.
Both of these types of testing have their place. Contract testing is done when youāre consuming an API from a 3rd party generally (whether itās another team, or company). These are good to help you both (provider and consumer) ensure that changes being made by the provider of the API donāt break things the consumer(s) need/use. This is down to things like datatypes as well as the structure of the responses and response code/response body pairings.
Functional tests actually make sure things still function the way they used to! A provider of an API may or may not make any changes, but we need functional tests to make sure we didnāt screw things up either.
I see those as one and the same, so Iām not sure how youāre differentiating here.
I answered a question during the live AMA about separating tests between API and database, for instance, if thatās what youāre getting at here?
For that, essentially āit dependsā of course, but you need to see what your database is doing vs what the API is doing, and verify theyāre doing their things correctly.
For instance, if a DELETE from the API isnāt actually supposed to delete from the database (it sets a flag in the record, a type of āsoft deleteā), then youād want to check both the API and database for that operation. But if itās really supposed to remove from the database, then an API GET for that same item you deleted would suffice in my opinion.
I think they both have their merits. I think there are some details here Iām not getting that would better inform an answer. I hate to say āit dependsā but it really does! Feel free to provide more detail and Iāll try to answer better
This depends on the API youāre testing. It looks like postman does support SOAP (which I didnāt think it did), so that might be a good place to start. There are a lot of great tutorials out there, and itās a pretty well used product so you can get support (on The Club, slack, or twitter) if youāre stuck.
Thatās something youād need to Google, I really donāt know. I would assume not - you might need to use SQL Server Management Studio or similar to do that
Once they have that familiarity, I think folks can do some tutorials with a tool like Postman to become familiar with that and how it interacts with the APIs.
Thereās a lot of value in using the same tools as the dev team, so having one of the devs run folks through how things work with your APIs in Postman would help, too.
Iām not super clear on the question here, but if youāre just looking to make sure tables are populated and thereās no existing API endpoint, you can query the database (assuming you have access to do so)ā¦