30 Days of API Testing Day 2: API Exploratory Testing

From our friends participating on Twitter:

Iā€™d start with reading Exploratory Testing an API post by Maaret PyhƤjƤrvi :slight_smile:

Iā€™d use exploratory testing techniques I already use for testing systems across the entire stack such as:

  1. Map out the endpoints & associated params of the API under test.
  2. Map out the endpoints & associated params of APIs that communicate with the API under test.
  3. Compare responses with their requests
  4. Experiment with different variations of params to see the API behaviour.
  5. Identify example messages from typical user journeys to derive variations of those journeys (e.g in/valid tokens).
  6. Replay troublesome user journeys from production through the API to reproduce undesired behaviour.
  7. Identify data being passed through the API & use the STRIDE threat model to discover vulnerabilities of that data.
  8. Compare the API with itā€™s documentation to identify discrepancies.
  9. Force the API to fallover - how is the error handled? What is the impact to the customer?
  10. Open the console in Postman! (Top tip in Alanā€™s video)

EDIT Iā€™ve just re-read Ash Winterā€™s ICEOVERMAD mnemonic
Daniel Donbavandā€™s SPEEDLOADS mnemonic & Linda Royā€™s VADER mnemonic - these would serve as great starting checklist from which I could launch deeper explorations.

3 Likes

I started to learn about API Exploratory Testing by reading post by @maaret on Exploratory Testing an API and Johnny Mnemonic ICEOVERMAD.

Following are the points that I would cover while doing Exploratory Testing of API-

  1. Explore the Scope of the APIs developed:
  • Parameters,Inputs,Constraints in API.
  • Requests/Responses
  • If the API/APIs are dependent on other APIs (Nested APIs.)
  • Server being used
  • Error Handling (Exception, Error messages being used)
  • Flowchart of the APIs.
  • Interact with developers who have developed the APIs to get the in depth knowledge.
  • Interact with the users( developers who will use the APIs in their code) to know the purpose of the developed APIs ( their expectation from the developed APIs.)

2.Understand the explored scope and come up with Testing Scenarios to cover the complete scope:

  • Passing Valid / Invalid inputs and parameters.
  • Nested API scenarios
  • Load testing scenarios to check the performance.
  • Scenarios to check the Authentication and Authorization.
  • Exploring the Scenarios in which the API will fail.
  • Checking if the API will justify the purpose for which it was developed after being integrated with the Application.

3.Documentation: Document the complete testing process so that it can be referred by the developers and can be used in future for testing. Documentation should be done from the start i.e from identifying the testing scope to compiling the test execution report.

3 Likes

I started to learn about API testing from my project KBB at KMS Technology. Below that is my approach for API Exploratory test.

  1. Understand API requirements

    • What is the APIā€™s purpose?
    • Parameters,Inputs,Constraints in API.
    • Requests/Responses
    • Error Handling (Exception, Error messages being used)
  2. Specify the API output status:

    • Verify responses code equals to 200 or not

    • There are five values for the first digit:

      • 1xx (Informational): The request is received and continues to be processed
      • 2xx (Successful): The request is successfully received, understood, and accepted
      • 3xx (Redirection): Further action needs to be taken to complete the request
      • 4xx (Client Error): The request contains the wrong syntax or cannot be fulfilled
      • 5xx (Server Error): The server fails to fulfill an apparently valid request
  3. Focus on small functional APIs

  4. Using some testing techniques(equivalence classes, boundary value analysis, error guessing) to write a test case for your API

  5. Run testcases and compare actual result with expected result.
    You can find more information related to API testing at https://www.katalon.com/resources-center/blog/api-testing-tips/.

  6. Apply automation testing for API

9 Likes

There is my approach for API testing:

  • Refer some documents to get the basic knowledge about API testing ex https://docs.katalon.com/katalon-studio/tutorials/introduction_api_testing.html
  • Learn about JSON, XML format, HTTP methods, the requests, responses, how to handle the error ā€¦ (because I donā€™t know about API so it will take a lot of efforts)
  • Get the basic knowledge about tools for API testing such as Postman, Katalon ā€¦
  • Define test case and apply API testing for a sample API testing project such as load testing to check the performance on the website
4 Likes

I think the approachment for API Exploratory Testing is based on many aspects, one of that is Software Development Life Cycle (SDLC). Different SDLCs have different approaches. For example:

In traditional Waterfall/V-model: The testers collect requirement, apply the testing technique to generate Test Cases/Test Scripts. After the program is completed its Built Stage, they execute the Test Suite and collect the results.


In Agile, the Test Ideas are not mainly come from reading Requirement, but also from exploring the working software. More cases are tested, and so more results are gotten. The testers will have more knowledge about the product, so they can explore it better, generate better Test Cases/Test Scripts.

In DevOps culture, the SDLCs can be shorter than in Agile model (for faster delivery, better stability, etcā€¦) and also more challenges. The testers will need CI/CD documents, explore the whole system (not just only the product), so they can generate even better Test Suite that covers more and costs less. They will have more time to understand the system, or learn new thing. Combining API Testing with other skill sets/knowledge will bring more positive change to the product.

Thatā€™s my approachment for API Exploratory Testing, from different SDLC points of view. I hope to know more about your opinion. Thanks!

6 Likes

My approach API Exploratory Testing get from Heuristic Test Strategy Model:

  • Focus working with limit understanding about API requirements: my testing target with the environment it depends on, purpose of users.
  • Research API tools (ex: Postman, katalonā€¦) to choose the tool suitable with my project
  • Determine scenarios for testing
  • Define: calls and operations, inputs and outputs, exceptions.
  • Write scripts for API testing
  • Run & report results
5 Likes

Thank all for sharing. Iā€™m a automation tester and not familiar with API testing as well as exploratory testing. After researching about exploratory testing and reading some of your opinions, I found my own approach as below:

  • Read/ Explore the documentation if it is available and try to understand
  • Ask around for what the intended uses are and what applications we know of are currently using our API
  • Review the code for the API or go over the API with the developer if possible
  • Know the endpoint, and what are the operations it supports
  • Compare responses with their requests
  • Experiment with different variations of params to see the API behaviour.
  • Identify example messages from typical user journeys to derive variations of those journeys (e.g in/valid tokens).
  • Checking if the API will justify the purpose for which it was developed after being integrated with the Application.
  • Document the complete testing process so that it can be referred by the developers and can be used in future for testing
6 Likes

Exploratory testing is simultaneous learning, test design, and test execution
(Exploratory Testing Explained by James Bach, v.1.3)

Exploratory testing is more like an approach or mindset than a methodology, itā€™s the way to understand product/features deeply and widely with the purpose of finding bugs through investigation and learning freedom based on testerā€™s skill, experience and adaptability.

In my opinion, via seeking and reading about API Exploratory Testing, I think it is use automation tools to test APIs within Exploratory approach. Besides, because API is almost automated test, it assists about reproduction and regression whatā€™s tested, which is a difficult when applying exploratory in manual testing.

5 Likes

Below is my approach for API Exploratory testing:

  1. Get information about this API (some item)
  • The API endpoint
  • The API method: GET, POST, PUTā€¦
  • Parameter (if any)
  • HTTP status Code
  • The response value
  • Authentication (if any)
  1. Play with the API:
  • Execute the API with valid/invalid parameter
  • Try some exception case.
  • Try with/without authen token. And invalid token.
  1. See the response:
  • Observe if the response is match your expected on: the response type, status code, value filed.
  • The status code of response should match your expectation.
  1. Document: write down your knowledge about this API for future using.

  2. DO automation:

  • Try create the automated script for your API.
4 Likes
1 Like

I think I will go API Exploratory Testing like below steps:

  1. Read more about the api testing for different testing types
  2. Read documents for api testing at basic level to explore the whole picture of api testing
  3. Try to find a tool
  4. Read the tourtorial of the tool and explore how to use the tool
  5. Read api testing example
  6. Practice the example on tool with different api testing example
  7. Note down
4 Likes

Iā€™m a newbie in API testing. After researching about exploratory testing an API on the Internet as well as referring your comments. I would like to share my own apinion about API exploratory testing approach as below:

Prerequisite:

  • Know the different between web service and API.

  • Know types web service likes SOAP/ REST.

  • The format of the API request/response ( XML, JSON,ā€¦)

  • What is a HTTP API request/response.

  • Is there a documentation of each service? How it works? What business function service perform?

  • What are HTTP methods - GET, POST, PUT, DELETE?

  • What methods which web service supports?

  • List of HTTP status code

  • Tools: POSTMAN, SOAPUI,ā€¦

  • Configuring a tool to request an API.

API exploratory testing approach:

  • First of all, read the document if it is available and make sure you understand it.

  • Talk to the developers to understand what inputs need to make an API call.

  • Know the end point and what form does the end point take.

  • Try to communicate with client or developers if there are any issues.

  • Make a request and review the results.

  • Make an invalid request and see the HTTP status and response returned.

2 Likes

When it comes to exploratory testing, there are some misconceptions that ET (Exploratory testing) is only used with GUI testing and intended for Devs;even some are surprised when ET is mentioned in API. Below is my approach for API ET after reading some articles related:

Before jumping to write API testcases, do the following preparation steps underneath:
1. Try to get familiar with API concepts:

  • HTTP => used to transfer web based information
  • Servers and Clients, Requests and Responses
  • Creating HTTP methods (get, put, post,delete,ā€¦)
  • How data (input/output) is formed in requests/responses (XML (Extensible Markup Language) and the more recent JSON (JavaScript Object Notation))
  • Status codes: are divided into 5 classes
  • 1xx (Informational): The request is received and continues to be processed
  • 2xx (Successful): The request is successfully received, understood, and accepted
  • 3xx (Redirection): Further action needs to be taken to complete the request
  • 4xx (Client Error): The request contains the wrong syntax or cannot be fulfilled
  • 5xx (Server Error): The server fails to fulfill an apparently valid request
  • Authorization (who can access the API?)
  • Endpoints
  • Pick up an automation tool that u find most comfortable to practice

2. To do API testing:

  • Read API docs carefully before writing API testcases
  • Pick up a few scenarios to write API testcases(not too complicated)
  • Determine inputs, outputs, responses
  • Organize your endpoints
  • Choose suitable verification methods:
  • Compare the whole response body content with the expected information
  • Compare each attribute value of the response
  • Compare matching with regular expression
  • Write negative and positive TCs

For more details, please refer as below:



https://www.signupto.com/news/digital-marketing/what-is-an-api/

That is mine, Iā€™d love to hear more thoughts from you guys :slight_smile:

1 Like

Really very excellent article. Got to know many new information. Thanks for sharing this article with us.

I am new to this terminology so I did some research on Google and come up with this.

Exploratory testing is a systematic approach for discovering risks using rigorous analysis techniques coupled with testing heuristics. A lot of times we test before thereā€™s a graphical user interface or finalized features available, and exploring is the mindset that helps us uncover things we do not know.

Approach:

  1. Understand Type of APIā€™s
  2. Read documents carefully before starting Testing
  3. Create Explore Plan
  4. Verify Negative test cases
  5. Document result of Exploratory testing

Important steps to follow to perform API exploratory testing ,

*going through the provided documentation, understanding it.

  • knowing the usability of the API

  • Discussing with the developer about the API, what kind of responses that can expected for the calls tat are made

  • try the different combinations of data, like with what value call is made and what response is obtained

  • know the environment the API depends on

  • know about who uses it and what for

1 Like

Let us assume for 1 minute that we are not so lucky to actually have a documentation for the API, no swagger, no collegues who did the implementation no nothing. Well if this is the case they the fun really starts because we must get creative

For me the process would be the following

  • Filter the calls to the the backend - Since the API has to be used by the current application that I am testing if I review carefully the requests made to the backend after a few tries I may find a pattern. I may be able to see that specific URL ( endpoint) that is appearing over and over
  • Write down all of the enpoints found
  • Differenciate betwenn the different verb available per enpoint ( GET, PUT, POST, DELETE etc)
  • Playing with GET
    • Try all different kinds of parameters ( no parameters, a lot of parameters that make no sense, the parameters that you can see in the request, there is no limit, be creative)
    • Definetly check and note down the errors codes
    • Write down in RED the requests that generate a response starting with 5 ( 500 is always nice :slight_smile: )
  • Playing with POST and PUT
    • Here the body is the most import
    • You need to find out what the standard body is for the request
    • Once that is known sending variations of that body is key ( different values for the parameters, no all parameters, no parameters)
    • Besides writing the response from you requsts you need to make sure that the Operation was performed succesfully. You can check the update either in the UI or via a GET Request
  • Playing with DELETE
    • Be very carefull where you delete item from ( one request and the data is gone)
    • You need to find out what the standard body is for the request
    • Besides writing the response from you requsts you need to make sure that the Operation was performed succesfully. You can check the update either in the UI or via a GET Request
  • Once you know now some enpoints and how to use them you can check if they work the same also on different systems ( if there are any)
  • Depending on the authorization of the API you can try to see if you can go around it. Different users, different API, using expired tokens. This all depends on context
  • Last but not least is performance. You have now several enpoints and you are not afraid to use them so you make a few thousand requests per minute ( think Black Friday sale) and see how the API and the application react.

For me it is always fun to do exploratory testing. The less you know the more you can find out. It is not always easy but in the end it is always fun.

2 Likes

Great process @restertest ! Thank you for sharing it!

1 Like

How would you approach API Exploratory Testing?

  • Mission. Define or understand mission or define a charter to stay focused
  • Product. Try to understand how API works for this product. Possible ways include:
    • read API documentation,
    • ask someone,
    • google to learn concepts or similar products
    • use tools, experiment, try calling the service, observe responses/requests.
    • create models or diagrams if they can help you to understand how system works
    • do not overcomplicate, just start playing with what you have is ok
  • Design tests. There are multiple approaches.
    • Common test types
      • Basic positive tests (happy paths)
      • Extended positive testing with optional parameters
      • Negative testing with valid input (for example, trying to add an existing username)
      • Negative testing with invalid input (trying to add a username which is null)
      • Destructive testing (for example, fill in long text into input field).
      • Security (does this response leak information which can be used to break into the system?.
    • Heuristics (e.g. SFDIPOT, SLIME)
    • What can go wrong. What are the risks?
    • What kind of tests would be useful to automate?
      • Could it be disposable, one time use?
      • The value of it is in the learning it provided now?
      • Are these tests that you want to see yourself maintaining?
    • Dependencies
    • Add variation to requests by changing type, range, and length of variables and/or omitting some variables
    • Keep tests as small as possible
  • Perform tests and experiments
    • Collaborate
    • Repetition helps to understand
  • Notes. Make notes on your observations
    • questions
    • issues (what makes testing hard or impossible)
    • bugs,
    • inconsistencies
    • organize tests in groups
  • What API testing gives us
    • Automation Testing
      • Spec
      • Feedback
      • Regression
      • Granularity
    • Exploratory
      • Guidance
      • Understanding
      • Models
      • Serendipity
1 Like