Me personally, I’ve seen more of what you describe - status response, response time and select contents. But the entire response idea looks interesting.
We use file matching a lot in our functional tests , where we have a reference file with the expected response then compare the actual response against it. The jsonunit and xmlunit Java libraries are pretty good for this, in particular for their ability to put an ignore flag into fields in the reference file so that they don’t get compared. This is very useful where you get things like unique identifiers that you know will be different in every response.
Thanks Adam, how’s the performance of the test itself when compare the whole response body. Does it take significant longer than just verify the status code and a couple of friends?
I’ve never done any actual inspection of response times, but anecdotally I can’t say I’ve seen any significant difference in the time taken to do a reference file comparison as compared to asserting against key parts of the response. That’s from running a test pack containing both types of tests, but I wouldn’t rule out the difference becoming significant for say, a load test using the same test pack.
On the side note, I would wonder what cases checking the whole response is useful…
In general, a consumer tend to use only part of a provider’s response (because a provider have 1 or more consumers).
E.g., a PlayerInfoService can return the name, age, and stats of a player. Some service will use only the name and stats - whereas other service will use the name and age.
If the contracts both consumers give to the producer include the three attributes, we will have a false positive when only the stats or age attributes are problematic, given these are not used by both consumers.
On my last project the API returned XML and the company had an XML Schema for the returned data. Testing was so simple with Mocha/Chai and xmllib.
Previous to that I started specifying the API, which should return JSON, with Swagger. It ought to be possible to generate a conformance test from the Swagger specification. I never actually got round to that, but someone might have already investigated it.
I’ll often have one test which tests the status code of the response is correct, another test that checks the schema of the response and a third test which tests everything else.
I then tag the tests so they can be run in different suites. When the developers check in code, we want to run the status code and schema tests. This way of they add information to the response, we’ll catch that and update as necessary. Then the full API tests will be on a schedule and run separately.
As you are asking about verification of response, you seems to be interested in automation of APIs. You can perform API automation using Java language as well through RestAssured library, it is easy way for API testing. For this, you can create easy framework to create/manage the test scripts.
Let’s take below example:
Here you must use the JSON format to send the body request and you will receive the response in JSON as well.
Request format:
String body = "{"+
"\"name\":\""+userName+"\","+
"\"action_id\":1,"+
"}";
// POST the request and save the response.
Response response = postRequest(body, url);
// Now you need to parse the response in order to verify the response status and userID, which is generated on sending user details.
JSONObject JSONResponseBody = new JSONObject(response.body().asString());
//Asserting that Status code is 200. [200 is used for success status]
Assert.assertEquals(response.getStatusCode(), 200);
// Get the ID of user
String userId = (Integer)JSONResponseBody.get(“id”);
In the same way, you can parse all the fields returned in response through JSON.