API testing - How to test timeouts and exceptions?

APIs can timeout or return exceptions i.e. HTTP 5XX statuses like 500, 503, 504. Should a QE test these scenarios? If yes, then how can we test it? So far, I have never seen ways to simulate the conditions which would cause http 5xx to be returned. I feel that such scenarios should be covered by unit tests of the developers. I think it is not worth the effort to test such things.

Example - Say that my API1 needs an external API2 to do some work. API1 can return a 5xx code when it has a problem or when API2 has a problem. When there are no ways to simulate either scenario, then how do I test those scenarios? Wouldn’t that require the developers to add some testability features to simulate those scenarios? Is it really worth the trouble?

3 Likes

Adding the test ability of those scenario’s is worth it. (in some cases I suppose)
Are your applications unstable or do you reply on other teams to develop the other API2? Then yea I would definitely do it. For example 503. Service Unavailable. With an appropriate message. So you can test if the message is good.

1 Like

This is where I take advantage of techniques such as mocking.

When possible I also control my environment, because it’s typically a docker stack.

So I can take API1 down (stop the docker container) or simulate it responding in a given way (via Mocking).

It’s sometimes possible to identify issues that are not easy to pickup in unit tests this way.

I’ve also written small web servers that respond in a given pattern, either with errors or adding delay for example.

1 Like

Have you looked at API mocking tools like WireMock (which I maintain), MockServer, Hoverfly etc.?

In the scenario you described you could configure WireMock to simulate API 2, then set specific stubs to do things like return 503, drop the connection or add a long delay before before responding. This would allow you to test the effect of these failures on API1’s behaviour.

Here’s some more detail on how to simulate faults with WireMock: Simulating Faults - WireMock

2 Likes

@tomakehurst - Thanks. To implement your suggested test approach, I’d guess that we should be able to choose whether API1 talks to API2 or a mock of API2, i.e. we should make API1 configurable. Then, we can test such scenarios easily and quickly. However, if the API1 cannot be configured at all or cannot be configured easily, then can we still use mocking?

As an aside, what can we do in the initial stages of API development to enable mocking?

@bencf1 - If an API is in initial stages of development, then what are the features we can suggest to make it easy to test timeouts & exceptions?

Regarding writing small web servers to return desired responses, are there any tools which generate such servers easily instead of writing/coding them? If not, could you please suggest any tutorials on how to write such web servers, say in Java?

If possible, could you please share one situation in which a unit test would not be able to find an issue which mocking would? I am wondering if the unit test could be made better or if the testing is better done with mocking/higher level tests.

@raghu yes, to do this you’d need some way to tell API1 to where to send its API2 calls.

In practice there are three ways you can achieve this:

  • Modify API1’s configuration to change the hostname/base URL for API2 (or create an additional environment configuration that does this).
  • Change the proxy settings for API1, either directly in its configuration or via its host. Sometimes this approach is helpful when you can’t modify the app but you can tweak the machine it’s running on.
  • Manipulate the DNS service provided to API1 so that the original domain name of API2 is remapped to your mock server. The difficulty of this varies depending on your environment, but quite straightforward if you’re running in Docker.

An additional wrinkle for the 2nd and 3rd option is SSL. If you mess with DNS or proxy settings and API1 is calling API2 over HTTPS, then you’ll need to find a way to make API1 trust a different SSL certificate.

Once you’ve figured out which of these strategies to follow, then next step is to start identifying which API calls are made by API1, then creating stubs for them with realistic happy-path responses. Then you can expand on this with more stubs for error cases as we discussed.

2 Likes