Contract testing: is it something only for complex high intensity consumer provider architectures?

Lately, I’ve been trying to get a grip of what contract testing is and how different is it from API testing.

Contract testing is the latest talked about approach which is basically writing some code to check API response schemas and codes. In some cases you’d be checking the request payload as well.

I think that API testing generally happens on the consumer side and in a full scale API testing suite, we’re basically doing what contract tests do. It is said that contract tests are “non-functional” since they don’t test an API in depth. Another point that has been discussed is that contract tests are beneficial in systems where there are several microservices communicating with each other. They’re also considered “faster” because apparently they work with mocks instead of the real thing? (Not sure)

For a simpler architecture where there’s a backend communicating with a front-end through APIs, you really don’t need a contract test but rather the traditional API test would do. Specially in cases where running through all your APIs would not take too much time and thus there’s no real need for “faster” tests on API level.

I must admit at this point that my programing prowess is not that great, and for now from just reading through blogs and articles, writing contract tests seems like an awful lot of manual labor for a seemingly simple task.

In my simple world, a good test suite could be api tests that:

  1. Go through end to end scenarios or user journeys
  2. Check each API separately to verify schema and response codes
3 Likes

I agree that it is a tool in your toolbox. How often do you need to maintain a relationship between a provider API and many consumer’s APIs? I wonder at what scale is that worth the effort?

1 Like

What if someone did it, not very often?

There’s a fundamental difference between the two.
First, API testing is not a real thing. Or at least, in the way it’s commonly used it isn’t - it’s just testing a product that happens to expose a RESTful (or other) API. You rarely hear of people testing the API itself as opposed to testing the application logic through an API.
But, rants aside, there are differences in purpose, size and scope.
Contract tests are intended to answer a single & focused question - Do I break my commitments between provider and consumer (which might be a server and a client). If I have logic problems and returns wrong answers? Some edge cases are crashing my application? contract tests are the wrong place for finding that.

Having an enforced contract provides better decoupling - I’m able to release only one component without testing it against the other (yes, contract tests are not the only type of tests needed in order to have this ability, but they are part of it).
The size difference is a real mindshift - since we are focused on the thin layer that communicates between both sides, we are mocking all the rest, getting what is practically a unit test (or a component test if we didn’t architecture for it properly). Compare that to sending a REST request to a full aplication, digging in the logs of 14 different services to see which of them has produced a mistake.

Finally, there’s the scope - API tests are centred around the application under test, and represent our best understanding of the other side’s expectations. Contract tests are focused on the interaction between two applications, and comes with the promise that both sides are running the same (or similar enough) tests.

Those differences mean that you’ll want to use those two types of checks in different situations - for instance, you don’t really want to attempt contract tests if you are developing only one side of the channel (unless you can collaborate with the other team and be sure both of you run them, and have a mechanism for updating contracts over new features).

As for your concern of excessive manual labour - I’ve played a bit with Pact (I think it’s one of two tools aiming to do contract testing, and from what I’ve seen, the de-facto go-to tool for most situations, but I’m not really knowledgeable in this area), and the effort is rather minimal - you write a normal unit test on the consumer’s side, but instead of checking every piece of output yourself, you declare those expectations in Pact format and ask the tool to verify it for you. The main hassle is how to coordinate when developing new features that change the contract.

1 Like

Thanks for the detailed answer Amit. I think I’ll give Pact a shot some time too. But for now since the API is already there and has been since the past couple of years. I think an automated API check would make more sense for me.

That’s sounds reasonable - the main motivation for contract tests is when you are developing both sides of the contract and want fast feedback before deploying it to a full system test.

example of why we are starting to do this:
We are creating a service based product, that interacts with a web-server (that we develop). As such, we have implicit contracts between each component of the first product, and between some of the components and the server (which in turn might have its own internal contracts, but that’s not part of our scope for now). Since deploying a full system in our case would require us to create a new AWS account for each run, that’s not feasible to do for every pull-request, so our way of mitigating this is to run contract tests as part of the PR and thus spot changes that break other components expectations.

Consumer-driven contract testing is more about decoupling the working methods and the services themselves from the state of the finished development when developing services. In addition, the word “consumer” is based on the fact that the provider can fulfil the necessary requirements of the consumer. API testing is more a step of verifying whether something responds correctly.

It could. It’s a promise and you need to know when the promise is broken.

I implemented contract testing recently in a microservices environment where teams provided APIs for each other. It worked well.

I probably wouldn’t implement it unless you’re crossing team boundaries - it’s quite a lot of overhead. In our case we had some outages which were caused by violated contracts so it seemed to make sense.

The bugs contract testing finds could always be found by end to end testing but that’s often more work. Contract testing can reduce the need for end to end testing while still maintaining confidence in the released software.

From what I understand pact is fucking awful. I didnt use pact. It’s another sad example of where the flagship technology of a methodology is a bit shit, even if the methodology is good.

Any links you can provide? I would see alternatives to pact.

So, I jury rigged a home grown solution using github actions, pydantic, hypothesis and a library called hypothesis-jsonschema.

I used:

  • Pydantic to generate jsonschema from a python class representing a consumer and a producer.
  • A script in github actions which uploaded this jsonschema file to github artefacts.
  • A script to download it in github artefacts.
  • A 3 line script that ran hypothesis-jsonschema with the two jsonschema files to check that they are compatible.

Using this process I could run bidirectional contract checks on every pull request between any producer and consumer written in any language that could produce jsonschema files for both producer.

3 Likes

If there’s some interest I could upload the code with some examples to github. There isnt a lot of it.

1 Like

World be nice to see a different approach to pact.

I treat it as regression. Same way I would a Unit Test. My method is a public api just like any Endpoint is. If someone changes or breaks my code I wrote, I want to inform them, same way if they break an endpoint they should know.

So after we create a Public Endpoint, we build a contract. Now everyone knows and can be confident that if they introduce a breaking change, they’ll be stopped before they take down the consuming service(s).

1 Like