Am I understanding the Contract Testing the right way?

🙋 Questions
21

@al8xr and @prescottlewis707 thank you both for replying on this thread with useful info, I registered for the workshop in February.

22

Contract testing pros and cons. (From my experience working with Spring Cloud Contract)

Pros:

  1. Carefully implemented contracts can minimize the number of end-to-end tests. Most of the scenarios for microservices can be covered using a combination of unit, component, and contract tests.
  2. Correctly established process of handling new and existing contracts (and contract tests) can decrease the number of potential integration issues (or breaking changes)
  3. Test engineers with a programming background can help developers with writing contract tests or at least - defining contracts for integration points (or defining the missing ones)

Cons:

  1. Contract testing tool setup requires time and effort (either from a developer or from a test engineer)
  2. Contract testing process is 95% about communication. So in order to gain benefits from contract tests you need a joint effort from ALL teams. Also, you need support from management - because writing contracts from scratch can take time.
  3. You need to provide additional training for developers (and for testers) in order to implement contract tests. Test engineers in most cases need to have a solid programming background (or good training) in order to implement contracts correctly.
  4. Implementing contract tests takes time. In our cases, it takes at least 1 - 1.5 years from PoC to a fully working process for multiple different departments (hundreds of microservices)
  5. It takes additional effort to cover with contract tests all legacy services.
  6. If you miss unit and component tests (or even basic end-to-end tests) - contract tests may not bring a lot of value in terms of preventing issues.
  7. You need to implement contract tests only if you have integration issues
5 Likes
23

This is fantastic. Thanks Alexander. @marie.drake what are your pros and cons?

3 Likes
24

I think Alexander has summed it up pretty well! But maybe to add a few more:

Pros:

  • It can prevent data providers and/or consumers from deploying broken changes to production via Pact’s can-i-deploy tool
  • It can serve as an additional source of truth and documentation for the team
  • It eliminates the need of having a dedicated test environment with all the integrated services

Cons:

  • Doesn’t work well with public APIs where the consumers are not known
  • Only works best if the APIs are created in house. Doesn’t work well with third party APIs
5 Likes
25

Thanks Marie!!! What are your thoughts @bas ??

1 Like
26

In my experience it is indeed a team effort (I should say company effort) as the contract strategy must be known and acknowledged by everyone (assuming that you are in a micro services world, you must have a contract strategy that is used by everyone).

Simple questions such as are the contract versioned? Who publish them and maintain them? Is everyone using the same tool?

The fact that they are consumer driven is also a change in most approaches I’ve seen, assuming that the consumer is driving the evolution of the contracts with new necessities is different than having the providers deploying new features.

Nevertheless, the concept of validating contracts is actually shifting left the APIs validations and lighten the testing burden further ahead so I will definitely go for it!

4 Likes
27

Very well put @cristiano.cunha - shifting left API validations

1 Like
28

For folks who are fully bought into contract testing, what’s the advantage it provides over existing tests? i.e. if someone breaks a contract, I expect my existing tests to fail.

I can see value if you don’t have the time/resources to write actual tests for the services, and maybe that’s the sweet spot? To provide a minimum safety net?

2 Likes
29

In my experience, contract testing not only provides coverage of your service from the perspective of the actual user of the service. But also improves the communication between users and the service itself, therefore improving the quality of the delivery of the application.

Come to my workshop and find out Introduction to Contract Testing with Pact

2 Likes
30

Hi there!
Contract tests lay somewhere in the middle between unit (or component tests) and end-to-end tests.
On the one hand, they provide quick feedback if integration is broken (almost as fast as unit tests), but on the other hand - in contract testing, you deal with a real integration and not with the assumptions of how something is working.

I described it in more detail in my post

3 Likes
31

If you want to hear about real life experiences with Contract Testing for Microservices checkout my podcast

2 Likes
32

thanks, @mirza for starting this thread on CDCT.

After going through some articles on CDCT, one question came to my mind.

how do you deal with the fact that the mock may become out-of-date if the real provider changes its behavior?

1 Like
33

It’s “Consumer-Driven”.
Just like in TDD, where the behavior is subjected to the tests, in CDCT the providers are subjected to the consumers, through the contracts.

Thus changes in the real provider only happen in accordance to changes in the consumer contracts. A change in the consumer contract triggers a Red, Green, Refactor cycle on the provider.

In this sense, CDTD (also like TDD) is more about design than testing itself - it’s a paradigm of how to develop APIs.

However, there is possibility of Bi-Directional Contracts, which take the definitions of both sides and check their compatibility. You can see details here.

1 Like