At what level(s) of testing do you cover your BDD scenarios?

Imagine this situation.

An application has all its business requirements covered through realistic examples of functional behaviour. These examples live in a BDD framework which serves as living documentation, as a communication tool, and each example is covered by an acceptance test. The acceptance tests consist of browser-driven tests exercising the entire stack. There are 1000 of them taking more than 1 hour total.

This doesn’t allow fast flow.

So maybe instead you define the acceptance tests as the lowest level technical tests possible which can verify the behaviour, using pytest-bdd or RSpec or whatever is appropriate for your ecosystem. There are 1000 of them and while they do not give the extra confidence you’d get from exercising the entire stack, they complete within minutes and are easier to debug.

Now you split into microservices.

So maybe you keep everything you had before, but separate the low level acceptance tests by service, mocking dependencies, and implementing contract testing to ensure the mocks remain in sync with reality. Now you miss that extra confidence of the full stack tests, so you introduce a limited set of complete end-to-end flows representing the most significant paths through the application.

These require running the entire application.

That’s too expensive for a decoupled service. So maybe you split the end-to-end flows off into a separate pipeline. Maybe you run those continuously. But when something goes wrong, you don’t know which deployment caused it and it’s harder for a team to see if they’re responsible. Maybe you are now in the best situation you can hope for?

Or maybe you separate the BDD framework from the tests altogether, using the behaviour examples only for communication and documentation, and have comprehensive tests at the appropriate levels that however do not connect directly to the specifications.

What approach would you choose / have you chosen that optimises for both fast flow and high confidence, whilst keeping the benefit of comprehensive living documentation representing all business requirements?

3 Likes