What layers do you consider when testing?

“You don’t just gorge yourself on everything that’s out there and then you can test all the things.” – Ben Dowen (@fullsnacktester)

During Ben’s masterclass – Teamwork Makes the Full-Stack Testing, Dream Work – he shared helpful approaches to collaborating with various people to gain a better understanding of the layers to consider testing.

It got me thinking about all the layers you could test. And I’m curious to know, what layers do you mainly focus on in your current context?

How do you decide where to focus?

How wide and deep do you go?

5 Likes

The highest layer by default for the team/project I am working on:

  • If it’s a web app, that would be the web interface - i.e. using playwright.
  • If it’s a microservice REST API then the test would hit the REST API.
  • If it’s a command line app then the test would run the command line app with the various switches, etc.
  • If it’s a library I will test its public facing API.

I will sometimes move down to lower layers, but I do a cost/benefit trade off to decide whether it is a good idea based upon the following criteria:

  • Determinism (i.e. is it less likely to flake?).
  • Speed (i.e. will it be much faster?).
  • Simplicity (i.e. will it be less work to test on a lower level?)
  • API stability (i.e. are the lower level APIs I would surround with tests be more likely to change or have they stabilized?)

For any code I write personally or I have full freedom over I follow this process more or less exactly. For the libraries I write, for instance, I pretty much never venture down below the public facing layer because the top layer is fast and simple enough. For a beast of a web app I might do so.

In a professional context I will sometimes write lower level tests where I don’t follow these decision making criteria but that will be because the team follows some other testing philosophy. It find this usually has negative side effects, but it’s not always worth the argument to convince somebody to adopt a whole new philosophy.

There are some really huge and underappreciated benefits to testing at the highest possible layer. These include:

  • Entire classes of bug can get caught which simply won’t get caught at a lower level.
  • It gives more freedom to refactor.
  • The test more closely resembles requirements which means it is easier to do some variant of BDD / specification by example (even if you use xUnit framework).
  • You can potentially rewrite the entire project in rust without needing to rewrite any tests (this has happened on two projects I’ve worked on).

The trade off, of course, is speed, flakiness and stability of the test suite. I find a lot of developers are too intolerant of a slow test suite and not appreciative enough of the 4 benefits I listed above.

5 Likes

It does appear that perhaps as a community we don’t talk about this enough, when there tends to be a “find the lowest layer to automate at and start there” approach.

Usually you want to refer to architectural documents if they’re available, the PM/EM should have them. That’s how I would go about finding the various layers that needs to be testing, for example there might be some hidden layers that you do not consider initially that might be only obvious after reading them!

Btw where’s the masterclass available at?

1 Like

Hi @okdashcam,

The recording is in the process of being edited for the MoT site. It’ll appear with all the other masterclasses when ready.

I’ll message you when it’s live. It’s a good one!

1 Like

Thanks! I’ll keep an eye out for it!

How would people define as a layer?

Is there a list of generic types of layers to consider?

Or could we come up with a list if one does not exist?

Genuine question from an old new person to testing. :sweat_smile:

1 Like

That’s a good question @rosie . I was implicitly using Layer (object-oriented design) - Wikipedia as my definition with the addition of the outward facing interface of the application.

So, outward facing REST API - top layer (testing at this layer would be an acceptance test or an E2E test)

view layer of REST API app - 2nd top layer - a test at this level would be what Martin Fowler dubbed a subcutaneous test

model layer - 2nd from bottom - unit or integration test

controller layer - bottom layer - unit or integration test

It’s complicated by the fact that layers of one app can serve as higher level of layers of another app - e.g. an android app consuming a REST API. So, an end to end test could be a test that either just hits the REST API or that uses the android app depending on where you consider the boundaries of the “project” to lie.

1 Like

I think it definitely depends on your skill set and ‘role’ with in your team/organization. Being a test engineer, I focus on automation and shifting left. Let me automate the boring stuff so my QA and BAs can focus on user experience and adhoc testing.

I definitely follow the guide lines of the Testing Pyramid. We focus heavily on the bottom layers and work our way up.

If we follow ideas of shifting left. We want our automation as close as we can get to the code. We strive for wanting to test early and often which will help us find issues before deployments. So heavy unit tests, a strong integration suite and a small E2E/UI Tests.

Since our devs smoke test their changes, the QAs will step through the UI and the BAs will step through the workflow to be able to demo it to stakeholders. We generally don’t focus on UI as it’s covered by a number of others. Also, working in the top layer/UI, means we need a lot more resources (The code needs to be deployed in some environment to be tested), this means we’re testing after the code is developed and deployed.

So, we generally try to focus our automation effort on Unit Tests and Integration tests. This allows us to have faster, stable and repeatable tests that run when developers alter any code. This generally catches most app breaking changes.

2 Likes

Here are some types of layers I haven’t seen mentioned which fed me with tools, info, ‘products’, and relationships for better testing:

  • Operations
  • Sales
  • Marketing, Communications
  • Content management
  • Support
  • Product development/business
  • Clients, users
  • C-level and VP-level
  • Product development/software: architecture, infrastructure, solution design, software development.

How do you decide where to focus?
Company vision, higher management expectations, clients/users needs, stakeholder demands, product manager prioritization, actual product implementation, available time and resources, with consideration of the above layers can give an idea of what matters more. Power dynamics change with time, department managers, revenue stream sources, business plan idea proposal owner, etc…so stay connected to people.

How wide and deep do you go?
Being somehow at the bottom of the power chain, I try to get a grasp of what the C level, dep manager, product manager would consider in regards to quality level in relation to the time of delivery. That drives most of my perceived impression of the coverage and depth, where I add my risk analysis, which I translate through a strategy of testing that I would then create to achieve that.

Good news!

Ben’s masterclass is now available.

2 Likes

A lot of my testing (in the broadest sense) has been what I might call ‘V shaped’ recently, due to the complexity of the system I’m working on.

  • It starts with a desired behaviour very connected to what the user wants
  • We test and question the requirements to make sure that they are going to achieve that aim
  • In technical refinements we come up with algorithms that will implement the requirements
  • After these algorithms have been developed, I go as far ‘down’ as I can to test that the algorithm has been implemented correctly - lots of checking numbers and calculations in different cases
  • I check that the requirements themselves hold
  • Then I get into the mindset of a user and make sure the overall behaviour works and hasn’t caused or revealed any other problems

This is only possible because I’m involved from the start of the process and trusted to help with design, and my developers are willing to spend time exposing workings and creating test tools that allow me to see and analyse what’s going on beneath the UI/end behaviour.

1 Like