Are we beyond the testing pyramid?

I recently asked a question via the Ministry of Testing Twitter account that I’d seen pop up somewhere else. I noted the question down at the time but forgot to note where it was originally asked :see_no_evil:

It’s the kind of question that I think should also be asked here on The Club as we can give longer-form responses to it/have a more detailed discussion about it.

Do you think we’re beyond the testing pyramid?

2 Likes

Gregory and Crispin (2014) said that for Agile there should really be another layer above the pyramid called “Manual / Exploratory”

3 Likes

One of the things often misinterpreted in the pyramid is that it’s not created for manual testing. It’s a test automation pyramid. So adding a manual/exploratory layer makes it a bit weird.

Looking at the responses on Twitter, I see a lot of people talking about the Test Honeycomb. I don’t think that’s the golden egg either. It’s basically just the same as the pyramid, just in another form and with some other names. The concept remains the same. The pyramid was never meant in a way that the bottom layer should have the most tests. It’s just an assumption a lot of people make.

I’ll need to think about my own answer, just wanted to share this already.

2 Likes

The problem with the Test Pyramid is that it basically talks about automated checking
image

Which is just a very small part of testing

image

The Round Earth model seems to give a more comprehensive explanation.

8 Likes

It’s beginning to look like there are probably as many definitions of testing as there are people who’ve written about it.

My own “super strict” definition comes from my time studying formal methods (a very long time ago!). The spec goes in at one end, software is developed, and testing attempts to “prove” the spec by reconstructing it from what the software does and how it does it. Of course, that’s not exactly agile or very efficient. Certainly not pragmatic! :grin:

1 Like

My gripes with the TP…

  1. Its not a pyramid, it’s a triangle of layers
  2. Its focus is on just automation with a lil teeny weeny bit of exploratory

I’m concerned that a lot of places it is used/referenced don’t really understand the real value of testing

4 Likes

Maybe it should be rebranded as just focused on automation. Or it could be turned into an actual pyramid with other triangles too :slight_smile:

I’m also wondering about the specifics of the layers - tooling is getting more and more advanced now, particularly with the UI level. I’m not sure UI-level automation is the anti-pattern to the same extent it was a couple of years ago.

2 Likes

No, it is a useful resource and a good foundation - maybe a refresher version, but we can never be beyond it, or we would lose sight of the basic foundations - just my view…

The origin of the pyramid is actually that it should be used for Test Automation only. It’s just that people started “abusing” it by adding Exploratory to it. So it’s just a case of having people use it the way it should be used.

2 Likes

The test pyramid stills being useful, of course, but it shouldn’t be considered a rule. It’s just a model that helps addressing a problem under a certain context, but everything (or almost everything) is context dependant, so this model could fit in some conditions and cause issues in others.

Regarding the other topic on this thread, I don’t think the test pyramid is incomplete. Its purpose is simple: helping you decide where to focus efforts when adding automated tests (or checks) for something. Anything else is out of the scope of this model and reducing your testing strategy to just this pyramid would be a mistake in my opinion, instead you should consider adding this pyramid to your testing strategy (if it fits).

2 Likes

I interview people who apply to work at my company. One of the questions we ask is to explain the test pyramid. I get different answers all the time. So I guess the question you are asking depends on how you define the test pyramid.

Just in the response from so far I see different people saving different things. Some believe the test pyramid is just about automated testing. Others give different levels. Rarely do I see people talk about why I believe the test pyramid is useful.

Is it about automation? I don’t think so. I started my testing career in 1998. There wasn’t a lot of agile at that time. Almost everything was waterfall. The software developers in my company started adopting agile software development. There was still a throw it over the wall attitude for testing and QA.

I’d get story 1. I’d test it. I’d get story 2. I’d test story 1 and 2. I’d get story 3. I’d test story 1, 2 and 3. After a while I could not test everything. So we started prioritizing stories. The most important stories would get tested first. Out of 500, the QA team might test the top 300 stories but the other 200 stories wouldn’t get tested before we got another release from development. The solution I, and many others, came up with was to automate everything. I was automating everything at the end and maintaining the test suite was difficult. But automating a lot of the stories meant we didn’t have to manually test everything. Automation helped with regression testing. No pyramid involved.

Always wanting to be efficient, I thought about how can I prioritize my testing? Are their tests I don’t need to do? WAIT. There are unit tests. I might have a test that could fail for 7 reasons. I had to gather enough information in the automation to know why it failed. But 4 of those reasons would have been caught at the unit level. So I really only had to code for the remaining 3 reasons the test would fail.

Even better, could I test two classes together and eliminate more reasons my system level test might fail? I could create 2 integration tests. Now there was only one reason my system level test would fail.

This made automating the tests easier. This made debugging failures easier. This made some defects found sooner (fail fast). Then I found out about Mike Cohn and the “test pyramid”. This basically put into words what I had discovered. The simpler the test, the easier it was to maintain (you can find many pictures of the test pyramid that talk about cost and how it goes up as you go up the pyramid). Additionally, if you had a system level test and there were a dozen reasons it could fail then it was time consuming to figure out why. But if you had unit level, integration level, contract level, etc. then you could spread out where the failures occurred.

If you test things in sequence, then you KNOW defects will be found at the unit level before they make it to the higher levels. Defects will be caught at the integration level before they make it to the higher levels. But if you start testing things in parallel or not in a sequential order then you, in my opinion, defeat the purpose of the test pyramid.

So to me, we automate so we can fail fast. The test pyramid is more about making things easier to maintain and debug.

4 Likes
  • It is about test automation
  • It is a good starting point for: explaining automation, starting discussion on testautomation strategy
  • It is a three layer model, don’t complicate stuff. Unit, service, UI. (also for starting reasons)
  • In most cases this is a good approach for keeping costs limited, although stubbern people always will start with UI before anything else is on the table. I always hope they will learn.

Exploratory testing is not a separate discussion for automation, because using the tools for automation in exploring will have much benefits, so if you need UI and API tools (orso) to help you with exploring and learning, that is part of the automation pyramid too.

1 Like

While I do certainly agree that the bulk of the testing pyramid is indeed about automation, that doesn’t mean that exploration (in whatever way, shape or form) doesn’t have a place in it. It has a place, and a very prominent one. It’s just not stated explicitly and it’s definitely not a layer on top in my opinion.

It comes down to how you design your tests and how you come to that. If, for whatever reason, I decide to write an automated test for some feature, I always start with collecting as much information as possible. I sit down with the dev who’s writing (or wrote) the feature, I look at the docs (if it’s documented), I fire up the application and look under the hood and try to see how all the moving parts are connected, I look at the ticket(s), etc. One might say that I start exploring…
Now, I’m going to need all that information so I can create a fast, reliable and meaningful test. And while the outcome of all my actions might come to fruition through code and automation, a large part of getting there comes through exploration. And it’s actually during this exploration phase you’ll encounter most of the bugs and inconsistencies. The automated test that comes after it just makes sure that it won’t happen again. And it doesn’t matter on which layer I want my tests to be. Depending on the test and the outcome we aim for, the test resides somewhere in the pyramid.

So while there are many things that might be wrong with the testing pyramid, I don’t think it being “only” about automation or not including exploration is actually true.

2 Likes

Really like that round earth model.

To me the thing is these are ‘just’ models and are about ways of categorising your testing at a high level. Basically IMHO they’re guides that if you’re doing something way out of line with what they show then it’s worth (re)evaluating your approach.

One thing my career has taught me is for every rule there’s an exception. However the rules (or in this case models) are useful for informing our thinking.

I think that’s fair.

The problem I see with the pyramid, I think, is that it’s so common now that it’s seen as The Way to organise automation (and sometimes the entirety of testing itself).

1 Like

Here we talk about fat or distorted pyramids but yeah I still find it useful for communicating.
You still want to avoid all tests are ui/e2e tests or even all test duplicate each other.

Not at all, is a very valid and simple model that still make sense for test automation. And that is its purpose.
I can understand people having issues with it not being usable to define their test strategy, a test strategy is much more than test automation, and this model is not fit for that, and it was never meant to.

It’s telling something very simple: Have more fast and cheap tests than long and expensive ones. As long as they are providing value at all. You can add more layers to it, if it makes sense for you, but that’s the whole point. In any other way it means that you’re having a slower and more expensive feedback than its probably optimal for a steady development.

But of course this is just a model, an abstraction, and it depends on its context to be valid or useful.

1 Like

For all test related people it would imho be helpful to come beyond it. Seeing it as ONE model, not THE model.
So far I still see many people aren’t.

" It was never a pyramid. It is in fact an iceberg" :grinning_face_with_smiling_eyes:
https://mrslavchev.com/2018/05/31/hindsight-lessons-about-automation-models-of-automation/
I find this two paragraphs very helpful:

  • Inverted pyramid of check complexity and time investment
  • Inverted pyramid of failure visibility
1 Like