Missing Layer From The Test Automation Pyramid

[friendlytester (Richard Bradshaw)]

I just shared this post on Twitter. I spent yesterday prepping for my upcoming talk at SauceCon about the test automation pyramid and designing an effective automation strategy. I found myself drawn to creating some new visualisations, after studying the geometry of a pyramid. I also learnt the word QUIDDLE (is to busy oneself with unimportant tasks to avoid more important (unpleasant) ones.) go figure

However, the final visualisations are these, I’ve added ‘testability’ to the base of the pyramid.
For me, a successful automation strategy is based on testability, so it makes perfect sense that it’s the base of the pyramid.

What do you think testability brings to the traditional pyramid model?

[friendlytester (Richard Bradshaw)]

Here is a potential exercise you could do based on the first model.

1. Draw your current model
2. Explore your testability
3. Identify some actions to improve it
4. Draw a goal pyramid
5. Continuously repeat!

[devtotest (Joe)]

Hello @friendlytester!

When we do a test engineering assessment, testability is one of the primary dimensions we explore. We look for opportunities to collect information sooner (such as unit tests), methods to improve transparency (logs, database records), and opportunities to influence designs (use of interfaces to facilitate mocking).

Joe

[ola.sundin (Ola)]

I like the visualization of the pyramid. This really allows you to explore the question on how and where you should do test automation and different options that are available to you. When I have doing test automation strategies and test strategies in general one on the most crucial questions is IF you should do the automation or rather to what extent. So while testability certainly one of the more important inputs to that questions it is not the sole foundation. For me the bottom line is always the return on investment. I don’t know how you would incorporate that in your model or if you would but it would be super nice to see how that would play out.

Will try to apply this model right away!

[mrpjones (Paul)]

@friendlytester I like the visualisation of this - can you provide context into the colouring? What do each of the sides represent? We’re looking at things like testability, coverage and visualisations at the moment so this feels potentially very relevant to me.

[mwinteringham (Mark Winteringham)]

like the visualisation of this - can you provide context into the colouring? What do each of the sides represent?

@mrpjones if you’re referring to the coloured pyramids to the sides of the testability base I believe they are different interpretations of the Automation pyramid.

[mwinteringham (Mark Winteringham)]

So @friendlytester I obviously like that you are highlighting testability in this exercise. There is a bit I do feel uncomfortable about and that’s the use of the word ‘Goal’ in the exercise. We’ve both discussed how the Automation pyramid can be used as a heuristic to check the state of your automation, but I worry using the term ‘Goal’ encourages teams to aim for the Pyramid structure as a strategy rather than reflect on the testability information and leverage that inside their own context.

[friendlytester (Richard Bradshaw)]

your ‘pyramid’ bias is getting in your way.
And you probably need to watch the talk from SauceCon.

Their goal ‘pyramid’ may not even resemble a pyramid. It’s a visualisation of the current number of checks against a specific layer, which is what the pyramid model is, however, this one would be tailored to them. It could be it looks like a plus sign, with few UI, 1000s of APIs and few Unit, and their ‘goal’ pyramid is to make the Unit layer slightly bigger by moving some of the API checks.

[gerardmccann (Gerard)]

Here is a potential exercise you could do based on the first model.

1. Draw your current model
2. Explore your testability
3. Identify some actions to improve it
4. Draw a goal pyramid
5. Continuously repeat!

Per the quote above and the discussion of pyramid strategy/goal/bias, is using the pyramid (or whatever shape) to help with test strategy good/bad/other?

I think seeing testability as the base of the pyramid is helpful. More testability == a better/wider base; a better/wider base == more can be built/layered upon it (regardless of shape[s]). Also, it seems intuitive to see testability as the base considering how close to the unit test layer, and thus source code, it is and how critical the software code is to testability - environment/data notwithstanding.

So I guess to look at point 4 above, if we have improved testability and widened the pyramid base, then is it good to have a goal improved shape (pyramidal or whatever) in mind?

[mwinteringham (Mark Winteringham)]

your ‘pyramid’ bias is getting in your way.
And you probably need to watch the talk from SauceCon.

Their goal ‘pyramid’ may not even resemble a pyramid. It’s a visualisation of the current number of checks against a specific layer, which is what the pyramid model is, however, this one would be tailored to them. It could be it looks like a plus sign, with few UI, 1000s of APIs and few Unit, and their ‘goal’ pyramid is to make the Unit layer slightly bigger by moving some of the API checks.

Do I have to watch the video? I already see enough of you

You’re right it is a bias, but unfortunately, others will have that bias too. So to confirm, the idea is you work out from the base and as you learn more you model the current state of your checks / opportunity for checks from the base. If they don’t match the pyramid shape, have a conversation and see why testability might be impacting your automation?

[friendlytester (Richard Bradshaw)]

Impact, perhaps.
But more asking the question, is our current shape the best we can do with our current level of testability? Is the current shape working for us?