How Do You Think About Test Automation Goals?

šŸ™‹ Questions
1

How do you think about test automation goals?

I know how NOT to think of it. It seems like a bad idea to say something like ā€œour goal is to have 50 testsā€ or ā€œā€¦one test for every jira storyā€.

Iā€™m a QA Manager trying to give good advice. A development team just asked me for advice on setting goals for functional Selenium tests. I feel like coverage is a good goal. But not code coverage. Iā€™m thinking more about, do we have the right Selenium checks? And it seems to me, the right ones are the ones we canā€™t cover with unit tests, API tests, Visual tests, or other cheaper automation approaches. AND the right ones are the ones we can code to be robust, check something critical, and do so reliably without ballooning our test run time span.

Hmmm. Thatā€™s a lot. I canā€™t frame the above as a goal.

7 Likes
2

Hi @ejacobson,

Thanks for sharing. And I appreciate the dilemma/challenge youā€™ve shared.

Have you considered a principles-first approach? As in, start with defining whatā€™s important when it comes to automation and then define goals off the back of that?

For example, I like @friendlytesterā€™s SACRED and TRIMS approaches that inform a set of principles.

2 Likes
3

@simon_tomes , thanks for the great links. I really liked that guyā€™s talks.

Okay, so your answer is: Instead of thinking you need a goal, how about just using principals. I like it. It comes at a time where I am so tired of the concept of ā€œgoalā€, ā€œobjectiveā€, ā€œmilestoneā€, KPI, etc.

3 Likes
4

I would advocate in a similar fashion:
Automation is a tool and needs to be applied context-specific. Much depends on the details, priorities and risks.
I see a rule like ā€œdo X %ā€ not helpful and healthy. In some cases it will be too much, in others to less.

At best the team understands why which automation is where helpful and doable.

1 Like
5

Hey Eric,

The right ones are the ones where the risk you are mitigating presents itself on that UI layer, or your testability dictates that you canā€™t test it any lower than the UI.

TRIMS is the heuristic Mark and I created for Automation in Testing (AiT), which we use to review our automated checks.

Are they targeted, as mentioned to Risk, and layer based on testability. Risk is based on system behaviour we are trying to mitigate. Who is the actor? What are the actions that actor is doing? When testing an API, the Actor is another system wanting to talk HTTP(S). Their actions are to send data/call APIs, there behaviour is to get data from your system, or to give you some. Identifying the actor usually leads to identify the right layer. Then testability detects if you can automate there.

Reliable focuses on them being deterministic. If checks are targeted, they are usually more atomic, aka, small. Less moving parts. Less system traversing. Less data. Which usually means, more robust checks and less flake. We talk about this as Visual Task Analysis with AiT.

Informative. Checks are living documentation. Which means they are very prone to change, and often do detect change. This is exactly what we want them to do. But when that change is detected, we need them to really help us out. We should code/design to be very clear about that intentions, via their name, and codified oracle (assertions). Well named steps and methods speed up investigation. Data gathering for reporting is crucial, screenshots, videos, HTML dumps, json dumps, logging etc. All this helps us understand if we have a problem, and gets the checks back to checking.

Maintainable focusing on making them easier to maintain, to change tooling, data and so forth. Technology moves very fast, so we should design our frameworks to keep up.

Speedy. We need things to be quick. Creating them. Running them. Debugging them. Maintaining them. ā€˜Fixingā€™ them. We talk about Mean Time to Feedback (MTTF) in AiT, as thatā€™s what automated checks give us, feedback and information. The sooner we get it, the sooner we can make decision on more testing/releasing and much more.

So, the goal should be to have TRIM automated checks. If youā€™ve decided that automating there is the right move.

But important thing to remember with goals, is they are fluid, as they should be based on a strategy. What is your team trying to do right now, whatā€™s the problem that led to you thinking you needed Selenium tests, and is automating on the UI the right solution? Problems before Tools, focuses on this. We often jump to the armoury we already have, instead of targeting on the problem.

4 Likes
6

@friendlytester , I appreciate the detailed response. Yep, nice mnemonic. And I especially like the MTTF concept. Itā€™s way sexier than merely saying ā€œshorten the feedback loopā€. Thank you!

1 Like
7

When considering various aspects of individual checks, a heuristic like TRIMS is quite useful.

I would offer an additional perspective that considers the bigger picture: What is the value of your automation? What are you doing it for? If you would be doing automation because it goes with Scrum, then you would basically have no goal for it. It does not help you making choices relating to automation. If you would be targeting 100% regression test coverage you would have a goal, but a rather poor one as it does not relate to business value. Or would the business start dancing on the desks if you should report that you had achieved that 100%?

So my advice is always to start with determining what you are doing it for. A business-oriented main goal for automation will help you make all sorts of decisions. I know of only five of these:

  • Cost: Saving cost used to be a popular goal and is still sometimes the main driver.
  • Quality: What is the expected impact of doing things one way or another on quality?
  • Efficiency: How fast can I bring a single user story to production?
  • Productivity: How much work can we get done in a fixed amount of time (like a sprint or a quarter)?
  • Fast feedback: How quickly can a dev know that things are okay and he/she can pick up the next task?

These goals help both with choosing what you will do next and with choosing how you will do it. Any more detailed objectives (like ā€˜increasing the regression test coverage of module Xā€™) are perfectly fine if they are aligned with the main goal.