How testers can help with acceptance criteria

(Heather) #1

Recently I was asked:

Do you have any good reads regarding testers helping flesh out user story acceptance criteria?

I couldn’t think of any articles or blog posts off the top of my head so I went with advice based on my own experience:

In my own experience, a lot of it is asking questions of others: how does X fit with Y?
If we add Z do we have a DB table that links that to A? Over time then you start to learn the entire system front to back. Being new to the whole team is an even better place to be because you’ll see things others haven’t or they will have to explain them to you when they might never have had to do so before.

This didn’t feel like a very complete answer to me. I thought it was okay but not fantastic.

Do you have any good reads for how testers can help with fleshing out acceptance criteria? Or what advice would you give to someone who wants to understand how testers can help flesh out user story acceptance criteria?

(Augusto) #2

I wrote about one possible answer to the question a few years back.

(Stefan) #3

Could you clarify what you mean by user story acceptance criteria?
What’s the scope of it? Is it to be used by someone to decide if the story is acceptable? Who’s that person/team. Why would you care about his/their opinion?
Is it a guess of the development team of what it would be acceptable to the client if given the product?
Why not build them with the client/stakeholder - or whoever is paying for the software and expects those product features that make the story delivered? Interview him…see what’s acceptable to him…

(Paul) #4

Probably the greatest service testers can provide is to study the text for logical incongruity, vaguely or ambiguously defined statements and incompatibility with current business processes or even common sense.

(Chris) #5

In a good environment that understands acceptance criteria (that they are insufficient, coded guidelines and not an abstract interface to checking tools as a replacement for thinking) I find that a few things help:

  1. Understand the story. You are no good if you can’t decide on the value of your input.
  2. Keep and refer to risk catalogues. What are the usual risks on your project? What’s gone wrong recently? Do they apply here?
  3. Keep and refer to strategy models. I like the HTSM, or usually a cut-down version of it that’s more applicable to my project and team.
  4. Look at the story BEFORE you discuss it in a group, wherever possible. Test the story, come up with risks ahead of time and establish questions you have. Then you won’t waste anyone’s time, plus you’ll look thorough and super smart.
  5. Make testability a goal of a story. A story isn’t complete until it’s testable. If you need generated test data, or access to a third-party system, or an interface to hook in a test tool like an automation suite, or a log file for observability then you need to ask for it and make sure it gets done.
  6. Apply critical thinking to the story, models and conversation. I like Huh? Really? So? And? for this.

Another tip is to use explicit models. Have a diagram with you or get someone to draw one, then you can test that. What does this box mean? What does this arrow mean? What if I remove this? What’s missing from this diagram? This helps explore people’s assumptions about what everyone else thinks and understands, and what nobody has thought of yet.

If you keep good notes and models handy you can act as a body of knowledge on product risk, testability, historic problems and potential catastrophe. Then people will be all like get a load of this person being all good and stuff.

(Joerg) #6

We have the 3-amigos rule (means: Code Developer, Business Analyst/Product Owner and a Tester) are starting the thinking process of acceptance criteria.

The tester often has a clear view on how to get things unanmbigiously, simple and clear described, so they really request the knowledge of the tester. Also testers thinking more about “negative” tests and is asking often the question how we want to handle these excpetions because these situations also need to be nailed down in acceptance criteria.

(Heather) #7

So in my own instance (I can’t speak for the person who asked me this), the acceptance criteria would be a grouping of acceptable to:

  • User of the product
  • Customer of the product (person paying is not always the person using)
  • The mathematicians within our company (some things you should not be able to do mathematically/statistically and this is not always known by developers)
  • Regulatory bodies (in our case the FDA particularly)
  • Developers and testers on the team, often times what the above people want needs to be designed in a way that is developable and testable

In my own case, the developers and testers sat with users, customers, mathematicians and people who knew FDA guidelines to flesh out their desires for features. We would then break those down in to smaller tasks (away from those stakeholders) with acceptance criteria that covered “acceptable to all of these people” and have a final review with those stakeholders. Once the feature was developed then, the other stakeholders would compare it to what they had originally defined as acceptable.

Our product was incredibly complex so acceptance criteria were time consuming and not all stakeholders time was easily available to craft the acceptance criteria together.

(Stefan) #8

I would call acceptance criteria any kind of request related to what the product should/shouldn’t be, and should/shouldn’t do - from the stakeholder. If your built product matches all interpretations of that criteria, it might be accepted by or acceptable to that stakeholder.

From your description I read that you interpreted some people’s desires.
Then you made your own criteria that would match at least partially your understanding of the interpretation of other people’s desires.
Then you develop and test based on that criteria.
Why would you call your criteria - acceptance criteria?
Are you sure that you won’t build the wrong stuff if you follow your own criteria?

Why not get in contact with the stakeholders:

  • after each small piece of the product is built and demo/workshop
  • have weekly calls/meetings or even more often if you can
  • have a direct line of contact - calls, emails
  • in order to check with them what you might have missed, or clarify questions
  • and then adjust the criteria you know - repeat often…even during the same sprint.

(Heather) #9

With the greatest respect to you, that’s an ideal world and in my particular situation, I was VERY far away from an ideal world.

What you describe above would be fantastic, great, ideal but I was not fortunate enough to be in that environment. I have acknowledged that in the original post. I’m looking for better descriptions and situations than the one I was in to use as examples for the person who originally asked me this question.

(Simon Godfrey) #10

How do you folk then manage the process from AC through to the creation and execution of a test.

Do you describe the test(s) you have designed and get them reviewed before creation of the test? Do you visualise the tests with the team before implementation? Or do you design & implement prior to review?