Is test scripting/execution evidence still required?

Questions
#1

Hi there, this is my first post on this site so go easy on me folks!

My questions relate to manual testing that is carried out during an iteration. So, our set up is that we work in an Agile manner(sprints of 2 weeks) and each iteration contains multiple user stories which require testing. The company is based at one site, so there isn’t any off-shore testing/development done

So going back a few years ago where I worked in a Waterfall testing team there would be a lot of work around writing up test conditions(based on requirements) and within those test conditions we’d write a series of steps outlining what needs to be done when carrying out the test execution. Then as part of the test execution test evidence would be taken in the form of screenshots.

The questions that I have are:

  1. In the Agile way of working is there a need to write detailed test scripts, or is can testing be done off the back of the test conditions/acceptance criteria?
  2. In terms of test execution, is taking test evidence necessary in terms of screenshots?
  3. How does exploratory testing tie in with the above and does this replace any of the above?(I’ve just purchased the book: “Explore It” so I’m hoping that might help!)

Whilst I’m getting more and more used to Agile concepts there are still a few things which are unclear to me(namely the above!) However, I don’t want to fall into the trap of going down the wrong path and be seen to be taking shortcuts when this is the last thing I want to do!

Would be good to have some insight from other members to gauge opinion

Cheers

Andy

#2

Hi Andy,

I don’t think the development process really affects the answers to your questions. They’re all dependant on the needs of your team and organisation.

E.g. I worked for a company who used Scrum to develop software but were regulated by the FDA so in that example we needed to work differently than my current company who use Scrum but are not regulated by the FDA.

You need to consider whether you get value from detailed test scripts and what the pros/cons are of an alternative approach. Likewise, with your test execution, do you need a result or do you need evidence of results? Again, this might come down to any QA regulation, internal company process or likelihood of being audited.

You state “shortcuts”, is this because you feel, or have been told, that Agile means less documentation than traditional Waterfall models?

Hope that’s of some help.

Simon.

3 Likes
#3

Andy, for the most part I agree with Simon. In my last role, the company was quite new to formalised testing and I found that having a detailed test script was helpful in that I could theoretically hand the script over to a user from the business and be reasonably confident that they could replicate the test. In my current role, I am with a software house, and although the completed test script is good evidence for the extent of testing that has been done, it does not need to be so prescriptive as we are all (to a greater or lesser extent) expert users of the application.

In a previous contract role, the testing had to be documented keystroke by keystroke and test outcomes recorded in the same way as that was for a company working with medical equipment and so it was subject to very strict regulatory requirements. Test evidence in the form of screenshots was sometimes required in that case; in other roles, not so much (or at all).

As for exploratory testing, similar comments apply. The medical equipment company had little scope for exploratory testing, in part because the end users’ workflow was very constrained (users would not be going ‘off piste’ with the app); my last role didn’t have enough exploratory testing because we found that real world users had a range of expectations and practices that had not been captured at the requirements gathering stage.

In each case, the organisation had its own interpretation of ‘Agile’ and that really didn’t impact on the extent of detail in the test process.

2 Likes
#4
  1. In the Agile way of working is there a need to write detailed test scripts…

No.

You might decide to use test scripts, but generally speaking I don’t find much use for them. It will depend on your context. If you do this you should be able to justify its cost with how these scripts serve your test strategy.

…or can testing be done off the back of the test conditions/acceptance criteria?

No.

Acceptance criteria and test conditions are insufficient to describe what you might test. Besides, what you do is exploratory and what you learn from the tests you do might lead to new risks, new ideas, and new tests. I don’t really know what “test conditions” are, but acceptance criteria are criteria by which whoever authored them thinks that the story cannot be considered done unless they are complete. If all acceptance criteria are met then the story could still not be fit for purpose, or still have problems. This arises from “all criteria not considered” plus all tacit criteria (e.g. “computer should not crash when code is run”). If that doesn’t convince you consider the fact there are infinite criteria. Any criterion you can think of I can say “okay, but what happens if you do that a second later” and it’s a different criteron - maybe that doesn’t matter, but then we’re in the position of guessing what matters given any change in any number of factors. Then combinations of those factors. You get the idea.

There are a few ways to approach testing something when given a story. One way would be to think of risks and then go and see if you can make them happen (e.g. “I think that the software may not handle weird data very well” or “I think that the software may not behave as I’d like when it’s run for a long time”). Another way would be to explore the product considering elements of the product (screens, code, media files, inputs, functions, etc) and thinking of risks that might apply to them. Saying that, use the acceptance criteria to your advantage. They are clearly important to the team, so should be considered when testing. Maybe it’s a suitable candidate to be augmented with a check in an automation suite.

  1. In terms of test execution, is taking test evidence necessary in terms of screenshots?

No.

If you’re going to take evidence consider what you’re doing it for. If you’re not under pressure to prove you did something then don’t waste any time or energy doing it. That being said screenshots can be awesome for communicating a problem (via a bug report or just messaging someone), comparing two states of a screen to look for problems, remembering settings while you work on something else, recording what you did in your test notes if you’re writing test notes, and so on. So use screenshots, don’t let screenshots use you.

  1. How does exploratory testing tie in with the above and does this replace any of the above?

Exploratory testing is a way of describing an approach to testing that leverages its exploratory nature by treating testing as parallel test design and execution, giving short feedback loops between what is learned and what activities happen. That doesn’t explicitly exclude or include any way of recording or reporting that testing. That’ll depend on you, your preferences, what serves your test strategy, what environment you’re working in and so on.

Hope that’s helpful!

4 Likes
#5

My general approach when it comes to Acceptance Criteria is to read them, try and understand them (sometimes depends on who’s written them) and try to understand what the writer of them was trying to achieve when specifying them.
When I’m planning some tests for that story, I’ll then make sure I’m including tests for the AC, plus many others, depending on the scope and complexity of the story.
As a freelancer working on many different projects and with many different teams, it’s always amazing to see how many different ways someone can write a user story and the accompanying Acceptance Criteria. Differences include: no details, not enough details, too much detail, unintelligible story and/or AC, AC spread through the description and comments, elements of the story or AC referred to in other systems which nobody can access except the original story writer - and so on.

#6

Hi Simon, sorry for the delay but I have been on annual leave. Thanks ever so much for your reply

In terms of the test evidence, I’ve found that where it is a relatively small team where everyone is based in the same office when it comes to any queries as a result of test execution there hasn’t really been an occasion where test execution steps have been asked to be verified. We tend to log details of any transaction Id’s for example so that data written to the db for example can be checked back later.

When I used the term “shortcut” it’s more something that I have felt in myself(rightly or wrongly!). By putting this post up I thought it would be good to get advice from other people as to their experiences.

Andy

2 Likes
#7

Thanks for the response Robert, appreciate the detailed response

1 Like
#8

Thanks for the detailed response Chris, there’s a lot of useful information for me to glean from this!

1 Like
#9

Thanks for the reply Pat.

It’s actually reassuring to read your comments regarding the many different ways that someone can write out the tests(rightly or wrongly!) In my last couple of roles I have been the only tester on the agile team so this site is going to provide invaluable in terms of getting different opinions.

Andy

1 Like