Test Bash Manchester 2018: Workshop ET 101 - Resources


(karo) #1

Hi everybody,

many thanks to all attending our workshop on Exploratory Testing 101 at TestBash Manchester 2018 - it was fantastic to see you all engaging so enthusiastically in the exercises and discussions. We had an absolute blast running the workshop with you!

We wanted to follow up providing links and references to the things we talked about, but also further reading, and especially some material on the topics coming out of discussion that we couldn’t cover in much detail. We’ll add this on here in posts by topic:

There is also an exercise for you to practice the things you have learned in ET101:

Please add on - how the exercise went, any questions you have, thoughts and references, things you found useful, things that are yet puzzling. We’d love to engage with you further.

You can also reach out to us directly, here on the Club, in the Ministry of Testing Slack or via twitter: @tbaxter78 / @karostol

Keep exploring!

Cheers,
Tracey & karo (&Dan!)


(karo) #2

Exploratory Testing

We got hands-on here, running two exercises exploring the building, one following a test script and one using a exploratory charter format and discussing the differences. A lot of discussions running the test script were around the test script (limitations, where it didn’t work!) and a lot of the discussions after the exploratory session around the building (what was found, limitations, risks). Go figure!

We used a Charter Format from Elisabeth Hendrickson:

Explore <target>
With <resources>
to discover <information>

We introduced different models of testing that can show the value and advantage of (Exploratory) Testing. Looking at a rough presentation of a Software Development Life Cycle, courtesy of James Thomas:

Black lines: spec to dev, dev builds application, test compare spec with implementation, using a pass/fail script. Red lines: All that’s missing - wondering about unknowns, the user perspective, how appopriate the spec is, capabilities of the application not covered by the spec, biases I as a tester have, the context etc.
Exploratory testing covers both black and red!

We illustrated further how testing focuses on uncovering the unknowns and looking at different dimensions via few more models:

Find those explained in more detail:

And we wrapped this discussion up with a shortened description on testing by James Bach & Michael Bolton:

“Testing is the process of evaluating a product by learning about it through exploration and experimentation”

Resources at a Glance:

… and some more:

What are your favourite resources, models, blog posts, videos?


(karo) #3

Heuristics & Oracles

Throughout the workshop, we’ve looked at using Heuristics and Oracles in Exploratory Testing Exercises.

“A Heuristics is a fallible method for solving a problem or making a decision.” (James Bach)

We use Heuristics all the time - “a rule of thumb” is a Heuristic, as is “gut feeling”. In testing we often use Heuristics to help us come up with different angles and perspectives, to generate test ideas, to discover risks or to determine how we can tell if something is good or bad, right or wrong, desired or a a hindrance - which is where Oracles come into place:

“An oracle is a mechanism for determining whether the program has passed or failed a test.” (Cem Kaner)

In detail we’ve looked at the FEW HICCUPPS Heuristic, which you can use as an Oracle, it essentially asks on consistency in the product. FEW HICCUPPS is a acronym, where each character stands for a consistency angle onto our product - Familiarity, Explainability, World, History, Image and so forth. We discussed if there are other angles and Huib Schoots mentioned Acceptability, while Dan Ashby brought Emotions to the mix.

Heuristics are a real treasure trove, and a lot fun, too. Use, expand, experiment with them! Great ways to get into is by using Cheatsheets or TestSphere cards - have a squint at the resources just below for further links and reading.

Resources at a Glance:

So what are your favourite Heuristics? What helps you in testing?


(karo) #4

Risks

We started of by distinguishing between risks for the product, project, people or business. We then zoomed into risks for the product, and its variables, i.e. properties and purpose of the thing we’re testing and the risks associated with those, using Dan Ashby’s Risk Model:

We looked at the “impact x likelihood” formula as a way to prioritise risk - which also can be a good way to drive test focus. If we don’t know yet how likely a scenario would be, but we know that it would have a catastrophic impact, can we run an experiment to find out?

Likelihodd_Impact

And we introduced some techniques and Heuristics to uncover risks:

  • Ask “What could go wrong?”, “What’s the worst that could happen?”
  • “Negative Internal Press Release” exercise (1, 2)
  • Mike Talks: Oblique Testing
  • TestSphere, especially used for RiskStorming

And had a quick go iterating through risks of a chair!

Resources at a Glance:

… and some more:


(karo) #5

Note taking

Note taking came up as an area of interest out of the discussions - so we made this a focus on the debrief of the last hands-on Exploratory Testing Session (although it could easily be its own workshop!)

For handwritten notes (although you can use the base structure also for electronic notes, using sections or markup) we introduced the Cornell Method:

And introduced practices we use when taking notes or reporting, e.g.

Structure in different sections, e.g.:

  • Charter
  • Environment & Resources
  • Time
  • Question
  • Bug
  • Observation
  • Summary for Dev
  • Summary for PO

When reporting, tell a testing story:

  • What I did and didn’t do
  • What I found
  • The values and risks

Further Resources:


(Tracey) #6

Exploratory Testing - Practice Exercises

Want to practice all you have learned in ET101?

Explore ZTY.PE which is a typing game, Go to ZTY.PE or download the the mobile app and carry out exploratory test sessions. You can use the following example charters, or even better - why not come up with your own!

  1. Explore ZTY.PE
    Using all available options
    To discover everything ZTY.PE can do

  2. Explore ZTY.PE
    With a range of entries
    To discover how the scoring system works

  3. Explore the ‘my stats’& area of ZTY.PE
    With a range of attempts
    To discover accuracy risks in reporting

  4. Explore the ‘load your own text’ area of ZTY.PE
    With a range of tools and format exploits
    To discover security vulnerabilities

  5. Explore ZTY.PE
    Using developer tools
    To discover if you can cheat!

We’d love to hear how you got on. What interesting things did you find? Did you use the example charters or come up with your own? Share your discoveries!