Tomorrow we’ll be joined by the awesome @ezagroba for an Ask Me Anything session all about test reporting.
I’ll make sure to add any questions that we don’t get to and any resources mentioned during the session to this thread.
If you miss the live session, a recording will be available on the Ministry of Testing website for all Club level members once we’ve edited it and added captions.
Have you got any more questions you’d like to ask after? Add them here 
How do you bring things together to report on Epics/Larger features when the development and testing is done incrementally over multiple sprints?
For context, I rarely get time to revisit an end to end feature having testing the integrations in sprint, unless I’m doing capacity testing.
What is your approach to test reporting? How do you break it down?
For those, who are working with people used to test reports being number of test cases passed etc, what advice do you have for them?
Do you report on flaky tests?
How do you structure and deliver your reports to get better engagement from viewers?
The Power Hour Elizabeth mentioned Power Hour - Exploratory Testing
Yes, I do report flaky tests.
For flaky automated tests, our nightly pipeline makes it obvious when a test fails. I try to be strict about getting the test passing every time or deleted before standup the next day. To balance the importance of the feature with how much time we’re spending diagnosing it, we’ve also got some tests commented out or running multiple times before failing.
For flaky behavior when testing without automation, structure your report like a cliffhanger murder mystery. You went to the page, you clicked this button, and you’ll never guess what happens next!
My approach: It depends. These examples below are what I do most often, but are not exhaustive.
For a standup, I want to include details that were particularly frustrating so I can find out if I need to keep hacking away, identify someone to help, or have someone tell me to give up.
When I find stuff while testing a user story, I’ll chat the developer directly. Usually it’s something simple (a missing constant, the wrong branch) that’s quickly resolved. Bigger questions that need more input move to the team channel or a video call. If you find something small and weird but unrelated, give your team the gift of starting your message with [not urgent] and a description of what your current context is since they’re probably context-switching. (Bonus points if you get them to pair or do nothing during testing so they’re not switching contexts.)
For a closing comment on a user story, I want to describe what I did well enough for me or someone else on my team to replicate it. Often I’ll start writing the report mid-way through the testing to discover what I forgot to test.
Number of test cases people: I haven’t worked in a context like this. I’m curious if the people asking think that all test cases are the same complexity, or why they’re asking. Dig into what they’re worried about to figure out what needs you should be addressing.
If you’ve delivered incrementally, hopefully you’ve got incremental reports too, with details about how you tested, what you didn’t test, what you noticed while you were doing it. Your past self can help guide you at crunch time.
If you don’t have incremental reports, take Nancy Kelln’s approach.
Yes, this is a great idea. Anything that makes it clear what version/point in time/circumstances you were testing under will help anyone looking at a document in the future.
It depends on your audience: what do they care about? What will make them pay attention?
Unfortunately my go-to reporting tools of mind maps and annotated screenshots don’t pair well.
I would love for this to be true, but it has not been my experience. You can have a really good report that changes nothing if nobody’s listening, or the change that’s needed is outside the influence of your audience.
What my reports look like: see this other response.
How do you organise: Assume the receiver is only half-paying attention. Put the most important information first so they want to pay more attention.
I haven’t used QA Touch or anything like it. I understand the desire to cover all requirements, but I disagree with the premise that all requirements can be stated explicitly. Recommended reading:
No, I don’t have a good answer for this.
I learned a lot about accessibility by reading the whole iOS Human Interface Guidelines when I was on an iPhone project and following the Hack Design course over a year, but I suspect there are more straightforward ways to get this information now.
It depends, but after attending many corporate indoctrinations, I can tell you that graphs that go up and to the right are the most dramatic.
I don’t think these are mutually exclusive. If metrics answer key questions, provide them. The metric I report most often is how many hours I have available for testing activities.
My best answer here is: no. I don’t think most test reports (or software) are needed for posterity. Nobody’s reading them, don’t save them.
Our pipeline saves logs of recent runs. This other question suggested another approach.
@simon_tomes and I wrote about test reporting and audits in a regular environment here.
“Pass” and “fail” sound like things that happen to individual tests, whereas “conditional go” sounds like a software release decision. What’s the worst that could happen if you released this software right now?
- had similar thoughts on this.