10 days of note taking experimentation - Start Mon 23rd

You mention just emailing the notes not being sufficient and I often find that the notes help me tell the story of the user I was envisaging. So for the debriefing actual interaction like a call or face to face tend to be needed :slight_smile:

This is inspiring! I couldn’t do this one but I am wfh on Thursday so will try this on Thursday.

I have a feeling I am going to come across similar findings, such as the talking. I am definitely a thinker. :slight_smile: Thanks again for sharing!

So Day 5 - Record the screen and Talk as you test
I didn’t manage to do this one today but have bookmarked thursday for this.

For today I skipped to day 6.
6 - Use a tool designed for exploratory testing

I used Rapid Reporter as my manager mentioned using it successfully and I am pretty sure I have seen Vernon use it successfully when we worked together.
Initially I struggled a bit as it contains different sections and has in a sense shortcuts.

What I loved was that it wasn’t invasive and does really allow you to just keep going and make notes.
Not being able to edit them was hard as I often write short bullet points and press enter which submits a section to rapid reporter.

I liked that you get an excel spreadsheet at the end that you could filter. I didn’t get on with the screenshots being in a separate folder. I am still pining for mind maps I think. But I want to try another of these tools to see as I think they can be really powerful especially in a place where evidence is key and being able to filter and sort testing notes could potentially be really powerful. :smiley:

1 Like

Thank you for your feedback.

Some more thoughts on recording my testing: I used OBS Studio, which was a recommendation of a teammate. We sometimes use it to record the actions taken to reproduce a bug. Initial setup was not very intuitive, but with his help done quite quickly. Now everything is set up, adjusting to screen resolutions etc is easy to do.

Watching my testing again with the knowledge of what would come later was a good experience. I noticed that I actually missed an issue the first time it was visible. During the session I recognized it many minutes later. So I guess this exercise helps to reflect, to learn, and to be more observing. And of course sharing the recording with others offers a learning opportunity for them. But I guess this is some kind of big step to take.

As I am not falimilar with any of the tools designed for exploratory testing, this challenge did not fit into my schedule yesterday. But I wanted to try TestBuddy and hope I will do this later so that I have some time to get an idea of the tool. For now I directly jump to

Day 7 - Use a mind map
I used xmind today. And I had some kind of plan. And… it did not work out :frowning:
I chose a charter bringing me to possibly all the areas of my application under test, so I started to document the areas I touched. I guess a mind map is a good tool to document the hierarchy of an application. But pretty soon I actually forgot to document. :open_mouth:
There were just a few issues, but I noticed behaviour I did not ever observe before. So my charter actually has allowed me to learn something new about the appiclation.
As my charter most likely did not require me to take screeshots (this is what I thought), I started to store the screenshots on issues not really related to my charter in different windows of Paint. It actually did not come to my mind to just paste a screenshot into the mindmap.
So overall, after the very strange but in retrospective very positive experience with day 5, today’s session was some kind of disappointing. It made two aspects very obvious to me:

  • I am not yet happy with my exploratory testing and esp. the note taking part. I guess I focus too much on documenting issues.
  • “Over time, learn the features of the tool.” I really should follow this advice given by Alan Richardson in his post.
2 Likes

Have you seen the bullet journalling idea to have an index at the very beginning of the notebook and then numbering pages and entering them into the index @ejaykay? It’s a simple trick once you think about it - well done to the guy who did it first - but makes things so much easier to find, and you can name the collections whatever you want and index under multiple headings like in a book.

Also:

So much this!

1 Like

Wow, I got distracted for a few days (so much for organized @punkmik but thanks :wink: ) - so much to catch up on now, I love how much you folks reflect on the techniques!

Day 2 recall

The good part of taking a break is that I could see how pen&paper fares for recall after more than a day. It worked quite well but I expected that, this being my primary method. However there is a downside to that - I used a lot of my own shorthand conventions and so the notes would be pretty rubbish to anyone else. Due to my handwriting I need to type up the most important takeaways anyway if someone wants them written but there is a lot of room for improvement.
Also even one additional color would be helpful for better highlighting crucial points.


Day 3 Text editor

Software under test: Notes feature of the Vivaldi browser
I did not get as much done in 20 minutes as I would like to but then it’s a very short timeframe and the feature has a lot of interesting nooks and crannies. I can see myself giving Vivaldi another try for online research though.

Recall: Disappointing. Directly after the session I can say where on the screen I was but what I covered and what is left for a next session, or where the issues were, I can’t remember. I’m a bit surprised that I rely on the visual structure of notes so much:

  • unlike on paper, I was able to “logically group” findings, entering additional comments under bullet points earlier in the file instead of processing and summarizing the sequence of actions
  • not drawing the PQIP etc. symbols resulted in forgetting them as soon as I typed their simplified forms

Invasiveness: High. I needed a second screen to work somewhat comfortably and it felt like it took ages to get anything written down. I type fairly fast and probably it really wasn’t slower than if I tried to write without my usual abbreviations but it felt like a nuisance and I just wanted to go back to the testing.
At least some editors can take care of entering the timestamps with a shortcut key!

Archivability: Potentially excellent, with simple text being a universal format, easy to search, version control, diff and whatnot. Something that would greatly improve searchability and which I never noticed as an issue when writing on paper are conventions. Verb forms, somewhat standardized syntax, symbols - they partially take care of themselves when I write by hand but also aren’t so important as there will be either verbal debriefing or an editing pass.

Overall result: Not at all impressed with the method itself but I was able to formulate several specific points to improve on whatever the medium. Thus, a good day!

1 Like

Today was use a mind map for me. I’ve been longing for this day!
I use them most days and as we were blocked on the feature we started documenting decisions paths in the code that we wanted to observe later on as well as the other features of the app around time outs and sessions and other quirks. Before I knew it my mind map wasn’t unreadable and it didn’t even contain any screenshots from actual exploring. :joy:
I guess as a detailed planning tool I tried to cram too much in. I tried to walk our new tester through my throughts and kept getting lost in the mind map.

In the end I kept the quirkier bits as mission statements to explore with some test data suggestions and created an excel table. Like a decision table (is that the name?) for conditions and if something should occur or not. This then was super easy to see at a glance under which circumstances we should see the new feature or not.

Sometimes a mind map is not the right tool. I thought it would always be! Haha!

For general exploring using mind maps I tend to have a simple template to start with containing missing statement in the middle, with a branch for set up and test data. One for questions, one for issues, and then execution ones. I tend to break these up into scenarios. I hope that makes some sort of sense. If I wasn’t on my phone I could attach it. I may try tomorrow. :slight_smile:

For collaboration I used to really like mindmup but we can’t use that now.

Funny that on the day of mind maps I actually created a table in excel! :joy:

2 Likes

Day 3 recall

Good. The notes being grouped rather than ordered chronologically give a good overview of the session results. They might well make sense to someone else as the ordering also enforced some editing.
Searchability can definitely be improved (free flow rather than a consistent notation), and I’d need to use set symbols rather than just coming up with random signs for less frequent remarks such as “actually very nice but not a common UX decision” or “look up if this is how it’s done these days” as opposed to “look up in product docs” (which I should just have marked as “look up” in both cases tbh).
The typing took too much time in the 20 minute session but for a longer one I could see myself considering it for efficiency. The act of taking a pen in the hand is sort of an important ritual for me though in general so I’m a bit reluctant to have to let go of it… At any rate, a good learning experience.


Day 4 Text editor + screenshots

Software under test: Randomly found memory game at codepen.io
A fairly basic but just right for a 20 minute session piece of software.

Recall: Pretty much the same as with the text editor alone. The screenshots stand out but trying to come up with a searchable and reasonable structure while typing balanced out that gain.

Invasiveness: Like with the text editor alone, high. I needed a second screen to work comfortably and typing knocked me out of the program flow which was less discreet than with the day 3 software. Naming the screenshots in an informative way was also a disruption.

Archivability: Potentially excellent, with simple text being a universal format, easy to search, version control, diff and whatnot. Images, especially with proper naming, are also helpful to recreate the state in question outside of the session. I tried to use conventions for capturing actions and results this time - it made things easier but for real use, I’d rather prepare a standard dictionary so I could be sure the notes will be readable to someone else than today’s me.

Overall result: Not as bad as the text editor alone. I might get used to it but would need a better concept framework for the notes.

Also I think I’ll use Codepen for finding the test subjects from now on, this is in fact what blocks me most in this challenge :unamused: