Day 1 recall
Earlier in the day I was able to remember 6 issues I would want to raise in a debriefing from yesterday’s session but if I were to reproduce them, it would already take a while to repeat the exact steps.
However, today was my last day at this job so it wasn’t very mentally taxing. After coming home, reading through a few articles and running the Day 2 session, I had to think noticeably harder to remember everything I expected from yesterday. After an intensive whole day’s work, a significant portion of the information would be lost.
Day 2 Pen & paper
Software under test: Goodreads recommendations feature
I eagerly customized my Amazon recommendations at some point in the past and thought I’d see what the supposedly social engine of Goodreads comes up with.
The session got heavily derailed by login issues so I did not get to actually explore the target feature but I kept it at 20 minutes to be able to compare the amount or information.
Notes: I tried to be diligent and got about half an A5 page of notes (the size I write, it’s a reasonable amount of text for 20 minutes).
I did not record timestamps although at a few places I would if there was a DB/logs to be checked. I did look at the clock though to ensure I don’t overstay.
Recall: We’ll see about that tomorrow but am not very happy. I unconsciously rely on the captured notes and before reviewing them, only have a few specific memories from the session. I captured 8 points to be raised in a debriefing.
(Note: This is an unpleasant aftereffect of heavy context switching in a past job, I have little problem starting over with a blank slate but unless I take - and review - notes or consciously focus like yesterday, information flows in and then right out of my head. That’s a big motivation for me to find a recall-effective way to take notes and to participate in this experiment.)
Listing the elements of a screen was boring and a waste of time but I know from experience that if I don’t do that, I’ll have gaps when/if retreading my path.
On the other hand there was at least one situation where I rushed ahead focused on the final result and missed an opportunity to check something. Something I need to watch out for regardless of note-taking.
I can scan the notes but with the way I write, they will not be OCR-able and thus not searchable.
This is my go-to method but I have plenty of room for improvement.
My notes are structured and marked with visible symbols so I can quickly return to important findings. Symbols/PQIP markers also support “perspective-based” overviews - these are things to praise and look for consistency with, these are questions to look up, these are potential charters etc.
Still, I know more visual elements could help me retain the information better, and it takes practice to know what I must write down and what I can leave out. An indented list structure is not the best structure when a reference between a line and one or more (even worse) lines that I wrote down before.
I love how scientific @maos post is and it is really making me think that I should have been more organised for this exercise.
So how did it go?
I actually only did a tiny bit of testing today and it was more “checking” rather than testing, so I had very brief notes and I used time stamps.
The time stamps weren’t that necessary as the testing was fitted in between meetings.
What I did notice is that I need a better structure for my testing notes and should consider some methods for this.
I should have reconsider these examples and techniques from @simon_tomes.
Example of Simon Tomes exploratory note taking using qeek
I love taking hand written notes but I find that my lack of structuring them in the moment means I often transcribe them into electronic format where it is easier to move the content about.
On top of that I noticed that my hand writing is nowadays so bad that I need to transcribe it to be able to read it 4 hours later. How embarrassing! I don’t feel only notes work for me.
What I did consider was taking pictures with my phone to improve my notes and make them more visible.
My basic format for the testing today was noting the task number - time and date at the top of my page.
I then added in the areas I wanted to explore of the application and the type of exploring I may have to do such as using different browsers or accessibility related tasks.
I used ticks and crosses to note what worked as expected and didn’t-t and dashes if I didn’t get to cover the section in the time I head.
So uses of symbols is great for a quick glance.
I did find that some of the issues I had to reproduce during the debrief with the developer as my personal handwritten notes did not work for them. So considering who your audience is, is very important too.
So tomorrow it is all about text editors which I used a LOT in my last place. I will be using Atom tomorrow, but have experimented with Evernote before so may mention that too from memory.
See you tomorrow!
Day 1 Memory.
I thought this would be a big challenge for me as I do struggle to remember things at the best of times. However I found the memory note taking a nice breath of fresh air.
Yesterday I was testing / checking a new database that had been created for our project. I was very tempted at first to note down how I should approach this and then realised nope not today smart ass!
In having to remember questions and issues I had found me a little nervous. I was concerned that I might not remember what I had found, or worse where I had found it. With so many new tables in the database and the relationships to each one I realised I was a lot more focused and my memory of what I was looking at seemed to really come to life. But alas as good as it sounds, I have to let you in on a little secret, I did have to write down some key issues I had found when I was testing. I had so many questions on quite a few tables and I honestly didn’t feel confident enough that I would remember them all. The last thing you want to do is look like you aren’t sure what you are doing, this is where I would find the note taking on paper / text editors etc… a lot of help in backing up your questions / issues.
Overall I actually think it was a great experience. I found my mindset in a different place, instead of going straight to pen and paper I actually interacted with the creator of the database (which I would do possibly at a later stage if I was already in the “zone”). We then ended up doing a lot of pairing and I got an even better understanding of the database set up (I have very basic knowledge of SQL Server).
Would I do it again? I think I would. Its great to change the way of how you think, and your process to what you are going to test, but it wouldn’t be my number 1 note taking method.
On for Day 2 (for me) Pen and Paper and I already cant bloody wait, I have missed my 4 colour pen! lol
Good Luck everyone, I cant wait to read your experiences
Day three! Using a text editor.
I have some practice in this and I always save files with the date format “YYYY-MM-DD - Project - details”
Today I only got to use my editor for note taking on a call and some general notes and ideas.
I haven’t experimented using #hashtags as suggested in the blog but I do do this in evernote for personal things and learnings.
I find it super useful for searching stuff.
I am hoping to have another chance tomorrow to repeat using a text editor for test sessions.
I may use headings to separate my bullet points, and use some symbols for easy scanning.
Comment on Paper and Pens.
Today I also got to make some notes on pen and paper and instead of using every line I used every other line which let me add in comments or test data ideas or so below each point as that data became available. I like that little hack even if it initially uses more paper.
Day 3: Only use a text editor
Today’s session was on a two webservices, one of them returning quite a complex structure of items. I chose this text based test item for today’s challenge, because on graphical user interfaces I like to take screenshots to highlight issues. At least this would not be required today.
I used Notepadd++ which is the text edtitor I normally use. I guess the features I use are mostly available in other editors as well (like Pretty Print for copied responses).
As I am stil making my first steps with exploratory testing my notes are not very structured. Actions I performed were just plain text, and I used and and for further notes. Thinking about it maybe one could define some shortcuts to insert those tags easily and define a custom language to automatically add colours based on syntax highlighting. I guess this would make my notes easier to scan. Has anyone tried this? For now using just a text editor feels not good enough to dig deeper on this.
Furthermore, as I do not have a structured way of using exploratory testing my notes currently are just temporary personal notes. So I have not yet tried to search through them. Maybe at some point in time I will find value in having a searchable log.
Based on the test items chosen today it was not an issues, but with my pen and paper notes I like to draw connecting lines, icons and the full area of the paper. All this is difficult with just a text editor.
Day 2 - Pen & Paper
Halle-fucking-lullah!!! I felt more like myself. Nothing like testing old school with pen and paper to make you feel like you are getting somewhere. I have an A4 notepad, and a 4 coloured pen which I bloody love! How can you not like Blue, Black, Red and Green!!
So yesterday’s session I was still looking at the wee databases and chose not to mindmap out my session on paper. I went for a more bullet point kind of structure. I have a colour code for my notes which I find (for me) very helpful. I love writing in blue so thats the colour I tend to use. I use red if an issue / bug / question has been found, Green if all has gone well, and then I use the black for actions I need to go over and things that may be important, depending on importance I will whip out the yellow highlighter .
With yesterdays experiment I felt heaps more confident in my testing, I didnt have to worry about solely remembering what was tested, found and any other items. I found myself a lot more eager and ready to delve straight into what I was looking at.
I did notice was that I didnt talk as much as I did on Day 1. Now I am not sure if that’s because Day 1 was where I was getting shown the database set up and having to talk through what I was seeing. I did find myself writing notes down and telling myself I will ask about those later.
I do prefer note taking, it gives me the option to draw out what I am trying to test, see if there is a flow, brainstorm, brain dump, and jot down my on the spot thoughts.
As much as I do like note taking I have found in the past it can be a pain in the ass to find certain notes that had been taken down at one time or another. I have yet to find a better way to save them. I have taken photos of them in the past and put them on something like Confluence, but even at that, they can be hard to find. For now I will do like I usually do, save all the notebooks and hope I can flick to that particular page. Good thing for me, I always time stamp my notes, regardless of what I am writing down. phew
Day 4 - Use a text editor and a screenshot tool
Here are some thoughts on the session I did yesterday using Word and Hardcopy, which I use to just take a screenshot of the relevant area. So I stored the images right next to my notes. This went pretty well, maybe because I focussed on consistent presentation of data.
So whenever I found something that was presented inconsistently I briefly described the inconsistency and added two or more screenshots to illustrate it. When selecting the relevant area for the screenshot I tried to make it obvious on which screen exactly this has been taken. Maybe due to the focus of the session I did not use any editing of the screenshots, because I think the issues were pretty clear when having a look at the notes and the different screenshots. Whether this holds true I will find out when going through the notes with someone else.
When I want to highlight aspects in a screenshot I typically use good old paint. It is sufficient for me to add text, circles, lines, … in different colours.
Overall, yesterday’s session was a good one and I am pretty confident the notes will be useful even when reviewing them quite some time later.
Day 3/4 -
I again did not get to spend much time testing yesterday but what was interesting is that my new colleague uses word and the print screen function to record her testing notes.
My main observation from it, is that I find it really hard to read over her notes and see what she was doing as her notes are very sparse. It is mostly screenshots that are not annotated.
She wants me to show her mind mapping today using Xmind, so I may not get to explore the text editor way as much as I would like.
Have you shown your notes to others as you went along? Can they read them?
Day 3 - Using a Text Editor
Man I did not enjoy this at all!! I used notepad ++ as my tool of choice. I am familiar with how its used and I mostly use it when editing data files and the like. As a noting tool for me, I found it hard. I found trying to organise my notes on the page a little difficult whilst trying to test. I am used to the freedom of paper to be able to make my notes where ever on the page and to add to them as I go through my testing process. The one thing I did notice was I felt I didnt write down as many notes as I had the previous day with paper, but this time I felt I was talking more to my colleagues about various questions I had.
Its funny, I am looking at my notes again this morning from yesterday and they dont look that bad. I can read what I had tested, I can clearly see the issues I had whilst testing, and I feel like they still make sense. In comparison to my paper notes, there are times where I am like wtf? why did I write that! lol
I think the reason I found using a text editor a little more difficult on this experiment was that I was constantly looking at the screen and what I was testing. When writing down on paper I give myself a wee break from the screen and what I am looking at. I have found that a few seconds of distraction can actually help in my thinking of what I need to do next. I didnt feel the same when typing out on notepad.
I guess its like most things in life, you use or do what suits you better.
Today will be a good challenge, I am unsure if there will be much testing today, however I will do my best to use a text editor and a screen shot capture tool. I think today I will give a go at Word (I bloody hate it), it will be easier to make the notes I hope and place the screen shots as I take them.
I used my notes to go through the issues and questions both with the text only and the text+screenshot version. I discussed the text only version with one of the developers and the text+screenshot version with product management.
The notes were maybe just enough for that purpose, but there were questions. I guess just sending them out would not have been enough. And I actually noticed that my notes were not even perfect. One of the issues I listed was not really in scope of the mission so I could not figure out the exact issue directly. But overall it was fine.
Day 5 - Record the screen and Talk as you test
Wow, that was intense. I already experienced what it is like to talk while you’re testing at a workshop for five minutes. Now recording it and doing this for about 15 minutes was kind of hard. As I work in an open space environment I decided to do it at home, but we would have enough rooms where I could do this without being disturbed or disturb someone.
First observation: It is very unusual for me to talk that much. I guess it slowed my down, which is good and bad. I did not manage to do as much as I wanted to, but I guess it helped to increase transparency and make clear the issues I found. So whenever I found something I directly tried to reproduce it and explain it again. I hope this will make my concerns easy to understand.
When I listened to my recording it did not feel good. I guess many people have troubles with hearing their own voice - so do I. And I could hear myself breathing. And I realized that I could not articulate my thoughts as good as I would like to. Yes, it is a uncut version and my first experience with it, so I should not compare it to podcasts or recordings like this e.g. done by Alan Richardson on youtube. All in all, I am not sure what to do with the results.
As there were some kind of regular issues I think it is fine to just share the whole recording. Just for communicating the issues this might not be the best solution in all cases. Having to watch e.g. 30 minutes just to understand 2 issues might not be sufficient. So maybe it would be good to take notes on issues with e.g. pen and paper in order to help your audience to find the most relevant part of the recording.
I really like the recordings of Alan Richardson’s sessions, because you can learn new perspectives on exploratory testing. So for making your testing transparent to others has more benefits than just getting the issues communicated.
My summary after 5 days: Until now I think there is no “one approach fits all” solution to note taking. Depending on the test item and scope of the session different ways seem beneficial. And my feeling is that a combination of tools for note taking might be best. So I am looking forward to week 2 of this experiment to get an impression of more approaches to note taking.
You mention just emailing the notes not being sufficient and I often find that the notes help me tell the story of the user I was envisaging. So for the debriefing actual interaction like a call or face to face tend to be needed
This is inspiring! I couldn’t do this one but I am wfh on Thursday so will try this on Thursday.
I have a feeling I am going to come across similar findings, such as the talking. I am definitely a thinker. Thanks again for sharing!
So Day 5 - Record the screen and Talk as you test
I didn’t manage to do this one today but have bookmarked thursday for this.
For today I skipped to day 6.
6 - Use a tool designed for exploratory testing
I used Rapid Reporter as my manager mentioned using it successfully and I am pretty sure I have seen Vernon use it successfully when we worked together.
Initially I struggled a bit as it contains different sections and has in a sense shortcuts.
What I loved was that it wasn’t invasive and does really allow you to just keep going and make notes.
Not being able to edit them was hard as I often write short bullet points and press enter which submits a section to rapid reporter.
I liked that you get an excel spreadsheet at the end that you could filter. I didn’t get on with the screenshots being in a separate folder. I am still pining for mind maps I think. But I want to try another of these tools to see as I think they can be really powerful especially in a place where evidence is key and being able to filter and sort testing notes could potentially be really powerful.
Thank you for your feedback.
Some more thoughts on recording my testing: I used OBS Studio, which was a recommendation of a teammate. We sometimes use it to record the actions taken to reproduce a bug. Initial setup was not very intuitive, but with his help done quite quickly. Now everything is set up, adjusting to screen resolutions etc is easy to do.
Watching my testing again with the knowledge of what would come later was a good experience. I noticed that I actually missed an issue the first time it was visible. During the session I recognized it many minutes later. So I guess this exercise helps to reflect, to learn, and to be more observing. And of course sharing the recording with others offers a learning opportunity for them. But I guess this is some kind of big step to take.
As I am not falimilar with any of the tools designed for exploratory testing, this challenge did not fit into my schedule yesterday. But I wanted to try TestBuddy and hope I will do this later so that I have some time to get an idea of the tool. For now I directly jump to
Day 7 - Use a mind map
I used xmind today. And I had some kind of plan. And… it did not work out
I chose a charter bringing me to possibly all the areas of my application under test, so I started to document the areas I touched. I guess a mind map is a good tool to document the hierarchy of an application. But pretty soon I actually forgot to document.
There were just a few issues, but I noticed behaviour I did not ever observe before. So my charter actually has allowed me to learn something new about the appiclation.
As my charter most likely did not require me to take screeshots (this is what I thought), I started to store the screenshots on issues not really related to my charter in different windows of Paint. It actually did not come to my mind to just paste a screenshot into the mindmap.
So overall, after the very strange but in retrospective very positive experience with day 5, today’s session was some kind of disappointing. It made two aspects very obvious to me:
- I am not yet happy with my exploratory testing and esp. the note taking part. I guess I focus too much on documenting issues.
- “Over time, learn the features of the tool.” I really should follow this advice given by Alan Richardson in his post.
Have you seen the bullet journalling idea to have an index at the very beginning of the notebook and then numbering pages and entering them into the index @ejaykay? It’s a simple trick once you think about it - well done to the guy who did it first - but makes things so much easier to find, and you can name the collections whatever you want and index under multiple headings like in a book.
So much this!
Wow, I got distracted for a few days (so much for organized @punkmik but thanks ) - so much to catch up on now, I love how much you folks reflect on the techniques!
Day 2 recall
The good part of taking a break is that I could see how pen&paper fares for recall after more than a day. It worked quite well but I expected that, this being my primary method. However there is a downside to that - I used a lot of my own shorthand conventions and so the notes would be pretty rubbish to anyone else. Due to my handwriting I need to type up the most important takeaways anyway if someone wants them written but there is a lot of room for improvement.
Also even one additional color would be helpful for better highlighting crucial points.
Day 3 Text editor
Software under test: Notes feature of the Vivaldi browser
I did not get as much done in 20 minutes as I would like to but then it’s a very short timeframe and the feature has a lot of interesting nooks and crannies. I can see myself giving Vivaldi another try for online research though.
Recall: Disappointing. Directly after the session I can say where on the screen I was but what I covered and what is left for a next session, or where the issues were, I can’t remember. I’m a bit surprised that I rely on the visual structure of notes so much:
- unlike on paper, I was able to “logically group” findings, entering additional comments under bullet points earlier in the file instead of processing and summarizing the sequence of actions
- not drawing the PQIP etc. symbols resulted in forgetting them as soon as I typed their simplified forms
Invasiveness: High. I needed a second screen to work somewhat comfortably and it felt like it took ages to get anything written down. I type fairly fast and probably it really wasn’t slower than if I tried to write without my usual abbreviations but it felt like a nuisance and I just wanted to go back to the testing.
At least some editors can take care of entering the timestamps with a shortcut key!
Archivability: Potentially excellent, with simple text being a universal format, easy to search, version control, diff and whatnot. Something that would greatly improve searchability and which I never noticed as an issue when writing on paper are conventions. Verb forms, somewhat standardized syntax, symbols - they partially take care of themselves when I write by hand but also aren’t so important as there will be either verbal debriefing or an editing pass.
Overall result: Not at all impressed with the method itself but I was able to formulate several specific points to improve on whatever the medium. Thus, a good day!
Today was use a mind map for me. I’ve been longing for this day!
I use them most days and as we were blocked on the feature we started documenting decisions paths in the code that we wanted to observe later on as well as the other features of the app around time outs and sessions and other quirks. Before I knew it my mind map wasn’t unreadable and it didn’t even contain any screenshots from actual exploring.
I guess as a detailed planning tool I tried to cram too much in. I tried to walk our new tester through my throughts and kept getting lost in the mind map.
In the end I kept the quirkier bits as mission statements to explore with some test data suggestions and created an excel table. Like a decision table (is that the name?) for conditions and if something should occur or not. This then was super easy to see at a glance under which circumstances we should see the new feature or not.
Sometimes a mind map is not the right tool. I thought it would always be! Haha!
For general exploring using mind maps I tend to have a simple template to start with containing missing statement in the middle, with a branch for set up and test data. One for questions, one for issues, and then execution ones. I tend to break these up into scenarios. I hope that makes some sort of sense. If I wasn’t on my phone I could attach it. I may try tomorrow.
For collaboration I used to really like mindmup but we can’t use that now.
Funny that on the day of mind maps I actually created a table in excel!
Day 3 recall
Good. The notes being grouped rather than ordered chronologically give a good overview of the session results. They might well make sense to someone else as the ordering also enforced some editing.
Searchability can definitely be improved (free flow rather than a consistent notation), and I’d need to use set symbols rather than just coming up with random signs for less frequent remarks such as “actually very nice but not a common UX decision” or “look up if this is how it’s done these days” as opposed to “look up in product docs” (which I should just have marked as “look up” in both cases tbh).
The typing took too much time in the 20 minute session but for a longer one I could see myself considering it for efficiency. The act of taking a pen in the hand is sort of an important ritual for me though in general so I’m a bit reluctant to have to let go of it… At any rate, a good learning experience.
Day 4 Text editor + screenshots
Software under test: Randomly found memory game at codepen.io
A fairly basic but just right for a 20 minute session piece of software.
Recall: Pretty much the same as with the text editor alone. The screenshots stand out but trying to come up with a searchable and reasonable structure while typing balanced out that gain.
Invasiveness: Like with the text editor alone, high. I needed a second screen to work comfortably and typing knocked me out of the program flow which was less discreet than with the day 3 software. Naming the screenshots in an informative way was also a disruption.
Archivability: Potentially excellent, with simple text being a universal format, easy to search, version control, diff and whatnot. Images, especially with proper naming, are also helpful to recreate the state in question outside of the session. I tried to use conventions for capturing actions and results this time - it made things easier but for real use, I’d rather prepare a standard dictionary so I could be sure the notes will be readable to someone else than today’s me.
Overall result: Not as bad as the text editor alone. I might get used to it but would need a better concept framework for the notes.
Also I think I’ll use Codepen for finding the test subjects from now on, this is in fact what blocks me most in this challenge