As part of the Brighton Software testing clinic we just covered note taking. I got completely sucked into this topic and loved the idea of experimenting with my note taking techniques.
From Monday I want to try and do this experiment by Alan Richardson and take a different approach each day and write some pros and cons and how it went. Feel free to join in and share your experience here, even if you just pick one type you maybe have never tried.
Here is the blog I am referencing.
10 Experiments to Improve Your Exploratory Testing Note Taking
I am starting on Monday, but do feel free to start earlier and start sharing.
1 - In Memory Only use your memory to track your exploratory testing. This experiment mainly helps me remember that I need to do more. I now feel very uncomfortable testing with just my memory, but when I started this felt natural. I changed.
I know I will struggle with this as I always note down something. I learn by writing something down in my own words.
Also for exploring, I find it super useful to write down my set up. This way when it comes to logging issues, or discussing anything I have the set up as reference ready and do not need to remember it and hence can concentrate on my actual session.
I also find massive value in writing time stamps so I know when I started the session and donâ't run over the time box.
So in short without notes I will struggle. Iâll see how I get on and if I can justify no notes today.
Have a good day! Looking forward to catching up tonight.
Here is the result of my first experiment:
In one sentence: âIn Memoryâ does not work for me.
I divide required note taking in two areas:
a) Notes on results and potential issues
b) notes on my actions
Notes on results and potential issues
In my session I focussed on aspects that are normally out of my focus and had quite a lot questions and potential improvements. So I really could not remember all the things and after some time I decided to stop the session early so that I donât waste time. In order go through my results later I really need something I can refer to in order just to discuss the relevant things.
I believe that this approach could work if there are not many things you need to follow-up on. But how often is it the case that you do not have a list of more than a couple of items on your list?
Notes on my actions
In my session I wanted to go through various areas of the application with respect to presentation. In various areas I found things that were not consistent or that did not match my expectation under certain conditions. Especially at the later stages when I saw something I was not sure whether the same condition was fine in another area of the application. Did I actually test this condition in that area?
I found myself switching from having a detailed look at one area to checking one aspect in different areas. Somehow keeping track of which aspects in which areas already have been touched sounds like a good idea. Not sure which kind of note taking would be best for this, but I am sure I will get some idea on that during the next days.
I kinda agree with Thomas above. Just in memory does not work for me anymore. I am pretty sure I used to do a lot more without taking notes and the output used to be issues and bugs, rather than any detailed notes.
Nowadays I make a lot more notes and have a lot more discussions rather than conversations through bug reports.
So how did today go?
Pairing in the morning.
I had the morning pairing on a new to me system. This was sort of cheating as my colleague had a big mind map and she wrote some scenarios on a piece of paper and we then executed those in parallel on different browsers.
She then updated the mind map with our scenarios and the output.
I also made some notes in our discussion about he issue and brain storming which turned into a few A5 pages of written notes. Caveat - I am still learning the system and find hand written notes during conversation help me remember this information.
So maybe for pair testing it is not necessary for both of you to make notes?
Afternoon
In the afternoon I worked on a less risky system and I had testing tasks I had written myself in our agile board, but I ended up going in circles and similar to Thomas not remembering if I had looked at something when I saw something odd.
I am pretty sure I wasted time and effort on those tasks and would have been more efficient with some electronic or hand written notes.
So in summary: in memory did not work for me. I felt lost and often didnât know if I had looked at an area already or not.
I also learned that I naturally make notes in my notebook during conversation and especially when I am told about a system or usage of a system. So for learning I like handwritten notes.
So I am looking forward to Day 2 tomorrow:
2 - Only use pen and paper Yup, you use your computer to test, but you make notes on pen and paper.
First, my plan for the experiment. In no way scientific but I wanted some common criteria against which to judge the various methods:
one 20 minutes exploring session per day
Web (at a minimum for days 4 and 5) or mobile applications I have not used before but from a category I am familiar with
for pen and paper, my regular work setup: A5 grid notebook + a single blue fine pen only, plus blank A4 page for day 7
For comparing the usefulness of various methods:
simulated debriefing and reproduction of found issues on the next day to check recall and reliability/completeness of the notes
perceived invasiveness of note taking in the testing flow
archivability
Day 0 In memory
Software under test: Zenkit app for Android
I have not used Zenkit before and itâs been a while since my last digital to-do software but I have a number of experience-based expectations that I used as my oracle. I went with the features path of exploration.
Recall: Weâll see about that tomorrow but even right after the session I am not sure if it was 6 of 7 findings I wanted to share in a debriefing.
However, I did expect it somewhat - I donât focus very well without an anchor such as a pen in hand and visual reminders and I tend to drift off either clicking around without stopping to memorize actions or just daydreaming Techniques such as memory palaces etc. are IMO for permanent knowledge, not a fleeting state of an application.
Reproducing the path I took to get to an issue in a slightly more complex application is a pain. However, to just get the lay of the land and perhaps know what areas to avoid or explore in depth, exploring without taking notes is good enough.
Invasiveness: Too low in fact as I get too distracted / sucked too deeply into actual doing to recall the steps. If I consciously try to repeat in my mind during the session what I want to keep, it has to be the whole set of findings every time and then it breaks the flow (and wastes time).
Archivability: zero until there is a working brain-computer interface which I will not adopt early either.
Overall result: Pretty useless for me if this was testing for real. Might be acceptable if it were 1-2 pinpointed questions that get debriefed right away rather than general exploration.
I had a similar issue as you regarding comparing things and going back and forth and being distracted.
It sounds like a simple table may even be enough to record these conditions and outcomes. and you could just mark each in a colour or with a tick as you see them. Or a decision flow?
Working memory capacity is said to be between 5 and 7 depending on the complexity of items handled in an average young adult, and even if yours is trained or naturally more capacious, it will probably be 10 at most. Then, it is additionally burdened by what you are doing in the moment so not all âslotsâ are available for the things you want to remember.
Iâm not a psychologist or neuroscientist but really donât believe the in-memory method works well for any larger areas or more complex subjects. It may be a neat exercise in memorizing and focusing but at work we deal with so much information already that it feels like an unnecessary burden.
Damn you Kim, this note taking talk has got me addicted to this post and me wanting to take this challenge. I am a day behind but thatâs OK. So today I am going to attempt to use my God awful memory and see what happens. I am actually quite intrigued as I know that what I need to remember will have to be straight to the point and not so complex. It will be weird not writing any notes or updating my mind map. I know I am a creature of habit, and it will take a lot of will power not to go to the pad or Xmind. I also like the fact that I am having to test in a different way mentally, not having to take notes means I need to have a clear idea of what I am doing, and possibly be more focused in one area, if that makes sense! lol Fingers crossed it goes well! lol
Earlier in the day I was able to remember 6 issues I would want to raise in a debriefing from yesterdayâs session but if I were to reproduce them, it would already take a while to repeat the exact steps.
However, today was my last day at this job so it wasnât very mentally taxing. After coming home, reading through a few articles and running the Day 2 session, I had to think noticeably harder to remember everything I expected from yesterday. After an intensive whole dayâs work, a significant portion of the information would be lost.
Day 2 Pen & paper
Software under test: Goodreads recommendations feature
I eagerly customized my Amazon recommendations at some point in the past and thought Iâd see what the supposedly social engine of Goodreads comes up with.
The session got heavily derailed by login issues so I did not get to actually explore the target feature but I kept it at 20 minutes to be able to compare the amount or information.
Notes: I tried to be diligent and got about half an A5 page of notes (the size I write, itâs a reasonable amount of text for 20 minutes).
I did not record timestamps although at a few places I would if there was a DB/logs to be checked. I did look at the clock though to ensure I donât overstay.
Recall: Weâll see about that tomorrow but am not very happy. I unconsciously rely on the captured notes and before reviewing them, only have a few specific memories from the session. I captured 8 points to be raised in a debriefing.
(Note: This is an unpleasant aftereffect of heavy context switching in a past job, I have little problem starting over with a blank slate but unless I take - and review - notes or consciously focus like yesterday, information flows in and then right out of my head. Thatâs a big motivation for me to find a recall-effective way to take notes and to participate in this experiment.)
Invasiveness: Medium.
Listing the elements of a screen was boring and a waste of time but I know from experience that if I donât do that, Iâll have gaps when/if retreading my path.
On the other hand there was at least one situation where I rushed ahead focused on the final result and missed an opportunity to check something. Something I need to watch out for regardless of note-taking.
Archivability: Low.
I can scan the notes but with the way I write, they will not be OCR-able and thus not searchable.
Overall result:
This is my go-to method but I have plenty of room for improvement.
My notes are structured and marked with visible symbols so I can quickly return to important findings. Symbols/PQIP markers also support âperspective-basedâ overviews - these are things to praise and look for consistency with, these are questions to look up, these are potential charters etc.
Still, I know more visual elements could help me retain the information better, and it takes practice to know what I must write down and what I can leave out. An indented list structure is not the best structure when a reference between a line and one or more (even worse) lines that I wrote down before.
I love how scientific @maos post is and it is really making me think that I should have been more organised for this exercise.
So how did it go?
I actually only did a tiny bit of testing today and it was more âcheckingâ rather than testing, so I had very brief notes and I used time stamps.
The time stamps werenât that necessary as the testing was fitted in between meetings.
I love taking hand written notes but I find that my lack of structuring them in the moment means I often transcribe them into electronic format where it is easier to move the content about.
On top of that I noticed that my hand writing is nowadays so bad that I need to transcribe it to be able to read it 4 hours later. How embarrassing! I donât feel only notes work for me.
What I did consider was taking pictures with my phone to improve my notes and make them more visible.
My basic format for the testing today was noting the task number - time and date at the top of my page.
I then added in the areas I wanted to explore of the application and the type of exploring I may have to do such as using different browsers or accessibility related tasks.
I used ticks and crosses to note what worked as expected and didnât-t and dashes if I didnât get to cover the section in the time I head.
So uses of symbols is great for a quick glance.
I did find that some of the issues I had to reproduce during the debrief with the developer as my personal handwritten notes did not work for them. So considering who your audience is, is very important too.
So tomorrow it is all about text editors which I used a LOT in my last place. I will be using Atom tomorrow, but have experimented with Evernote before so may mention that too from memory.
Day 1 Memory.
I thought this would be a big challenge for me as I do struggle to remember things at the best of times. However I found the memory note taking a nice breath of fresh air.
Yesterday I was testing / checking a new database that had been created for our project. I was very tempted at first to note down how I should approach this and then realised nope not today smart ass!
In having to remember questions and issues I had found me a little nervous. I was concerned that I might not remember what I had found, or worse where I had found it. With so many new tables in the database and the relationships to each one I realised I was a lot more focused and my memory of what I was looking at seemed to really come to life. But alas as good as it sounds, I have to let you in on a little secret, I did have to write down some key issues I had found when I was testing. I had so many questions on quite a few tables and I honestly didnât feel confident enough that I would remember them all. The last thing you want to do is look like you arenât sure what you are doing, this is where I would find the note taking on paper / text editors etc⌠a lot of help in backing up your questions / issues.
Overall I actually think it was a great experience. I found my mindset in a different place, instead of going straight to pen and paper I actually interacted with the creator of the database (which I would do possibly at a later stage if I was already in the âzoneâ). We then ended up doing a lot of pairing and I got an even better understanding of the database set up (I have very basic knowledge of SQL Server).
Would I do it again? I think I would. Its great to change the way of how you think, and your process to what you are going to test, but it wouldnât be my number 1 note taking method.
On for Day 2 (for me) Pen and Paper and I already cant bloody wait, I have missed my 4 colour pen! lol
Good Luck everyone, I cant wait to read your experiences
I have some practice in this and I always save files with the date format âYYYY-MM-DD - Project - detailsâ
Today I only got to use my editor for note taking on a call and some general notes and ideas.
I havenât experimented using #hashtags as suggested in the blog but I do do this in evernote for personal things and learnings.
I find it super useful for searching stuff.
I am hoping to have another chance tomorrow to repeat using a text editor for test sessions.
I may use headings to separate my bullet points, and use some symbols for easy scanning.
Comment on Paper and Pens.
Today I also got to make some notes on pen and paper and instead of using every line I used every other line which let me add in comments or test data ideas or so below each point as that data became available. I like that little hack even if it initially uses more paper.
Todayâs session was on a two webservices, one of them returning quite a complex structure of items. I chose this text based test item for todayâs challenge, because on graphical user interfaces I like to take screenshots to highlight issues. At least this would not be required today.
I used Notepadd++ which is the text edtitor I normally use. I guess the features I use are mostly available in other editors as well (like Pretty Print for copied responses).
As I am stil making my first steps with exploratory testing my notes are not very structured. Actions I performed were just plain text, and I used and and for further notes. Thinking about it maybe one could define some shortcuts to insert those tags easily and define a custom language to automatically add colours based on syntax highlighting. I guess this would make my notes easier to scan. Has anyone tried this? For now using just a text editor feels not good enough to dig deeper on this.
Furthermore, as I do not have a structured way of using exploratory testing my notes currently are just temporary personal notes. So I have not yet tried to search through them. Maybe at some point in time I will find value in having a searchable log.
Based on the test items chosen today it was not an issues, but with my pen and paper notes I like to draw connecting lines, icons and the full area of the paper. All this is difficult with just a text editor.
Halle-fucking-lullah!!! I felt more like myself. Nothing like testing old school with pen and paper to make you feel like you are getting somewhere. I have an A4 notepad, and a 4 coloured pen which I bloody love! How can you not like Blue, Black, Red and Green!!
So yesterdayâs session I was still looking at the wee databases and chose not to mindmap out my session on paper. I went for a more bullet point kind of structure. I have a colour code for my notes which I find (for me) very helpful. I love writing in blue so thats the colour I tend to use. I use red if an issue / bug / question has been found, Green if all has gone well, and then I use the black for actions I need to go over and things that may be important, depending on importance I will whip out the yellow highlighter .
With yesterdays experiment I felt heaps more confident in my testing, I didnt have to worry about solely remembering what was tested, found and any other items. I found myself a lot more eager and ready to delve straight into what I was looking at.
I did notice was that I didnt talk as much as I did on Day 1. Now I am not sure if thatâs because Day 1 was where I was getting shown the database set up and having to talk through what I was seeing. I did find myself writing notes down and telling myself I will ask about those later.
I do prefer note taking, it gives me the option to draw out what I am trying to test, see if there is a flow, brainstorm, brain dump, and jot down my on the spot thoughts.
As much as I do like note taking I have found in the past it can be a pain in the ass to find certain notes that had been taken down at one time or another. I have yet to find a better way to save them. I have taken photos of them in the past and put them on something like Confluence, but even at that, they can be hard to find. For now I will do like I usually do, save all the notebooks and hope I can flick to that particular page. Good thing for me, I always time stamp my notes, regardless of what I am writing down. phew
Day 4 - Use a text editor and a screenshot tool
Here are some thoughts on the session I did yesterday using Word and Hardcopy, which I use to just take a screenshot of the relevant area. So I stored the images right next to my notes. This went pretty well, maybe because I focussed on consistent presentation of data.
So whenever I found something that was presented inconsistently I briefly described the inconsistency and added two or more screenshots to illustrate it. When selecting the relevant area for the screenshot I tried to make it obvious on which screen exactly this has been taken. Maybe due to the focus of the session I did not use any editing of the screenshots, because I think the issues were pretty clear when having a look at the notes and the different screenshots. Whether this holds true I will find out when going through the notes with someone else.
When I want to highlight aspects in a screenshot I typically use good old paint. It is sufficient for me to add text, circles, lines, ⌠in different colours.
Overall, yesterdayâs session was a good one and I am pretty confident the notes will be useful even when reviewing them quite some time later.
Day 3/4 -
I again did not get to spend much time testing yesterday but what was interesting is that my new colleague uses word and the print screen function to record her testing notes.
My main observation from it, is that I find it really hard to read over her notes and see what she was doing as her notes are very sparse. It is mostly screenshots that are not annotated.
She wants me to show her mind mapping today using Xmind, so I may not get to explore the text editor way as much as I would like.
Have you shown your notes to others as you went along? Can they read them?
Man I did not enjoy this at all!! I used notepad ++ as my tool of choice. I am familiar with how its used and I mostly use it when editing data files and the like. As a noting tool for me, I found it hard. I found trying to organise my notes on the page a little difficult whilst trying to test. I am used to the freedom of paper to be able to make my notes where ever on the page and to add to them as I go through my testing process. The one thing I did notice was I felt I didnt write down as many notes as I had the previous day with paper, but this time I felt I was talking more to my colleagues about various questions I had.
Its funny, I am looking at my notes again this morning from yesterday and they dont look that bad. I can read what I had tested, I can clearly see the issues I had whilst testing, and I feel like they still make sense. In comparison to my paper notes, there are times where I am like wtf? why did I write that! lol
I think the reason I found using a text editor a little more difficult on this experiment was that I was constantly looking at the screen and what I was testing. When writing down on paper I give myself a wee break from the screen and what I am looking at. I have found that a few seconds of distraction can actually help in my thinking of what I need to do next. I didnt feel the same when typing out on notepad.
I guess its like most things in life, you use or do what suits you better.
Today will be a good challenge, I am unsure if there will be much testing today, however I will do my best to use a text editor and a screen shot capture tool. I think today I will give a go at Word (I bloody hate it), it will be easier to make the notes I hope and place the screen shots as I take them.
I used my notes to go through the issues and questions both with the text only and the text+screenshot version. I discussed the text only version with one of the developers and the text+screenshot version with product management.
The notes were maybe just enough for that purpose, but there were questions. I guess just sending them out would not have been enough. And I actually noticed that my notes were not even perfect. One of the issues I listed was not really in scope of the mission so I could not figure out the exact issue directly. But overall it was fine.
Day 5 - Record the screen and Talk as you test
Wow, that was intense. I already experienced what it is like to talk while youâre testing at a workshop for five minutes. Now recording it and doing this for about 15 minutes was kind of hard. As I work in an open space environment I decided to do it at home, but we would have enough rooms where I could do this without being disturbed or disturb someone.
First observation: It is very unusual for me to talk that much. I guess it slowed my down, which is good and bad. I did not manage to do as much as I wanted to, but I guess it helped to increase transparency and make clear the issues I found. So whenever I found something I directly tried to reproduce it and explain it again. I hope this will make my concerns easy to understand.
When I listened to my recording it did not feel good. I guess many people have troubles with hearing their own voice - so do I. And I could hear myself breathing. And I realized that I could not articulate my thoughts as good as I would like to. Yes, it is a uncut version and my first experience with it, so I should not compare it to podcasts or recordings like this e.g. done by Alan Richardson on youtube. All in all, I am not sure what to do with the results.
As there were some kind of regular issues I think it is fine to just share the whole recording. Just for communicating the issues this might not be the best solution in all cases. Having to watch e.g. 30 minutes just to understand 2 issues might not be sufficient. So maybe it would be good to take notes on issues with e.g. pen and paper in order to help your audience to find the most relevant part of the recording.
I really like the recordings of Alan Richardsonâs sessions, because you can learn new perspectives on exploratory testing. So for making your testing transparent to others has more benefits than just getting the issues communicated.
My summary after 5 days: Until now I think there is no âone approach fits allâ solution to note taking. Depending on the test item and scope of the session different ways seem beneficial. And my feeling is that a combination of tools for note taking might be best. So I am looking forward to week 2 of this experiment to get an impression of more approaches to note taking.
Techniques such as memory palaces etc. are IMO for permanent knowledge, not a fleeting state of an application.