Mnemonic != Heuristic

🗄️ Archive
1

A discussion yesterday on the Ministry of Testing Slack channel about heuristics and mnemonics resulted in @martin.hynie saying “mnemonic != heuristic”.

Some context for what resulted in this, someone asked if there was a detailed breakdown anywhere for all of these test heuristics.

One of the first replies was a link to mnemonics

Another reply linked to James Bach and Michael Bolton’s blogs where they talk about mnemonics in great detail.

I always thought I knew what the two meant until Keith Klain posted “conflating or confusing those two is a big problem with a lot of testers…”

How would you define a mnemonic? How would you define a heuristic? Do you know of grey areas where the mnemonic could be a mnemonic of a heuristic?

1 Like
2

A mnemonic is a way of remembering something. A heuristic is a fallible way of solving a problem or making a decision.

I don’t like the habit people have of providing a mneumonic as a way of teaching something when the underlying information isn’t apparent. I could teach FEW HICCUPPS to someone, even what the letters stood for, and they’d be no better off. It’s only when you know that “history” means “the current version should be like the previous versions” that it becomes useful.

I once tweeted some mneumonics to satirise this concept.








5 Likes
3

This should be in #rants :smiley:

Mnemonics have never helped me, despite Bolton’s best training. They never stick in my brain context :wink: Heuristics on the other hand are great rules of thumb, with or without mneno… (letter thing).

My favorite: “as soon as I think I need a haircut - it’s time for a haircut”.

2 Likes
4

When I took Bach with RST he said with regard to the HTSM manoptics to “use your own words” which always stuck with me. I think mencomics made from letters that form a phrase only really work if you adopt the words or translate them into something you can adopt.

2 Likes
5

How would you test that you are using a heuristic and not simply falling back on a repetitive pattern based test approach (dare I say checklist) in the form of a mnemonic?

If possible, I would look for examples of how you test yourself objectively on this. More importantly, how would you falsify the hypothesis that you are using heuristics, and not simply running through a series of test approaches that one typically falls back on?

I once considered a heuristic… “When I am not sure if I am really using heuristics, ask someone to observe me when I test”. I would try to explicitly explain why I am falling back on one heuristic vs another given what I think I know, and how/why I am choosing to seek new unknowns where I am currently looking. Describe the reasons for me not simply applying an appropriate test approach. How am I adapting, and learning about the testing problem that I face based on my experimentation so far?

Note: that likely would result in an awful mnemonic;)

4 Likes
6

I dunno, you could probably make a good mnemonic out of that, but it may link to a series of heuristics.
Part 1: Observe Me - OM
Part 2: When I Test - WIT
But then you get to the fuzzy part. OM WIT when I think I’m using a heuristic. OM WIT when I doubt my path. OM WIT if I think I can get better. OM WIT when two are better than one. OM WIT for a hundred other situations.

And then you get to the point where you realize that the mnemonic actually doesn’t do anything for you, because what you really want to do is work in a team and ask for help and do testing in pairs.

So you throw away the mnemonic realizing that yes, it would result in an awful mnemonic.

2 Likes
7

I’d say that falling back on a repetitive pattern based test approach and using checklists are both heuristics. And I’d say that running through a series of test approaches one typically falls back on is a heuristic. I’d also say that using a mnemonic as a memory retention aid is also a heuristic.

So mnemonics are heuristics, but heuristics don’t have to be mnemonics.

2 Likes
8

We don’t have to use letters, of course. A mnemonic is any mechanism to remember something.

“If you watch me while I test then I will learn to do my best” is a mnemonic. And a heuristic.

Or “Quality can seem quite mystic, so I’ll use a new heuristic”.

Or “When you go out for a jog, never pet a burning dog.”

4 Likes
9

In my experience Mnemonics used in testing normally refer to a ‘testing model’ and provide an easier way to remember the model. By using mnemonics in this way it can trigger ideas of ways to test a product which in turn can lead to using heuristics.

Heuristics has its own issues in that it can lead to biases and influence judgements you make whilst testing. Being aware of these biases is what helps to improve your testing efforts. Daniel Kahneman talks about system one and system two thinking and that heuristics are normally in the fast and frugal system one and liable to error. As a tester when using heuristics to enable fast testing based on your experiences, knowledge and feeling it can be very efficient. However you need to critically analyse your use of heuristics by slowing down and engaging system two thinking.

To summarize Mnemonics are a useful tool to aid memory recall of different testing models you can use when testing. Heuristics are the triggers for ideas and thoughts you can use when testing but are influenced by biases.

2 Likes
10

I look at the difference this way:

Mnemonic - a tool to make remembering something easier. Can often be in the form of an acrostic (Many Volcanoes Erupt Mulberry Jam Sandwiches Never Under Pressure to memorize the order of the planets is an example, as is the one I learned for the Mohr hardness scale: The Girls Can Flirt And Other Queer Things Can Do (Talc, Gypsum, Calcite, Fluorite, Apatite, Orthoclase, Quarts, Topaz, Corundum, Diamond). The rhyme for the month lengths is another mnemonic (30 days hath September, April, June, and November…). Typically a mnemonic acts as a key reference to the thing you want to remember but doesn’t tell you much if anything about the thing you’re trying to remember.

Heuristic - a rule of thumb/partial oracle used to aid in quickly measuring something. Some examples from outside testing are using stride length as an approximation for meters or yards, using your thumbnail to measure apparent size of something in the atmosphere. In testing, they’re generally used to survey application behavior in order to quickly locate potential issues.

I suspect the main reason the two get conflated is that people often use mnemonics to remind themselves of heuristics.

4 Likes
11

A heuristic is a fallible way to solve a problem.

The root of “heuristic” is “heuriskein”, “to discover” or “to find out”. The word had fallen into disuse until it was revived by George Polya in his book How to Solve It (1945), which provided heuristic approaches to solving math problems. (This is a terrific book if you struggle with math; a very helpful set of ideas.)

One really good and accessible book on the subject of heuristics is Gut Feelings by Gerd Gigerenzer. Another is Discussions of the Method by Billy Vaughan Koen. “The method” referred to is the engineering method. Koen suggests that everything in engineering is heuristic. He says that “the engineering method is the use of heuristics to cause the best change in a poorly understood situation within the available resources.” Every approach to solving a problem is heuristic, he says, because because we don’t know everything and everything is vulnerable to potential failure. Indeed, he claims, all is heuristic, including the assertion that all is heuristic!

Heuristics are fallible and context-dependent. In general, they tend to become less fallible when they’re applied by people with the appropriate levels of skill and wisdom in appropriate contexts; more fallible when applied carelessly or foolishly or in inappropriate contexts. Yet the nature of heuristics is that they’re always vulnerable to failure.

A mnemonic is a special kind of heuristic. A mnemonic is a fallible way to solve the problem of remembering stuff. Mnemonics are so named after Mnemosyne, the Greek goddess of memory. Interestingly, this fact doesn’t help me to remember how to spell or say her name.

Everything we do and use in testing is a heuristic. All models, oracles, and test techniques are heuristics. None is guaranteed to work; all hold the possibility of failure. No test is guaranteed to find a problem.

We use guideword heuristics to express, in a compact way, things that we might think about or do or cover or… All lists of guideword heuristics are themselves heuristics. A checklist is a special kind of heuristic; it’s a fallible way to solve the problem of forgetting things. As Kate points out, we might use a mnemonic (a special kind of heuristic) to help us remember the items (heuristics) on the checklist (also a heuristic). Like turtles, it’s heuristics all the way down.

A model is a special kind of heuristic. Applying a model is a fallible way to solve the problem of understanding something complex by representing it in terms of something simple.

An oracle is a special kind of heuristic too. Oracles are fallible ways to solve the problem “am I seeing a bug here?”. Oracles can fail to point us to a bug, and oracles can fool us into believing that we’re seeing a bug when we’re not.

A test technique is a heuristic for solving the problem of how to test something.

I’d like to offer to Kate that although heuristics can certainly be used to measure things, they can be used for much more than that. I’m also not sure what you mean by “partial oracle”; can you help me out?

7 Likes
12

How does a heuristic differ from a method?

13

I found an answer to my question over on StackOverflow, in a discussion of algorithms and heuristics, where KrisS, then a developer at Wallix, said that: an algorithm is the description of an automated solution to a problem. What the algorithm does is precisely defined. The solution could or could not be the best possible one but you know from the start what kind of result you will get. You implement the algorithm using some programming language to get (a part of) a program.

If you can’t develop an acceptable algorithm in a reasonable amount of time time, you may be able to find a not too bad solution much faster, by applying some arbitrary choices (educated guesses) – and that’s a heuristic. A heuristic is still a kind of an algorithm, but one that will not explore all possible states of the problem, or will begin by exploring the most likely ones. In some cases you’re not searching for the best solution, but for any solution fitting some constraint. A good heuristic would help to find a solution in a short time, but may also fail to find any if the only solutions are in the states it chose not to try.
KrisS wraps it up by saying that a heuristic is an algorithm that helps us navigate in the solution space of a particular problem. Or, it is a strategy to modify an algorithm to make it practical. (NOTE:) All methods are algorithms, and a heuristic is definitely a method.
.

2 Likes
14

interesting with a touch of NP,P lingo. I wonder if heuristics are NP… https://en.wikipedia.org/wiki/NP_(complexity)

15

Three strategies I use are:

1 - Consider whether I have more than one heuristic that might help me solve the problem and whether I’ve considered them. If I only have one or have no other ideas then I’ve possibly not thought hard enough about the problem.

2 - Remember that heuristics are fallible, have I considered how this heuristic may fail in ways that would be significant in this context. If I’ve not thought about this then I’ve probably not though enough about whether this heuristic is good enough for the problem/context.

3 - Remember that heuristics have limits of use (the scope within they may usefully solve a problem) and do I understand this limit and its fit with the current problem context.

NFL - Number, Fallibility and Limits of Options/Heuristics :slight_smile:

I’m more likely to apply this to larger or more impactful bets than I am to smaller ones with fast feedback and limited impact.

3 Likes
16

John… I don’t see how all methods are algorithms. An algorithm is a formal method that has three specific characteristics: it can be entirely encoded; it is guaranteed to produce a correct answer; and it is guaranteed to terminate. But many methods are not formal; cannot be entirely encoded (when they depend on tacit knowledge; when they must be adaptable; when they’re based on context-sensitive human preferences, etc.); are not guaranteed to produce a correct answer (or even a satisfactory one); and are not guaranteed to end.

Even applying an algorithm is a heuristic, in the sense that an algorithm applied in an inappropriate context or for the wrong purpose will very precisely and correctly do an inappropriate thing. But there are many other methods for solving a problem that are not algorithmic.

17

Using your definition of algorithm, I agree with you. It’s unfair of me to criticize or explain KrisS’s statements, since she’s not present in this conversation, but I think her motivation was to find similarities between heuristics and algorithms as a way of explaining heuristics, and she went too far ("[A heuristic] is still an algorithm, all methods are, and a Heuristic is definitely a method.). Nevertheless, StackOverflow readers awarded her 73 “up” votes.
Knuth (Vol 1, 3rd ed, p4), identifies 5 important features in algorithms:
“Besides merely being a finite set of rules that gives a sequence of operations for solving a specific type of problem, an algorithm has five important features:
”(1) Finiteness. An algorithm must always terminate after a finite number of steps. …
"(2) Definiteness. Each step of an algorithm must be precisely defined; the actions to be carried out must be rigorously and unambiguously specified for each case…
"(3) Input. An algorithm has zero or more inputs: quantities that are given to it initially before the algorithm begins, or dynamically as the algorithm runs. These inputs are taken from specified sets of objects. …
"(4)Output. An algorithm has one or more outputs: quantities that have a specified relation to the inputs. …
"(5)Effectiveness. An algorithm is also generally expected to be effective, in the sense that its operations must all be sufficiently basic that they can in principle be done exactly and in a finite length of time by someone using pencil and paper. …"

Can your “can be entirely encoded” fit in one of those features? I think it fits in “Definiteness”. Regarding correctness, I think he would say it is correct with respect to a specification. Do you agree with that?

1 Like
18

Nevertheless, StackOverflow readers awarded her 73 “up” votes.

Confused StackOverflow readers awarded her 73 “up” votes.

“Entirely encoded” is consistent not with one of the features, but with ALL five. “Entirely” suggests finiteness; otherwise it would be incomplete. Definiteness; yes. The inputs to an algorithm must be defined, just as the ingredients for a recipe are listed along with the procedure. The algorithm must produce some kind of explicit output; and the “pencil and paper” part incorporates the notion of encoding. You must be able to write an algorithm down; it must be explicit. Heuristics in general do not require that; special cases of heuristics (like algorithms or recipes) might.

And yes; we can only say that an algorithm is correct with respect to a specification; just as we can say that a test “passes” with respect to one or more explicit conditions. This is why, in Rapid Software Testing, we reject the notion of testing being reduced to test cases, or the idea that tests “pass” or “fail”. (For one thing the test doesn’t do either one; a product “passes” or “fails” the test.) We don’t ask whether tests are passing or failing; we ask “is there a problem here?”.

Testing itself is heuristic.

—Michael B.