Breaking the rules with heuristics

šŸ™‹ Questions
, ,
1

To understand how a system should behave, in order to test it, I naturally build mental models. I sometimes make use of informal models such as mind maps, or more formal models such as state transition diagrams.

Iā€™ve been thinking recently, about I might be able to build tests that exercise a system, and heuristically assert that some generalised rules apply. I want to do this without explicitly following known steps and looking for eact outcomes.

This got me thinking about Domain Models, and how some behaviour is fairly universal or part of an external standard, some are generally expected within a wider industry domain, while the rest are very specific to a given application and itā€™s current implementation.

Behaviour from external standards or guidelines

A good example of an external standard are the Web Content Accessibility Guidelines. These are well known, external, and versioned. So much so, that I need not create my own tools or design my own tests, as great tooling already exists.

Behaviour from industry norms

Next come industry norms, things we can expect because the type of application is well known. For example in an E-commerce site I can expect it will have a shopping cart or basket, and I can expect I can add items as I shop. I can also expect there will be page where I can view what is currently in the basket, remove items, change the quantity of items. Iā€™m sure there are many other generalised assumptions I could make, that would be true for a majority of such sites.

Domain specific behaviour

Then we come to the domain specific behaviours, the business rules that are unique to your system under test. Itā€™s these behaviours that are most interesting to identify when building our models. An example could be, how to determine the next question to asked when checking symptoms, or when to throw a roadblock if a user enters a very serious symptom and they should be directed to emergency care immediately. Other domain specific rules could be easier to programmatically test for with a heuristic, like compliance to a style guide states all headings should be a shocking pink.

So, while I havenā€™t yet experimented creating a heuristic, rules based test, Iā€™ve not got some ideas where such rules might come from, and I can already predict that working with Domain specific rules might pose a bigger challenge and be less reusable.

2 Likes
2

I like the way youā€™re thinking! It sounds like this could fit well with exploratory testing, where certain rules/heuristics are in your charter and you simply set off exploring and see if any of them are broken. For example, if the basket is meant to recommend me a related product according to some rules (domain specific behaviour), in addition to testing the implementation of those rules I could also explore from a customer perspective what sorts of products are recommended and whether they break the heuristic of ā€˜the recommended product is similar to or complements something in my basketā€™.

When you say ā€˜build testsā€™ though I assume you mean automated tests, which canā€™t rely on a human judging the behaviour and needs to be programmable. Would you be limiting yourself to things that are easier to measure? The bit about not explicitly following known steps intrigues me too. Are you thinking a chaotic approach, eg locate all the things on the page that can be clicked, and randomly click on one of them? Or maybe a modular approach where there are multiple possible steps but they can be taken in different orders. For example, with steps ā€˜search for product and add it to basketā€™, ā€˜remove product from basketā€™, ā€˜change quantity in basketā€™ and ā€˜click buyā€™, the first three can be repeated several times with different products that could be selected randomly before the last step is taken.

2 Likes
3

I love the train of thought - and yes, my thinking does come from Exploratory Testing, and I am talking about automation. The subtlety I can add to this, is that I often combine the two together. Building automation with more or less rigidity, and using them to enhance more exploration.

So maybe I would build a tool that tells me all the H2 tags tare donā€™t have a H1 parent and H2 sibling. This alone might not be a total fail, but it could give me clues where to go hunt for inconsistencies.

1 Like
4

Oh interesting! Iā€™ve also been thinking about ā€˜automated testsā€™ that arenā€™t complete tests in themselves and donā€™t lead to a binary pass/fail, they are more of a tool to aid human testing, eg they go off and do the repetitive data collection then I look through and use my judgement to gain insight and maybe end with pass/fail.

2 Likes