How to make a Test Case matrix?

[ThePirateTester (Lee Marshall)]

At my workplace, we often use a test case matrix to plan out what tests we want to do, which generally looks like a very elaborate decision matrix.

Within my team, I’m the only one who knows how to make them, and I’ve been asked to do a session on how to make it.

My first thought is, what I call a test case matrix might be different to others, so I thought I would say what I’m wanting them to do.

For this scenario, it is working out what tests they would do, for a report that lists all withdrawl requests => £6000. However, if the customer is >= 60, then the threshold is lowered to £3000.

My immediate thoughts are this.
For amount options, there’s six different amounts I would look at:

1. £2999.99
2. £3000
3. £3000.01
4. £5999.99
5. £6000
6. £6000.01

And then for age options, there’s three I would look at:

1. 60 years old -1 day (I wouldn’t say 364 days old, as what about leap years?)
2. 60 years old
3. 60 years old +1 day

If you times them together, I would get 18 possible tests, of which some I would expect to be on the report, and some not. But would all 18 combinations be needed, or would it be better for them to show them, and then discuss if it is necessary to test to not?

You (and they) might ask, why do three range BVA? Well, how do I know it hasn’t been coded for = £3000 for example, rather than >= £3000?

So the below is a sample of what I would expect them to build, to show if somone should or shouldn’t be shown in the report.

Amount     3000 2999.99 3000.01
Age        60   60      60
On report? Y    N       Y

Now, if I really wanted to curveball them and raise the difficulty, I would get them adding another layer which is what if there is more than one person on the account? And then all are below 60, all above 60, and one is below 60 and one isn’t?

If I gave you the above scenario, would you have ended up with 18 combinations?
Would you even use a matrix, or do something else, like a flow chart to look at combinations instead?
Are there even more things that I’m missing and should be mentioning?
Do you even use a matrix such as this to work out your tests?

[wildtests (Stu J)]

Starting caveat to say… anything people comment here is subjective and the best answers will come from an in-team conversation with all those who are building the solution. They will know more of the quirks of how they’re building it, the fields they’re using etc.

I used matrices heavily in the past and there are definitely applications of them, excel is a great friend in this regard!

You’re making an assumption here that the only valid tests are e2e - what do you know about how the logic for each element is implemented? Could you test the amounts in isolation, the ages in isolation, the “on report” stuff in isolation, then run a couple of e2e tests to check things are flushed through the system correctly? What is already covered by unit and integration tests, and what do those tests look like - are they sufficient to handle any of the component logic with a few higher-level e2e tests checking things look good in the final state?

Regarding the numbers I’d look at, BVA is a good tool but I’d suggest median values can also provide some value if it’s not too intensive to add them (eg writing a long stepped test case which makes you want to die) - £1500 and £4500 (and I don’t know, £150,000) are possibly closer to your customers’ average use cases (find out!) and therefore worth considering. Also, when considering boundaries, why not look at £0.01/£1 (smallest possible), £100,000,000,000,000 (biggest possible) etc and check they’re handled correctly?

Honestly, if you gave me this problem in my own place I’d probably write a handful of scripted tests to cover the 9 (or 18) scenarios you describe, automate 3-5 of them, and cover the bug hunt with a number of exploratory charters around each variable you mention, rather than building a matrix, but I get that we all exist in different cultures

I suppose my main tl;dr point is… matrices have value but shouldn’t be the only way you approach testing problems, like anything else they’re just a tool in your belt.

[ThePirateTester (Lee Marshall)]

@wildtests

matrices have value but shouldn’t be the only way you approach testing problems, like anything else they’re just a tool in your belt.

Sadly, The Powers That Be firmly believe that everything needs to be written into a test case matrix to show what tests we want, and then heavily detailed scripts building on that to show how you test what is in the matrix.

So, I’m having to work within the best of the restriction around me sadly, Stu.

In terms of looking at what unit tests there are, in the past I have used a matrix to influence the dev by saying “This is what I want to test. Does your code handle it?”, and it either improves/increases their unit tests, or makes them review the actual code to handle what I want to do.

As for integration testing, that is where we focus our efforts, rather than looking at the generated report, as our end users will do that and confirm that it looks/feels right.

Also, I would build the matrix in Excel, as that is what we’re encouraged to use, and rightly so for something like this when it’s lots of variable fields.

[wildtests (Stu J)]

I hear that, I know those “powers that be” only too well… respect you doing good work regardless Lee

[ThePirateTester (Lee Marshall)]

To ease them into it, I’m thinking I show them an example, and explain how I came up with it, as well as why certain combinations I would avoid.

The scenario in this case is Delete a customer who was created over 1 year ago, with no account, no ID, no complaint

The way I do it in this case, is show a valid scenario, so over 5 years old, no account, ID or complaint. Then only change one thing from the positive baseline to fail, leading to this combination.

Test	1	2	3	4	5	6
Created date	> 1 year	= 1 year	< 1 year	> 1 year	> 1 year	> 1 year
Account	N	N	N	Y	N	N
ID	N	N	N	N	Y	N
Complaint	N	N	N	N	N	Y
Delete?	Y	N	N	N	N	N

(Also, pasting that in and didn’t realise I could do a grid here, that’s funky!)

The reason I only change one thing is if test 3 showed we would delete that customer, we can narrow down to where the problem is.
But, I also don’t do every combination, with a customer having more than one failing criteria, as this should be proved by the single failures, and would lead to an endless stream of tests as the number of conditions increase.

[hattori (George)]

@ThePirateTester

If I gave you the above scenario, would you have ended up with 18 combinations?
Would you even use a matrix, or do something else, like a flow chart to look at combinations instead?
Are there even more things that I’m missing and should be mentioning?
Do you even use a matrix such as this to work out your tests?

Hi Lee,
Here are three possibilities you may want to consider, in three responses. I hope they’re helpful.
George

1 of 3: You can use a pairwise test case generator to cover the interactions of all pairs of values. That is, all pairs of values appear for:

• Withdrawal amount in £s with Person 1 age in years
• Withdrawal amount in £s with Person 2 age in years
• Person 1 age in years with Person 2 age in years

in at least 1 of the 18 test cases. Here’s an example using Testcover.com.

Wed Oct 09 15:37:41 UTC 2019 Withdrawal requests Request data: Withdrawal requests Withdrawal amount in £s Person 1 age in years Person 2 age in years # 2999.99 3000 3000.01 5999.99 6000 6000.01 59.997 60 60.003 59.997 60 60.003 #1. No combination functions No substitution functions Test Case ID Withdrawal amount in £s Person 1 age in years Person 2 age in years Combo Countdown 6 Values 3 Values 3 Values 45 1 2999.99 59.997 59.997 42 2 3000 60 60.003 39 3 3000.01 60.003 60 36 4 5999.99 60 60 33 5 6000 60.003 60.003 30 6 3000 59.997 60 27 7 3000 60.003 59.997 24 8 3000.01 59.997 60.003 21 9 3000.01 60 59.997 18 10 6000.01 59.997 59.997 16 11 2999.99 60 60 14 12 2999.99 60.003 60.003 12 13 5999.99 59.997 59.997 10 14 5999.99 60.003 60.003 8 15 6000 59.997 59.997 6 16 6000 60 60 4 17 6000.01 60 60 2 18 6000.01 60.003 60.003 0

[hattori (George)]

2 of 3: You can use a simple function to compute the expected results, e.g. one or more output equivalence class. In this example, a simple function (Number_60_and_up) returns the number of persons over 60 for each test case.

function Number_60_and_up($age1,$age2) {
$number=0;
if($age1>=60) $number++;
if($age2>=60) $number++;
return $number;
}

Wed Oct 09 16:14:43 UTC 2019 Withdrawal requests and number of persons over 60 Request data: Withdrawal requests and number of persons over 60 Withdrawal amount in £s $Person_1_age $Person_2_age Number age 60 and older # 2999.99 3000 3000.01 5999.99 6000 6000.01 59.997 60 60.003 59.997 60 60.003 Number_60_and_up($Person_1_age, $Person_2_age) #1. No combination functions Substitution function(s): Number_60_and_up() Test Case ID Withdrawal amount in £s $Person_1_age $Person_2_age Number age 60 and older Combo Countdown 6 Values 3 Values 3 Values 1 Value 57 1 2999.99 59.997 59.997 0 51 2 3000 60 60.003 2 45 3 3000.01 60.003 60 2 39 4 5999.99 60 60 2 35 5 6000 60.003 60.003 2 31 6 6000.01 59.997 59.997 0 28 7 3000 59.997 60 1 25 8 3000 60.003 59.997 1 22 9 3000.01 59.997 60.003 1 19 10 3000.01 60 59.997 1 16 11 2999.99 60 60 2 14 12 2999.99 60.003 60.003 2 12 13 5999.99 59.997 59.997 0 10 14 5999.99 60.003 60.003 2 8 15 6000 59.997 59.997 0 6 16 6000 60 60 2 4 17 6000.01 60 60 2 2 18 6000.01 60.003 60.003 2 0 18 Subs

[hattori (George)]

3 of 3: As you point out, changing one variable at a time can be a reliable way to find faults for that variable. You also can consider a pairwise design (below). It can find faults for interactions between any pair of variables. There also are ways to constrain the design to valid scenarios (beyond the scope of this response).

Wed Oct 09 17:29:42 UTC 2019 Delete a customer Request data: Delete a customer Created date Account ID Complaint Delete? # >1_year =1_year <1_year N Y N Y N Y N Y #1. No combination functions No substitution functions Test Case ID Created date Account ID Complaint Delete? Combo Countdown 3 Values 2 Values 2 Values 2 Values 2 Values 48 1 <1_year N Y Y Y 38 2 <1_year Y N N N 28 3 =1_year Y Y Y N 20 4 =1_year N N N Y 12 5 >1_year N N Y N 6 6 >1_year Y Y N Y 0