Test cases thoughts

Discussions
,
1

Morning,

I know the topic of “test cases” are one of the most divisive topics but do even the nay -sayers believe that there are occasions where it may be required.

For me, there are occasions where this is essential, I also find that having the test cases can sometimes give that assurance to others that the tester understands fully what the expected outcome is (especially on occasions where the specific detail of what is supposed to happen is important). And for some people the presentation of the test cases and scenarios just adds to the optics of the test function.

Even if exploratory testing bolsters up the specific tests, the specific tests can just prove the foundation is understood and works exactly as required.

Would you support the idea that perhaps, even if session-based approach is a preferable method, then being able to write well presented scenarios when required is a necessary skill for testers to have??

2 Likes
2

Welcome to the biggest software quality community in the universe Paul.

I have no problem with people who have no test cases. Nope, none. I have my context, and my context is tens of thousands of customers all taking slightly different journeys, these are not task-based workers who do just one very samey thing, and they all expect security too. And so I 'm expecting them to do a bit of this,

Untitled
Untitled270×602 98.4 KB

and perhaps a bit of hacking.

No test cases, opens you up to a lot of abuse in my line of work if we don’t. But some products simply don’t warrant captured test cases. Testing should never be about optics, but rather about communication of facts and shared understandings. It’s very context specific.

8 Likes
3

I bloody Love that picture!

2 Likes
4

Hi @monsieurfrench,

Great question and thanks for asking.

I think there’s value in test cases if the context is appropriate and the test case approach is considered as part of a wider testing strategy. I’ve found test cases – even a simple checklist – invaluable when it comes to collaborative team-based regression test sessions.

So learning how to write helpful test cases is an important skill for testers to have.

Of course, we could debate the value of test cases in isolation yet I don’t think there’s much value in having that discussion.

I think the key skill is learning to write in a way that resonates with an audience. And that’s valuable for any approach to testing – with test cases, testing notes, test reports, session-based tests, mind-maps etc. I often reflect that we don’t talk enough about how to write. Cos once we get good at that we can apply those writing skills to all sorts of contexts.

Thanks again for your question.

3 Likes
5

Ha, me too. That’s hilarious! Thanks for sharing, @conrad.connected. :grinning:

1 Like
6

In my case I do test cases to prove that I have tested a particular piece of functionality. I work for a very “hands-off” manager, but I dread the day he actually asks me what I have done on a particular day. I may not be able to remember unless I have a test case for that date, with bugs tested, steps, outcomes etc.

We work in a loosly agile environment. As long as all is going well, my manager is not concerned with what I am doing. On the other hand however… Please let that day not come! :rofl:

3 Likes
7

Would everyone define a ‘test case’ in a similar way? Does it mean, represent, is used in a similar way?
I take the definition of Cem Kaner: a test case/idea is a question you ask of a program with the scope of learning something about it;

I think it’s more the ideas behind the different interpretations:

  • if you let it mean that testing = test-cases, then you reduce the entire profession to a few simple stories;
  • if the use cases that the business writes and wants to go through are called test-cases, then you get the feedback that you don’t need testers; business, developers, pm, support,…whoever can do some checks;
  • if you get a less knowledgeable manager or leader to guide your testing strategy as a tester telling you to write down detailed test-cases; your skills as a tester might never be used, and your authority and knowledge never reveal the true power of testing…
  • if you use test-cases as a way to document a product, it’s the wrong reason; if any product documentation is to be stored, maybe the business can classify and store the requirements, specifications; maybe the devs can comment their code and describe their infrastructure and choices…
  • if you believe that what a person learns testing is transferable to other testers by the way of written test-cases, you fall into a huge trap; people have different backgrounds, mental models, products evolve, history rewrites itself, knowledge is mostly implicit(80-90%);
  • the opposite is also valid for new testers, restricting how new people learn to usage of the test-cases will make them miss things that would develop their learning quicker;
  • writing, maintaining and reusing many detailed test-cases so that they are precise enough of what it’s demonstrating and clear to everyone takes away time; that time is not used searching for problems;
  • writing test-cases for the purpose of using them for automation to get rid of testers and reduce testing costs;

I don’t know how others are experiencing the evolution of testing as a profession, but I see from dozens of jobs and companies that testing is now: write maintain execute test-case, automate test case and that’s it. And it saddens me as the human potential is so big and we could reveal so many problems that can improve software, but we don’t and software is getting worse.

2 Likes
8

My thoughts here might relate a bit to this.

For me, the test cases can have the role of demonstrating that we understand what actually needs to be tested. Both from the tester’s perspective, are they happy**, and also from anyone who wants to see the testing that has been done (so the test cases being part of the artifacts created - scenarios, notes etc).

**In my (current) case there may be times when the user stories etc don’t have full detail in them as it might be tapping into existing processes, or there might be some assumptions of knowledge of existing processes (maybe not the best practice I know).

It’s all context though, some functionality can be quite mechanical (i.e. no user experience involved), vs other testing around features where the UI/UX plays a big part in the assessment of the finished article. Sometimes a lighter touch is OK (checklists + basic expected results) , sometimes something more details and thought through is more appropriate.

9

Software, does sometimes seem to be getting worse @ipstefan . I think the corollary is that software does more and executes in larger environments today than ever before. So our perception of decline is really an impact of so many side effects on it. Software has to now sell into a bigger market than ever before in order to generate the same revenues as a SAAS product that an monolithic on-premise SQL application used to do for that huge 100K price tag that brought home the bacon back in 1990. the market competition got stiffer and so we all opened up our doors to more requirements in order to stay competitive?

So I don’t really buy the broad assertions that seem to be getting louder lately that testing itself is in a bad state. And that’s why cool questions like Paul’s are so revealing. My job is under a lot of pressure to capture and manage test cases today, and that has coincided with things like ISO27001 certification, with 9001 probably not far off for me either. But sometimes I see apps, that definitely would not benefit hugely from structured regression testing, and that means jobs for humans putting real eyeballs onto desktops, browsers and phones. But also working at their analysis skills and sharing implicit knowledge as a way of staying in business with apps that do just one thing, and do it well and without fluff that needs testing corner cases to death.

2 Likes
10

Might be a bit rambly this one.

Perhaps when I say test cases, I mean communication - I.e. How evidence of the testing done is communicated to others people. I do think that for me, it demonstrates, or contributes to the demonstration, that the change is understood. Especially when working with people newer to the team or perhaps offshore.

Test case might be the wrong term - what I meant now perhaps on reflection is a scenario but with some detail around the expected results, rather then detail of the steps involved (which yes, can on the large be quite pointless) .

Maybe to give an example.

Supply chain domain. An api used to determine the earliest date we think we can supply an item. About 6 different factors at play in order to calculate the estimated earliest delivery date. At a basic level some detailed scenarios here to show the basics are working are important. I.e some detail of the factors and conditions and what the expected outcome is when the api is called. If we have a set of scenarios with some detail, I’m more than happy to have this propped up with lighter notes like a checklist or notes. Especially if a lot of the general expected results are similar.

I’ve always considered test scenarios to essentially be mini “sessions” rather than test cases in the “classic” or “Istqb” sense. So a test idea to test rather than an explicit set of steps to follow. Just sometimes we need to ensure that the actual result we’re checking is fully understood.

1 Like
12

You don’t need test cases to do that. You can use notes, screen recordings, screenshots, bug reports and whatever else you need to do. Session based test management was invented to cope with the need for documentation in a world of session-based testing. That’s one example, you don’t have to use sessions.

13

Wish I’d read this far down before my first reply.

What you seem to be saying here is that it’s important, when testing, to be able to be specific. I agree.

I should point out that this emphatically has nothing to do with test cases in any sense I’ve heard of them.

If I am doing a session on this API I will look at all of these factors, collect them, note them down, then I will determine what level of functional coverage I’m going for, write them down in a grid or run them through a pairwise system or whatnot, and execute as I need to. I perform testing and I make notes to the degree of specificity and exactness that makes the most sense for my purposes. Too exact and I cannot get enough done, too vague and I cannot draw reliable conclusions, so I go back and forth between the two. A tester has to be able to think both broadly and narrowly as the situation requires.

Sometimes I am very specific in my testing. I can use lab conditions. I might collect specific data and use statistical analysis. Also every tester must be able to work with confounding variables and be able to make consciously specific observations with proportional inferences based on the reliability of the quality and quantity of variations of those heuristic observations (although they don’t have to actually know most of those words to do it).

I never used a test case to achieve any of those ends. They are appallingly wasteful and error-prone in their communication, for doing testing, and for training purposes and they are costly in their storage and maintenance. They are an impractical way to approach testing. Give me any example of their value and I’ll likely find you a better way to solve that problem.

Session-based testing actually doesn’t ban anyone from using any tool, including test cases, if you found some insane context that calls for it, because while test cases focus on the process a session focuses on the purpose and while test cases are instruction-driven sessions are tester-driven.

Edit:
I’m trying to get my head around the perspective that leads to the idea that lower-script testing (not using test cases, using stuff like sessions instead) is vague or undisciplined. If it’s because it looks unstructured, that’s because the structure is internal, in the mind of the tester. It exists, we just don’t write it all down.

1 Like
14

@kinofrost superb reply there.
Agree with all of that there, and especially that last paragraph before the edit.

1 Like
15

Upon reflection I’d say my choice of subject is perhaps a little misleading. The term for “test case” does conjure up quite specific things in terms (exactly as @kinofrost says) procedures rather than expectations and outcomes. I’m not an advocate for specific test cases with detailed procedures. (though not against some supplementary reference documents which detail some example procedures for how a process may be enacted).

1 Like
16

Thank you, I hope I went some way to answering the question, though, which I’m beginning to believe I did not. I wonder if it’s a communication question. I think a good start to communicating what happened during testing is knowing who you’re communicating to and why you need to communicate it.

Communication of testing is actually quite a big subject. What we report depends on who we’re reporting to and why, but if I need to communicate what I did during testing (for example during a debrief) I take notes that might include text, videos, screenshots, tables, graphs, diagrams, documents, whatever I need to tell my story. Then I talk through those.

I think it’s worth being wary of abstraction problems. A description is necessarily missing information about what it is describing. If I describe to you how I drove from one place to another I am forced to select what about that process was important when describing it. That means I have to make informed estimates (perhaps informed by asking you, but still estimates) about what you think is important. It’s worth remembering that anything you write down is not what you did. It’s just one particular perspective with selected details that misses a lot of context. That’s why there are so many books about the Tudors.

What I’m getting at is that being specific and communicating that specificity are two different problems. I’m specific about the api factors and the observations I make and the parts of the state I’m controlling that are important to the process because I want to draw strong conclusions from my setup, operation, and observations. A reasonable documentation of those factors might look like a table that means very little to anyone else. It fails to describe what the results infer. It does not say how I came to the conclusion that this table is a good idea. It does not explain what I have not yet tested. Those things, and how I go about communicating them (such as what sort of language and tone I use), require an audience - I need to know who I’m writing for so that I can estimate their needs and write appropriately.

It depends on contextual information. If nobody needs to know but me then I’ll write down as little as I need to remind myself of what I’m doing and keep track of my tasks. If another tester needs to know I will dress it up with purpose. If I’m training someone I might list how I came up with my ideas so I can express that or send it to them for reference. If the government needs to know I might be particularly exacting or follow some standard that I know they approve of.

Writing things down is time you could be spending testing, and it’s often hard to write things down while you’re deep in thought about quality criteria and risk assessment, so ideally we write down nothing… then add to that whatever has value that outweighs the cost.

I hope that is more useful to the idea of communicating what happens during testing.

2 Likes
17

Another great reply and I appreciate your time there. I will also offer some more asap.

Something we need to consider also here is methods and approach that can give us information quickly as possible. Following test cases isn’t necessarily going to be the best way of doing this at all. (maybe far from it)

I’m my api example, I would most certainly favour grid for some specific detail of the test procedures as these are going to be standard. I.e. Showing each factor and it’s condition. Propped up by some scenarios and test ideas.

18

I’m not sure what is being talked about anymore.
Is it about reporting? Showing progress while testing? documentation ahead of testing? what and how is being documented the testing? the test approach which is more scripted sometimes? is it about test-cases, scenarios, ideas, sessions, or representations of a to-be-performed or performed test? is it about doing testing sessions with sometimes a structured and more documented approach and other-times without much left after? or maybe about different representations of a particular approach that can be more explicit in terms of paths or outcomes (a graph, a table, pairs, matrixes, etc…)?

19

The conclusion I came to was that it started as test cases (in error), became about the value of cases in proving test activities, then about recording specificity in testing and ended with using documentation of test activities to tell the testing story.

I tried to make the bus stop at all the stations you didn’t already cover, so I think everyone got off.

1 Like
20

I support the idea that test design is a necessary skill for testers to have. That means more than just approaching things from an exploratory or a scripted perspective.

It means being able to understand when a specific technique should be used and when it shouldn’t. It means having an understanding of many different techniques: scenarios, domain tests (boundary + equivalence class), combinatorics, functional, accessibility, long sequence, flow testing, state diagram, etc.

It means knowing how to create appropriate test documentation to support those various techniques. (Note: documentation will also vary based on your technique).

I do want to clarify something:

works exactly as required.

Remember our documentation, even detailed scripts, don’t document things exactly as required or exactly as they exist. All documentation is to some degree missing information that might be important. Have you ever included what processes or programs run along-side your program in your test documentation? Ever included all traffic conditions or network speed variances in your tests? Probably not.

Our oracles (expected results) are incomplete at best. We may think we know what the expected result is going to be (and we might be able to predict it to a high degree), until something unforeseen happens and then we get it wrong.

If you want to give people detailed scripts (you might call these test cases) because you think it helps, great, as long as it does in fact help.

2 Likes
21

Maybe I miss it but I find the best use of test cases are in onboarding a fellow tester who are new to the company.

If we are talking about communicating test results, I don’t think test cases are the only way. However, there definitely needs to be some records or notes (if not for stakeholders, for your future self).

I have often taken the notes from my exploratory testing sessions and convert them to test cases. The converting to test case might take more time but I like to think that future me or some tester down the road will appreciate it. Heck, even if it’s some knowledge, document it in internal wiki.