What are the definitions of a test?

I am looking for definitions of a test (from books, blogs, etc.). In specific, I am looking for definitions which distinguish between a good test and a not good test.

I have seen definitions of testing (as opposed to ā€œtestā€). No matter how lofty the definitions are, it seems, like I can come up with silly test ideas, yet fit the definition of testing.

I think there should be a litmus test for a test/idea. It canā€™t be wishy washy.

I saw a great post by Liz Keogh - she said (in the context of BDD) that she doesnā€™t bother writing tests which are intuitive. She only focuses on non-intuitive tests.

Iā€™d prefer not to talk about test automation in this thread.

3 Likes

Iā€™m interested - what would you class as a ā€œsilly test ideaā€? For me, it all depends on context and goals. A test that might seem silly to one person might actually be totally valid and relevant to another, in a given context.

Put extremely simply, a test to me is an experiment; a way of conducting research. I see a lot of cross-over with, and ideas we can draw from, the area of science. I donā€™t think we need to get super specific about what a test is, because the possibilities are so broad. I think thatā€™s a good thing, and one of the fun things about testing.

5 Likes

Tests must provide fast, meaningful, and reliable feedback on quality and business risks. These are IMO the basic quality attributes of ā€œgoodā€ tests.

Worth to mention are also the FIRST principles:
ā€¢ Fast: Execute quickly
ā€¢ Isolated: Have a single reason to fail
ā€¢ Repeatable: Produce the same result when executed multiple times
ā€¢ Self-verifying: Automatically detect whether they pass or fail
ā€¢ Timely: Written concurrently with the code under test
Those principles contribute to the above mentioned attributes.

2 Likes

Are you talking about a noun or a verb? Important distinction I think. Why are you worried about ā€˜silly test ideasā€™?
Iā€™m interested when you mention the elimination of ā€˜non-intuitive testsā€™ā€¦ you need to provide a quote so that we donā€™t misinterpret this. Leaving aside the fact that intuition is vastly different from person-to-person, I think youā€™d want to cover happy path scenarios (is that what youā€™d call intuitive, since itā€™s how the thing is supposed to work) as well as the more complex edge cases etc

1 Like

ā€¢ Fast: Execute quickly
ā€¢ Isolated: Have a single reason to fail
ā€¢ Repeatable: Produce the same result when executed multiple times
ā€¢ Self-verifying: Automatically detect whether they pass or fail
ā€¢ Timely: Written concurrently with the code under test

I think the above are all attributes of a good UNIT test, not necessarily true of other levels of testing and especially UI test cases performed by a human.

2 Likes

Consider looking into other keywords used to mean a similar thing like: ā€˜test ideaā€™, ā€˜test caseā€™, ā€˜test scenarioā€™ . This is an old article but still a great read:

3 Likes

The classic response would be: a check if a (software) product fits the prerequisites.

There are actually many tests which are not experiments. Verifying requirements, without deviation is one. Running test automation is another.

I donā€™t think there is value in glorifying all of these as experiments?

1 Like

Thanks for this. I think the bigger question is - what is a bad test case.

1 Like

I see where youā€™re coming from, but I still see these as kinds of experiments, though they may not be innovative or ground-breaking.

When I think of experiments in science and research, two big things come to mind - a (null) hypothesis (maybe taken from a requirement or test charter) and repeatability (like steps to recreate or regression tests, or ā€œchecksā€, as you may prefer to call them).

While I appreciate your point re glorification, I also wonder what the value is in limiting the definition to exclude activities / elements which are still part of testing, as a whole. Maybe Iā€™m missing the motivation behind your original post. Iā€™m also still curious about what a ā€œsillyā€ test idea is for you :slightly_smiling_face:

2 Likes

I notice that Wikipedia has no current definition of ā€˜Test Caseā€™, since there was some copyright infringement it seems, as of literally days ago!
If you want to see the last definition it can be viewed here in the history:

Perhaps in return for all the information provided, you can explain your definition or thought on the above?

1 Like

what is a silly test idea

There is nothing new in this. Confirmatory tests with a known outcome form the basis of process driven testing and test automation. I have no regard for them.

the value in limiting the definition

If I have ā€œtestā€ in my job title, why would I promote superficial tests? You donā€™t have a problem with that?

1 Like

I didnā€™t really find an answer :frowning:

Since you ask about definitions. I have found some statements in the past:

  1. ā€˜ā€¦a test case is a question that you ask of the programā€¦ā€™. The premise is that you donā€™t ask questions for which you know the answer.
  2. Testing is something that we do with the motivation of finding new information
  3. Avoid labeling an activity as a ā€œtestā€ unless it represents a sincere effort to discover a problem
  4. I tend to ā€¦ avoid having conversations about really blindly obvious stuff

I am leaving out the sources to avoid appeal to authority. DM me if you have questions.

What I was looking for is a definition from a book or article or talk

1 Like

Found this after some searching:
A TEST THAT IS NOT DESIGNED TO REVEAL A BUG IS A WASTE OF TIME.

Interesting to see the author: testing - Why does Cem Kaner consider a test not revealing a bug a waste of time? - Software Engineering Stack Exchange

1 Like

I prefer to look for something valuable or useful over something thatā€™s new. New doesnā€™t always mean better. And when is something no longer new enough to be worthy? Perhaps thereā€™s a good reason why youā€™re seeing this pattern in responses / definitions and (as far as I see) havenā€™t come up with something youā€™re happier with either.

Iā€™m a bit concerned about your disregard for confirmatory tests. Have you ever opened up an app to test it, just to find that the most basic acceptance criteria havenā€™t even been fulfilled? Wouldnā€™t you have liked it if the dev had at least tested / checked that part before asking you to take a look? I interpret your words to mean you think this type of thing is beneath you, which I believe carries a number of risks.

I try not to restrict myself based on job titles. And I donā€™t think itā€™s about promoting superficial tests. I think itā€™s more about understanding the tools and when to use them.

If you ever find a definition of a test that youā€™re happy with, feel free to come back and update us.

1 Like

Not too sure where you are going with this. The ideas I mentioned are nothing new. You can disagree with them if you wish. Also, I am fine with the ideas of test (as I have listed).

My question was specifically whether ā€˜testā€™ is defined/explained in any book, blog, talk.

1 Like

Thanks for this discussion on this so far. Iā€™ve added some of the ā€œtestā€ terms to the glossary.

(Anyone with a Professional account can add a definition too)

I like this from @mwinteringham:

Good tests and bad tests do not really exist. Itā€™s up to us as testers to create quality tests and determine what are the most suitable tests to discover information about specific risks. We can do this by questioning and discussing ideas with our teams. Additionally by discovering risks, forming test ideas around those risks, and being observant when executing them. Not every test will be of the same quality, or be structured in the same way, so you need to remain vigilant. Itā€™s up to us, as we test, to question what we have done and what we have learned, to see if assumptions were made or if details were missed. We can always learn and improve to deliver better testing.

Source: Designing Tests: Whatā€™s the difference between a good test and a bad test?

2 Likes

A test is a kind of check done to verify whether on performing a certain action the actual result matches the expected result or not.

Whether it is software or any other item, while performing any kind of test the motto remains the same as above.

While performing any kind of test:

  • pre-condition/pre-requisite is a must, there might be something that needs to exist before we perform any kind of test

  • steps to perform the tests, obviously without which the test cannot be proceeded

  • expected result this is crucial because we have to compare this with the actual result and decide whether the test case is passed or not

  • actual result, while performing any test, it is important to record the actual result.

Tests are the no of ways we can learn about the product , about its behavior or risks. In SD you know, you do, and then u should try different ways (which are tests here ) to know whatever u have done is correct to deliver.