I am looking for definitions of a test (from books, blogs, etc.). In specific, I am looking for definitions which distinguish between a good test and a not good test.
I have seen definitions of testing (as opposed to “test”). No matter how lofty the definitions are, it seems, like I can come up with silly test ideas, yet fit the definition of testing.
I think there should be a litmus test for a test/idea. It can’t be wishy washy.
I saw a great post by Liz Keogh - she said (in the context of BDD) that she doesn’t bother writing tests which are intuitive. She only focuses on non-intuitive tests.
I’d prefer not to talk about test automation in this thread.
I’m interested - what would you class as a “silly test idea”? For me, it all depends on context and goals. A test that might seem silly to one person might actually be totally valid and relevant to another, in a given context.
Put extremely simply, a test to me is an experiment; a way of conducting research. I see a lot of cross-over with, and ideas we can draw from, the area of science. I don’t think we need to get super specific about what a test is, because the possibilities are so broad. I think that’s a good thing, and one of the fun things about testing.
Tests must provide fast, meaningful, and reliable feedback on quality and business risks. These are IMO the basic quality attributes of “good” tests.
Worth to mention are also the FIRST principles:
• Fast: Execute quickly
• Isolated: Have a single reason to fail
• Repeatable: Produce the same result when executed multiple times
• Self-verifying: Automatically detect whether they pass or fail
• Timely: Written concurrently with the code under test
Those principles contribute to the above mentioned attributes.
Are you talking about a noun or a verb? Important distinction I think. Why are you worried about ‘silly test ideas’?
I’m interested when you mention the elimination of ‘non-intuitive tests’… you need to provide a quote so that we don’t misinterpret this. Leaving aside the fact that intuition is vastly different from person-to-person, I think you’d want to cover happy path scenarios (is that what you’d call intuitive, since it’s how the thing is supposed to work) as well as the more complex edge cases etc
• Fast: Execute quickly
• Isolated: Have a single reason to fail
• Repeatable: Produce the same result when executed multiple times
• Self-verifying: Automatically detect whether they pass or fail
• Timely: Written concurrently with the code under test
I think the above are all attributes of a good UNIT test, not necessarily true of other levels of testing and especially UI test cases performed by a human.
Consider looking into other keywords used to mean a similar thing like: ‘test idea’, ‘test case’, ‘test scenario’ . This is an old article but still a great read:
I see where you’re coming from, but I still see these as kinds of experiments, though they may not be innovative or ground-breaking.
When I think of experiments in science and research, two big things come to mind - a (null) hypothesis (maybe taken from a requirement or test charter) and repeatability (like steps to recreate or regression tests, or “checks”, as you may prefer to call them).
While I appreciate your point re glorification, I also wonder what the value is in limiting the definition to exclude activities / elements which are still part of testing, as a whole. Maybe I’m missing the motivation behind your original post. I’m also still curious about what a “silly” test idea is for you
I notice that Wikipedia has no current definition of ‘Test Case’, since there was some copyright infringement it seems, as of literally days ago!
If you want to see the last definition it can be viewed here in the history:
Perhaps in return for all the information provided, you can explain your definition or thought on the above?
There is nothing new in this. Confirmatory tests with a known outcome form the basis of process driven testing and test automation. I have no regard for them.
the value in limiting the definition
If I have “test” in my job title, why would I promote superficial tests? You don’t have a problem with that?
I prefer to look for something valuable or useful over something that’s new. New doesn’t always mean better. And when is something no longer new enough to be worthy? Perhaps there’s a good reason why you’re seeing this pattern in responses / definitions and (as far as I see) haven’t come up with something you’re happier with either.
I’m a bit concerned about your disregard for confirmatory tests. Have you ever opened up an app to test it, just to find that the most basic acceptance criteria haven’t even been fulfilled? Wouldn’t you have liked it if the dev had at least tested / checked that part before asking you to take a look? I interpret your words to mean you think this type of thing is beneath you, which I believe carries a number of risks.
I try not to restrict myself based on job titles. And I don’t think it’s about promoting superficial tests. I think it’s more about understanding the tools and when to use them.
If you ever find a definition of a test that you’re happy with, feel free to come back and update us.
Not too sure where you are going with this. The ideas I mentioned are nothing new. You can disagree with them if you wish. Also, I am fine with the ideas of test (as I have listed).
My question was specifically whether ‘test’ is defined/explained in any book, blog, talk.
Good tests and bad tests do not really exist. It’s up to us as testers to create quality tests and determine what are the most suitable tests to discover information about specific risks. We can do this by questioning and discussing ideas with our teams. Additionally by discovering risks, forming test ideas around those risks, and being observant when executing them. Not every test will be of the same quality, or be structured in the same way, so you need to remain vigilant. It’s up to us, as we test, to question what we have done and what we have learned, to see if assumptions were made or if details were missed. We can always learn and improve to deliver better testing.
Tests are the no of ways we can learn about the product , about its behavior or risks. In SD you know, you do, and then u should try different ways (which are tests here ) to know whatever u have done is correct to deliver.
