I am looking for definitions of a test (from books, blogs, etc.). In specific, I am looking for definitions which distinguish between a good test and a not good test.
I have seen definitions of testing (as opposed to ātestā). No matter how lofty the definitions are, it seems, like I can come up with silly test ideas, yet fit the definition of testing.
I think there should be a litmus test for a test/idea. It canāt be wishy washy.
I saw a great post by Liz Keogh - she said (in the context of BDD) that she doesnāt bother writing tests which are intuitive. She only focuses on non-intuitive tests.
Iād prefer not to talk about test automation in this thread.
Iām interested - what would you class as a āsilly test ideaā? For me, it all depends on context and goals. A test that might seem silly to one person might actually be totally valid and relevant to another, in a given context.
Put extremely simply, a test to me is an experiment; a way of conducting research. I see a lot of cross-over with, and ideas we can draw from, the area of science. I donāt think we need to get super specific about what a test is, because the possibilities are so broad. I think thatās a good thing, and one of the fun things about testing.
Tests must provide fast, meaningful, and reliable feedback on quality and business risks. These are IMO the basic quality attributes of āgoodā tests.
Worth to mention are also the FIRST principles:
ā¢ Fast: Execute quickly
ā¢ Isolated: Have a single reason to fail
ā¢ Repeatable: Produce the same result when executed multiple times
ā¢ Self-verifying: Automatically detect whether they pass or fail
ā¢ Timely: Written concurrently with the code under test
Those principles contribute to the above mentioned attributes.
Are you talking about a noun or a verb? Important distinction I think. Why are you worried about āsilly test ideasā?
Iām interested when you mention the elimination of ānon-intuitive testsāā¦ you need to provide a quote so that we donāt misinterpret this. Leaving aside the fact that intuition is vastly different from person-to-person, I think youād want to cover happy path scenarios (is that what youād call intuitive, since itās how the thing is supposed to work) as well as the more complex edge cases etc
ā¢ Fast: Execute quickly
ā¢ Isolated: Have a single reason to fail
ā¢ Repeatable: Produce the same result when executed multiple times
ā¢ Self-verifying: Automatically detect whether they pass or fail
ā¢ Timely: Written concurrently with the code under test
I think the above are all attributes of a good UNIT test, not necessarily true of other levels of testing and especially UI test cases performed by a human.
Consider looking into other keywords used to mean a similar thing like: ātest ideaā, ātest caseā, ātest scenarioā . This is an old article but still a great read:
I see where youāre coming from, but I still see these as kinds of experiments, though they may not be innovative or ground-breaking.
When I think of experiments in science and research, two big things come to mind - a (null) hypothesis (maybe taken from a requirement or test charter) and repeatability (like steps to recreate or regression tests, or āchecksā, as you may prefer to call them).
While I appreciate your point re glorification, I also wonder what the value is in limiting the definition to exclude activities / elements which are still part of testing, as a whole. Maybe Iām missing the motivation behind your original post. Iām also still curious about what a āsillyā test idea is for you
I notice that Wikipedia has no current definition of āTest Caseā, since there was some copyright infringement it seems, as of literally days ago!
If you want to see the last definition it can be viewed here in the history:
Perhaps in return for all the information provided, you can explain your definition or thought on the above?
There is nothing new in this. Confirmatory tests with a known outcome form the basis of process driven testing and test automation. I have no regard for them.
the value in limiting the definition
If I have ātestā in my job title, why would I promote superficial tests? You donāt have a problem with that?
I prefer to look for something valuable or useful over something thatās new. New doesnāt always mean better. And when is something no longer new enough to be worthy? Perhaps thereās a good reason why youāre seeing this pattern in responses / definitions and (as far as I see) havenāt come up with something youāre happier with either.
Iām a bit concerned about your disregard for confirmatory tests. Have you ever opened up an app to test it, just to find that the most basic acceptance criteria havenāt even been fulfilled? Wouldnāt you have liked it if the dev had at least tested / checked that part before asking you to take a look? I interpret your words to mean you think this type of thing is beneath you, which I believe carries a number of risks.
I try not to restrict myself based on job titles. And I donāt think itās about promoting superficial tests. I think itās more about understanding the tools and when to use them.
If you ever find a definition of a test that youāre happy with, feel free to come back and update us.
Not too sure where you are going with this. The ideas I mentioned are nothing new. You can disagree with them if you wish. Also, I am fine with the ideas of test (as I have listed).
My question was specifically whether ātestā is defined/explained in any book, blog, talk.
Good tests and bad tests do not really exist. Itās up to us as testers to create quality tests and determine what are the most suitable tests to discover information about specific risks. We can do this by questioning and discussing ideas with our teams. Additionally by discovering risks, forming test ideas around those risks, and being observant when executing them. Not every test will be of the same quality, or be structured in the same way, so you need to remain vigilant. Itās up to us, as we test, to question what we have done and what we have learned, to see if assumptions were made or if details were missed. We can always learn and improve to deliver better testing.
Tests are the no of ways we can learn about the product , about its behavior or risks. In SD you know, you do, and then u should try different ways (which are tests here ) to know whatever u have done is correct to deliver.