If when validating a work item and one of its acceptance criteria fails, but after discussing this failed status with your Product Owner she decides to overlook for now this one acceptance criteria (because it is a ārare use case, after allā), what test status would you give this "failed but now dropped " test scenario / step: Failed, Blocked, or Untested?
I do not want to create a test report that has any red / failed status, given this particular circumstance. (Note that I have fully documented the test run with these details, for the sake of posterity ā in the event this use case returns to us later in the form of a feature enhancement request or customer complaint.)
My solution was to mark the test step as āFailedā, but mark the test run as āPassedā; while including the POās commentary in the test runās Actual Result section.
I try to avoid any test reporting with hard wired status.
Itās a limitation, which hides important information.
There are more status than any developer could predict.
e.g. āI found 3 bugs, had to work around 2, still have to investigate 1, and currently pause because of an interruptionā
I think a lot depends on your corporate culture and the nature of the product under test.
If you are a small company and you have a good professional relationship with the PO; and/or if your product is not public-facing and its functions are not critical to life, limb or prosperity, then by all means mark the test run as Passed but with reservations, or with non-material failures (and, of course, record your findings and, if theyāre not recorded anywhere else, the conversations with the PO, their decision not to address the failure before release, and any reasons they gave).
If, however:
you are a large organisation and these decisions are a long way down the food chain from senior management; or
your product is public facing, and failures would impact public health, safety, or their livelihoods or savings
then you must mark the test prominently as FAILED and make it clear what recommendations you gave the PO and that the decision to release was not based on your opinion. I know weāre not, as a testing community, in favour of testers āsigning offā on products before release, but thatās not the way a lot of the public and the judiciary see it, and if the product is not, in terms of the testing, āfit for releaseā, then any blame (and Iām sorry to make it into a blame game) that may accrue for the consequences of that release shouldnāt come to rest on your shoulders. Talking to a hostile media corps should not be in any testerās job description.
A chestnut of a question, and after a few years, I have to agree with @sebastian_solidwork , itās not binary. Also with @robertday , how for me, the only interesting tests, are ones that failed. Iāve been trying to read a horror non-fiction work about a poorly executed IT project, itās a big book and curiously most of the failures and reasons the system cost so much was down to humans not paying attention to testing. Iām about half way through only, because each page reveals how important communication is, but also that the testers, never ever want their test reports to become entered into a court case. That just should never happen.
There is no fail, there are only unknown product risks. A certain large company I worked many years at used a āpassed with issuesā overall status. Which was bizarre, and we gradually changed that messaging to āit failed, lets triage the big ones and release once they are fixed.ā Failing tests are noise in a system and generate unnecessary interest, they often do need removing if they cannot be manually reproduced. One evil way of doing this is āignoringā the test, and that bring in the 4th kind of test result, skipped. Skipped tests can live in your system for ages, and are more benign even than the failed test. Yes, skipped tests are part of your āunknown riskā!
@yelbmik1
If we found failures the status is Failed. The test result is as it is.
Itās something different when the PO decides to deliver despite that.
Your company should not mix test results with management / PO decisions.
Donāt fear to create test reports with red / failed.
Maybe your company needs in addition another column (or a new report) to show the POs decision about your result.
If this scenario happened to me, Iād remove or amend the acceptance criteria. As clearly, itās not valid, otherwise weād immediately be fixing it. But Iād make sure that the PO understands this. If they want this to be addressed they need to make sure to add a ticket for that to happen. Iād be cautious of āoverlook this for nowā,
Iād be sure to share my narrative with the PO, explains anything I could foresee happens, knock on effects, and if they are still happy to not address it, Iām happy with that.
If you have to, Iād do what youāve done and make sure this decision is documented on the ticket somewhere.
I probably wouldnāt keep a test case/scenario for it, I could maybe automate the scenario, s Iām told if this behaviour ever changes, on the basis it was accepted behaviour and if it changes it may no longer be. But Iād need more context to make that decision.
Iād mark it as failed, and keep the notes, and have it be red as thatās what happened. If a PM or someone else decides to go on because itās not important, thatās fine, but the test results are that it failed.
This has the extra advantage of making sure this doesnāt get lost in the noise, and someone can triage in the future if this is an acceptance criteria that needs updating, or if there are bugs that need to be reported/fixed. Having it visible as a fail and then later flipping to a pass is also useful, as it would highlight which version/release fixed it.
At the company where I work, a fail is a fail and is expected to be marked as such. Is it the test that is in error or out of date? Is the acceptance criterion too strict? Is the failure a trivial matter that should not be a blocker? Could be, but if the test does not pass as written then fail it ā and document your decision in a way that will be helpful to engineers, fellow QAers, and the product manager. Most of our failed tests do not block the release, nor should they. YMMV of course.
