Invalid password vs empty password in login form

First I’ll just say if you stop looking at testing as a series of lines of checks and more as an exploration based on your understanding of the product guided by risk you can achieve more with much less writing, reduce the cost of upkeep, avoid going over old ground again and again, plus it’s more engaging and fun and you might find some really good problems that the test cases don’t account for. You’ll find that you can do more interesting test cases if you don’t write them down in that way. Move towards a checklist of suggestions rather than a case-by-case basis. This deformalisation will let you do more testing and less writing with more freedom to ask interesting questions of the product. What you actually consider when you test is, let’s face it, not determined by what’s on the test case list but with your real-life interactions with the product.

You pose a lot of interesting hypotheses. If all of our hypotheses about the product were true then there’d never be any bugs in it - after all the product is developed to work and one big hypothesis is that everything’s okay because we made it to be okay. This is clearly not true, so we must learn to distrust our hypotheses and test them to turn them into more robust theories of the product.

To the specifics:

Isn’t it just a repetition and unnecessary additional test case when considering blank password, if we already consider the invalid one?

Maybe. Perhaps some part of the code that expected to receive an input simply falls over when there’s no input provided. Perhaps it is supposed to display a different message…

( We don’t consider error validation messages)

Don’t you? Because if it was full of swearwords or said something dangerous I bet you’d tell someone. If it had a spelling mistake in it would you keep it to yourself? Because that would erode the user’s confidence in your product from the very first time they interacted with it. And it’s so easy to take a cursory look at - provided you don’t have to write down the results every single time, of course, but that’s the curse of explicit formal test cases.

Isn’t blank password and invalid password processed in the same way?

If you don’t know then perhaps you should check to see if they both work. Or perhaps you could dig into the code (or ask someone else to) to see how they are processed so that you can better judge the risk. Even if the code treats them identically, and the code that runs the code treats them identically maybe the database doesn’t. Maybe some other third party system doesn’t.

I mean: is the scenario that we are unable to login with valid username and invalid password, but able to login with valid username and blank password even possibile at the same time?

Firstly, yes. Perhaps it’s possible to create a user with a blank password, or update a user to have a blank password, or edit the database to have a blank password, thus making a blank password a “valid” one.

Secondly even given the assumption that they are mutually exclusive scenarios that doesn’t necessarily mean that they are processed identically. Perhaps the code that protects against things like SQL injection attacks isn’t prepared to receive blank inputs. Your system should probably be using prepared statements, so when that data is fed to the DB how will it react? Does the prepared statement data code work with blank inputs? Maybe it’s some DRY code somewhere that interprets data types on the fly for some ungodly reason and it passes it as some kind of null value. Or, heaven help us all, you have some flimsy copy-pasted SQL injection filtering code full of holes that falls over. Your test results could help trigger an investigation to find out all of this kind of thing and potentially expose enormous design flaws. Or playing with the ideas could give you a lot to think about with the login page. You could look at the DB and see what you can do with it, and what you can get away with.

The important thing here is, I think, that you’re splitting up the cases into these equivalence sets that are rife with assumptions and abstraction leaks and treating them with much more respect than they deserve. Making groups of ideas like this is useful and important, but you must remember to treat them with much humility. For a start there’s everything you don’t know or haven’t thought of. Also everything about what you say might be true, but when you turn that into action (only checking one of them) you fail to find something important because you limit your actions and thought. You also never think deeper about the product or get feedback from it directly that makes you think of other problems that you might want to explore. Luckily you’re already doing this because here you are on a forum questioning your partitions, for which you should be proud, but I think it’s worth stating explicitly because it’s a useful way to think.

Finally I’ll say don’t worry about being confused. If you go through a career in testing without ever feeling confused you are terrible at your job. Ask the questions, try to find some clarity, and when you come to the overwhelming realisation of the sheer complexity of even simple computer programs and can wade in with tools, experience and knowledge and cry like three times maximum then you will know you have transcended the comfort of thinking in assumptions into the glorious world of thinking in assumptions but knowing that so you’re more careful.

3 Likes