Context
The context in which we perform testing defines how useful that testing will be. Login functionality may be important for security, or it may not. It may be important that it work on multiple platforms or just one. It might be browser-based or it might not. Each of these vastly changes the direction we use our resources. We could test that it works on bespoke touchscreen devices on the factory floor, but only if it’s to be used in such a way. I will try to deal with possible ideas at a higher level that might be applied to multiple contexts, but anything I say may be entirely non-applicable depending on the context.
Scope, Limitation and Exploration
The constraints of test cases is that they specify extremely narrow concepts, and expanding them into real-world testing makes them very time consuming to write out. So I will instead write charter-style points for investigation, and questions that I might ask.
The issue is that if we expand test cases with heuristics we can do so indefinitely. This creates a new issue of scope. Choosing to limit ourselves to the scope of the test case means that our job is already done. If we examine it with respect to, say, CRUD, then we might decide that a case fails to cover delete actions - but should it? If we are writing cases (which I would not normally do) then what we’ve discovered is that there should be additional cases.
I’d say that these test cases are perfectly designed for exactly what they do, and badly designed for everything they fail to do. These cases are test artefacts for communication, and if they achieve that communication to the recipient, with all their contextual and domain knowledge to a sufficient degree of specificity then that case has achieved its goal. We can say that the login case doesn’t cover failed logins or input field testing or security testing or password reminder systems but the case doesn’t specify that it’s trying to achieve those ends.
I think that this issue isn’t that the cases are badly designed, but the testing may be badly designed. If we are to explore the idea of logging into a system then perhaps one written example of “don’t forget to check that you can log in okay” is a fine thing to write down. If there’s more to be done it may be contained in other cases. If there’s a question of a lack of understanding about what needs to be done that may be covered in design, user feedback, project management and resource allocation, and of course the exploratory nature of all testing.
I’d say that the vagueness or over-specificity of a test case is a matter of context. If something does not matter or we need more information before we know what does matter then testing under vague concepts is wise. If we need to really nail down an idea because it’s important, or we need to describe it in detail because it is large, complex or any other testability issue, then specificity may be a good tool for that.
Test Case 1: Login Functionality
Steps: Enter valid credentials and click “Login.”
Expected Result: The user should be logged in successfully.
Examining the Test Case
This is to do with login functionality, includes users of some kind, credentials of some kind, and the result of “logged in successfully”. We can here ask what we are logging into, what the credentials are, what makes them valid, what a user consists of, what logged-in entails and what it means for it to happen successfully and be successful.
Logging In
In order to log in the person must be able to use the interface to do so. This will depend on what that interface is, but there is an assumption in the case that this is possible. One example would be a website with a username field, a password field, and a Login button. That also implies probably a computer mouse and keyboard for inputs, or a touch screen in the case of common mobile devices.
This brings up another question of accessibility. If this system needs to be accessed by people with impairments that may affect their ability to read instructions, see the fields, remember credentials or provide inputs. They also need to own any third party system, so if you require a computer or smartphone to use a system then you exclude anyone who does not own a computer or smartphone (or at least inconvenience them) - this may be applicable where other systems are required such as for a bank that may need someone to prove their identity over a phone call for telephone banking. Is it compatible with third-party accessibility software, such as screen readers?
Credentials
A credential, in the case of a login system, is proof of identity and granted authority. Anything that can identify a person and whether or not they have the authority to enter the system (and therefore what they can do there).
Usernames and Passwords
Assuming a username and password there’s some concern over what valid credentials might mean. One factor may be the types of characters allowed. If we allow someone to create a user then the set of possible credentials for a login page input must contain that set. If we allow 100 character usernames, the username field must be able to accept 100 character usernames. If we permit certain email formats then both must be covered. If we sanitise the inputs we have to do it for both cases. If we also have an API user creation/login that has different input limitations that must be compatible with the login page. We are playing here with the concept of what is technically valid to the system in different places and trying to find a way to create “valid” users that cannot login.
Validity can also change. If we make changes to the rules on how we make usernames and passwords we have to cover that in all cases, and also have that be backwards compatible with existing users. We need to look for dependencies up and down the workflow.
There’s also matters of taste. If a credential is offensive and will be displayed for other users that may be of concern. A typical example is an online multiplayer game where usernames must be moderated, and some words excluded when creating a user. A username may be “valid” in terms of “the system permits its creation and allows for it to be used to log in” but not “valid” in terms of matters of user policies and agreements, legality or offenses to common decency.
Other Credentials
It could be that this case is linked with the others, such that it has a checkout process, and is therefore quite likely to be an online shopping website. It might be a custom system on a nuclear submarine using keys or RFID tags or fingerprints or face scans.
Other examples include public key certificates, X.509 certificates and the like.
For hardware credentials like RFID that may generate its own testing, or that may be simply trust that we put in the third party that made them.
An interesting case would be one of face scans or voice identification. What constitutes a valid credential - in the sense of one that represents the identity of someone correctly - is more difficult to understand. It may be that what “valid” means actually lies with some third party software in a face scanner, and it may be necessary to test that without understanding its inner workings, just through its inputs and outputs.
If there are multiple credentials, such as a retinal scan and a fingerprint and a numeric key code there’s some concern about timings. What order can these be entered? If one is entered and the other is not entered is there a time-out system to forget the attempt? If someone enters a retinal scan and fingerprint then realises they forgot the keycode and wanders away is there something to stop the next person accessing with just the keycode? Again these are technically valid credentials to the system, but they are not valid in the sense of identifying a user.
Authority
I think there’s some useful thinking to be done about why we believe this person to be who they say they are, what the impact is if we are wrong, and how we might mitigate that.
One interesting and common example is two factor authentication which generates its own set of questions regarding logging in. If someone with 2FA turned on follows the steps they will not be logged in, they’ll be prompted for a security code.
Valid credentials might also become invalid if the user is, say, temporarily banned. The case fails for a game, logging in with valid username and password where someone was banned for a week for bad behaviour. Then they should have valid credentials but be unable to log in. APIs sometimes restrict or ban users who go over their allotted rate and access limits. Users might be banned, shadowbanned, or deleted. It’s worth considering the possibilities of what can happen to the concept of a user and how that would affect the validity of their credentials and the effects of functions like logging in.
Sometimes people reset their password using their email as proof of identity. Sometimes their password is reset for them, such as if a company experiences data theft and resets everyone’s password as a matter of security. Then “valid credentials” in the eyes of the user are different to those of the system, and they must be informed of the change.
State
One thing that this case probably assumes is that there’s no current user logged in already. We assume that we have access to the system, nobody is logged in from a particular device and then attempts to log in. What if we access the login page (or API or other interface) while someone is already considered logged in.
Another concern is the state the system is in. If it’s under stress, if any part of it has failed, and so on. If the database cannot keep up with demand then what should happen? Or if the site becomes unavailable while the login page is open? Or indeed then after login - are they logged out?
Logged In Successfully
Being “logged in” is often more than a matter of access to a system, but to parts of a system. There’s often admin users that have more access than others, or users that have paid for certain functionality. It may be that this test case doesn’t intend to cover these points, but it’s worth considering that it’s part of the “logged in successfully” experience to have accurate levels of access to a system and its data.
When someone logs in it may be that there are things other to the web page displaying data that are important. It could be that the database is updated with the last date and time of login, or increment the number of logins, or add to a list of times someone is logged in. This can be useful for metrics about the usage of the website, for performance testing, for non-repudiation issues in security auditing and so on. It may generate a security email to the user saying they were logged in, especially from an unusual device, IP or geographical location.
It may have far-reaching implications, such as logging into a vehicle control panel that permits the vehicle to move so that it knows who is controlling it. It could be said that a car key fob that permits the car to start is a login system with credentials that allows the driver to be logged in successfully (able to start and drive the car).
To login successfully might entail that the user has access from multiple places over time. It could be that success means to be logged in for a week, if I access the site from the same browser. It might even mean that others cannot login, if part of some sort of lockout safety system in an industrial context, or to lock documents or files against change, or single-user access like remote desktop systems. Perhaps it would log out other users.
System Elements
It’s worth considering that logging into a system often entails a lot of detail about what that means. Communication to and from a database, generating a web view, setting a cookie, checking a certificate, and each of these play their part in the concept of “logged in successfully”. Perhaps we could say that it’s not successful if the cookie is not formed properly, maybe with an incorrect expiry - this would not be obvious from a browser, but is still part of the logging in system and could generate bugs if ignored. We’d need to understand what the system actually does, what functions and in what order. And then we could also consider race conditions and other synchronisation issues.
Performance
While performance isn’t mentioned in the case if the whole thing played out but took 15 minutes to occur I would feel that is too slow in most cases. If the user gives up and walks away then that is not a successful login worth having, even if it’s eventually successful. For e-commerce that login time is extremely important, where seconds of performance difference can have a big impact on sales.
Also if the system is slow to allow inputs, slow to give errors or feedback, slow to issue 2FA codes these all impact the login experience.
Heuristics Used
A lot of these are from experiences I’ve had with systems, learning about how various parts tend to work and failures from other concerns and so on. All heuristic oracles.
I also looked into the HTSM for some inspiration, and did some research into credential types.
I don’t know that I’ve improved the test case, as I mentioned towards the start. But I have given some concerns around the idea of a valid login, and that would certainly improve the testing in terms of concepts, coverage, techniques and other ideas we can use to find problems that the context suggests are important.
Your Challenge
Poorly Designed Test Cases
Your Task