How to test error messages

(Heather) #1

Today I entered my email address incorrectly into Linkedin. I was shown the following message:
image

Having read some of Troy Hunts site have I been pwned, I see how many Linkedin accounts are listed (it’s a scary number). If I was looking at the story for that functionality from a security perspective I’d say that’s not a really good idea. If I was looking at it from a UX perspective I’d say it’s a nice and friendly way to let the user know exactly what’s wrong based on the blog post How to Write a Perfect Error Message.

Equally, I’ve filled in a page long form of different input fields, clicked submit and been told “Oops something went wrong”. Great. Thank you, that’s super helpful :confused:

How can we test error messages? How can we ensure that the user is getting enough helpful information but we’re not giving away the wrong kind of information? I’m interested in error messages of any kind (API, UI, etc) I’ve just used the login page as an example because it happened to me today :slight_smile:

Ask Me Anything: Security Testing
(Chris) #2

Hi, I’m currently testing an API and I completely agree we need to test/check error messages.

Too often things go wrong and you’re left with meaningless UI, API or worst of all, poor logging.

As well as checking things work as expected, I have a substantial number of tests where I submit invalid, incomplete requests and check the response to ensure I get the expected, and hopefully, meaningful message.

(Nina) #3

I usually use my gut/brain to see if the error messages make sense.

Most of the time though it can be hard since we as testers aren’t in charge of the copy. In my current position it’s almost impossible for me to change the copy (I work in an agency so us in the development team usually don’t have direct contact with the clients), but when you work closer to the product (as in, working in a product company) I think it can be easier. If something doesn’t make sense, then it’s something that (imo) should be highlighted.

About how I test it:
Something to think about is the target customer. Is the customer tech-savvy? Yes? No? Will they be able to understand the message? If not, I usually think of a person that I know that isn’t tech-savvy and question myself if I would understand the message if I were in their shoes. If they’re supposedly super tech-savvy, then they might want something that is more relevant to them so they can understand what the problem is (ergo, their side or client side for example). I also like to take the end-customer’s side on privacy. If it’s something that I know will potentially be harmful for the customer, then I advice against it (like your above example with Linkedin - we have actually recently recommended against having an error message like that to a client :slight_smile:).

Hope this was useful!

(Patrick) #4

We have a lot of error messages that display to customers on our website, and in our app, at different times during their application journey

  • Apply page (invalid characters in names, DOB outside of our accepted age range for a loan, invalid numbers/email addresses, as well as blank fields on page submission as all these fields are compulsory)
  • Budgeting page (any incomplete fields, any fields where the value is lower than expected amount per their credit file, or national household averages)
  • Repayments page (invalid card/bank details, empty fields)

In terms of testing, it is literally just a case of using heuristics such as naughty strings, bad data, etc to ensure the messaging makes sense from a human aspect (ie; “DOB field contains invalid characters” isn’t user friendly, but “You must be between 18 and 75 to apply for a loan” is). Although this relies on the tests being done manually, and not automated.

I’ve also been testing error messages for when something goes completely wrong - in those instances, a lot of it is down to working out where the break points are, and working with a dev to simulate each breaking in turn so you can then see how your device/website/app handles it. So in those cases, some forward planning and modelling is useful.

IE; connection issues - if its the device’s connection, you can use airplane mode to simulate the issue and test the response. If it’s a connection to your internal servers, you can “break” that connection on purpose and see the results.

(Heather) #5

Out of interest, would that display for all errors in the DOB field? It makes sense to me that it would but just curious. It’s not giving a heap of information away but it’s pretty user friendly (I think anyway).

(Tobias) #6

I’m curious about that as well - especially because I believe that this message shouldn’t be displayed for all errors in the DOB field.

Imagine entering the DOB in the wrong format (24.12.2017 instead of 2017-12-24). If the software isn’t able to parse it I’d expect an error message explaining the format I’m supposed to use - ideally with an example using the date of today.
Just telling me that I’m either too young or too old doesn’t feel helpful in this case.

(Vishal Dutt) #7

Testing error message needs a scenario based approach to figure out when the system will return an error. Every software testing company should focus on below mentioned ways to encounter errors in application.

  1. Error messages from DB: Sometimes, we get an error code from DB while performing some action. It means the form data is not handled by the DB server.
  • So, while submitting a form provide boundary values in all field. It will ensure that lower end and upper end limits are accepted by DB tables. Sometime the length of the DB tables are not set appropriately thus resulting in error code.
  • Data is not matched with the expected pattern of the field. For example, in date range provide special characters or spaces.
  1. Error messages from Web-server (HTTP error): When user send a request from web-browser then the server responds with a status code. If there was a error, you can get additional information about the error.
  • HTTP 403: It comes when server is not reachable. This is a permissions issue. Try to create too many connections at the same time. Perform a task that is based on SSL layer and needs authentication.
  • HTTP 401: It comes when credentials are not matched for the target resource. Provide invalid username and password to replicate this issue.
  • HTTP 500 : This is an unexpected error and can be checked after making invalid configuration.
  • HTTP 404: This error message is shown when a site or folder on a server are requested but cannot be found at the given URL. So, while testing an application provide invalid path of an object to replicate this error.
  1. UI errors: Sometimes error is encountered based on the data provided on the page. So, it is validated with some parameters that a field accepts a particular value.
  • Provide space or just hit enter key after bringing the focus in the field.
  • Use Keyboard instead of mouse and press tab key to navigate through the fields.
  • Some fields accept blank values which later on create issue in DB.
  • On providing large values in field UI gets distorted which destroys the layout of the screen.

Hope this information is helpful for you.

(Ihor) #8

Easy. These ones must have requirements and should be tested as other functionality. These are not negative tests, but positive tests on functionality which returns errors.

(George) #9

This is an interesting topic, probably because there can be so many situations which require error messages to the user. I’d like to add two observations:

  • Some errors resulting from user actions (e.g. configurations and inputs) cannot be avoided completely. The error conditions must be detected and the messages displayed. But as vishaldutt pointed out, other errors (say from the database) can reflect inconsistencies in the design. These are candidates for fixing—because an avoided error message is best, and a user error message is likely a better UX than a database message. Finally, if error logs are available to testers, they can be a big help.
  • Often error conditions can be simple to set up, with a single value out of range, two inconsistent configuration values which are not valid together, and so on. It is important not to test more than one error condition at a time because they can mask each other. The detection of any of the error conditions may prevent the expected processing of another. It all depends on the details of the design.