Testing email confirmation

Does anyone have any experience of testing a system that sends a verification email to a user?

Iā€™m trying to think of the best way to go about this. I have an automated journey that creates a new record for a user and we are currently looking to add an email verification step into this at commencement of the journey.

Iā€™d like to imagine there was a way of mocking this part of the journey, ideally iā€™d like to avoid having to hardcode a permanently valid email address.

Any ideas are welcome :slight_smile:

3 Likes

Is your question about testing the confirmation system, or about automating a task?

If itā€™s the latter then it depends on what tools/tech/language youā€™re using, what level youā€™re working at, and how this check supports your testing (what exactly youā€™re looking to check). You could code a UI check end-to-end that creates a known user record with a known email and then have a service that (based on what triggers your verification emails) pings emails back to your test system, where the contents are read. Youā€™ll need some sync code for this, obviously, youā€™re testing a distributed system. This checks that it was received (in this particular setup) and you can check anything you like about the email contents - if thatā€™s what you want to know. It doesnā€™t check a whole lot of other things, but thatā€™s a matter of choice on your end. Maybe you donā€™t even want to check that many things.

If itā€™s the former then youā€™re asking a wider question. Why are you verifying the email, and how important is it? Some stuff might include:

  • Security. Can I get around the verification step? What can I acheive without verification? What if I use a disposable email address, do I need to take steps to block them? Can my system be used to spam someone by requesting verification emails over and over to an email I donā€™t own? Can I DoS the verification system? Can I SQL inject the system via the user record? How long is the validation link valid for? Does a new link invalidate the old one? Can link URLs be reused - how are they tied to the user and what does that reveal about the system? Can I use a bot to create accounts? Can I verify other peopleā€™s accounts by editing the return URL? What does the return URL expose about our API, and how can that be used?
  • Scalability. Does it work under stress? What about under load - what happens, for example, if I break the message queue that sends the emails? What if the email server goes down - whatā€™s the user experience?
  • Functionality. Does it send? Can I receive the verification? What if the link is clicked twice, or 20 times in frustration? What if someone spams the ā€œresend emailā€ button? What if I refresh the page that caused the link to be sent - will it quietly send another and invalidate the last one?
  • Usability. What if the email goes into spam? What if itā€™s not received quickly? Do I have to type in verification numbers manually, which can be a pain, especially for people with related disabilities? If some of my customers are not terribly tech savvy then have I given them suitable instructions to verify this email? As a user is it worth the effort of verifying my email?
  • Secondary Purposes. What can go wrong when we assume that the email really is valid when itā€™s not? When should we use it as the only means of communicating something, such as a policy change? What is the email address actually used for, and what if it starts bouncing mail?
  • Up/downstream dependencies. What if I use a really long, badly formatted or otherwise invalid email/name/body string? Are the email sender, validation system, database and UIā€™s interfaces all data compatible (simple example: email too long in UI to store in DB). Where is the data stored and used, and what can I get past the filters?

And so on.

PS. There are third-party email verification tools abound online, I donā€™t have experience with them, though.

1 Like

Thank you for the in-depth response. Its about both I suppose, the idea behind the question was about strategies for integrating with automation, I wasnā€™t very specific.

A confirmed email will be required for the user to continue through a process, this is currently not included in the automated regression. Iā€™m not concerned about the ā€˜whatā€™ to check for this bit of work for now, only the method of moving through the journey.

With the intent being essentially to simulate a user responding to an email confirmation email, the question becomes how much would I want to wrap in automated tasks and how easy would that be. It would be straight forward enough for the script to start on the users confirmation landing page passing in fuzzed auth details and continuing from there but Iā€™ve then bypassed the functionality part of the email client that you rightly mentioned above.

Iā€™m coming to realise however that this question may be beyond the scope of such a forum due to it, as you say, depending upon various things. Namely the chosen email client and whether there are any hooks/libraries that would let me interact with it in a test in the chosen language (currently js but may change to Java).

Primarily, I was simply curious if anyone has had to implement such a system test before.

Iā€™ve always done the email verification part manually, and used workarounds like running SQL updates to get the result I needed where I had to automate user registration.

That said, Iā€™ve used Guerrilla Mail as my ā€œfake-realā€ email tool of choice for a while now. Itā€™s an HTML interface where you can use a chosen email or the scrambled version GM automatically generates. The emails themselves are deleted after an hour, so itā€™s relatively secure, and any address with the right domain (they default to @sharklasers.com) is automatically valid.

If you absolutely had to you could probably use that to pull the information, but honestly, the more third parties you run through UI automation, the more fragile your tests are going to be.

3 Likes

I immediately think ā€œlikelihood of failureā€ is low? In which case, and you allude to this, mock it. Depends on your stack but should be as simple as Kate suggests, stub with a script (SQL update) to service your automated test suite.

If there is risk of failure, perhaps it stacks up as a high priority to fix and subject to manual checking until that is so.

1 Like

I usually find any system which sends a verification email has a configuration setting for the email server. All the systems which I have been testing can be configured with SMTP as their email protocol. In these cases I just find a simple SMTP server, configure the system being tested to use the simple SMTP server I set up.

For example, you could use https://github.com/Nilhcem/FakeSMTP. This is a Java project. It creates a jar file that you can run on any system. You then configure the system under test to use FakeSMTP as your email server.

I usually have the output directory for the FakeSMTP be something my test automation can access (an UNC share if it is Windows or a SAMBA share for Linux/macOS). Then my test automation can access the file share and read the emails from there (no need for an SMTP client to retrieve the email).

Another one, with a web interface, might be https://github.com/djfarrelly/MailDev. I found this by searching for ā€œSMTP server with web interfaceā€.

It has been a while since I did this kind of testing. So I donā€™t remember the exact SMTP server I was using.

Iā€™ve also set up SMTP servers on UNIX systems but configured them to queue up the email in a spool then never send it. The emails would then sit in a directory on the server. I could ssh to the server and read the email. A little more complicated to set up if you havenā€™t done this sort of thing before (used to be a UNIX system administrator; so setting up an email server was second nature).

1 Like

Just to say the responses on here have been good :slight_smile:

1 Like

I will add this on - as I donā€™t see it mentioned yet.

Manually testing email - Iā€™ve had good success with MailTrap https://mailtrap.io/

Basically, you configure your app to use it as the SMTP server in your Local/Dev environment, and it captures ALL mail being sent, to any email address. OR you can configure it it lots of ways to route email based on address/domain.

Iā€™m not sure how well this integrates with automation, for but manual / exploratory testing itā€™s great!

1 Like

Mailtrap is really good for automation as they have a nice, easy to use API. Iā€™d definitely recommend this as an option if you have control over the SMTP settings for your project.

Other options Iā€™ve seen are usually paid for such as Mailinator but the pricing was a bit steep.

I also had a go at using Gmail via IMAP though you can get held up by the delay of the mailbox refreshing and I think Gmail have some T&Cs which may rule this out if youā€™re doing anything at scale.

I covered some of the basics here: https://amido.com/blog/how-to-automate-your-email-testing
Updated link to repo https://bitbucket.org/raubreysmith/mailtrap-client

1 Like

Iā€™ve just used Gmail with their +syntax ā€“ Thereā€™s a handy gem (Ruby) that allows me to check/retrieve/delete messages.

1 Like

Iā€™ve used https://www.mailinator.com/
You could send an email to any random string@mailinator.com, and then find the email on mailinator by searching for that random string.
The enterprise plan makes it secure with your own domain (others cant see the emails), and also gives you access to the api, and access to libraries in various languages that interact with this api.

Email feature is one of the most important feature of every application. It provides us benefits to get members notified for updates, approvals, tasks and etc. Its importance is very high in e-commerce domain.

As this is very important feature, hence its testing also become very important for every software testing services provider. We need to consider following factors while testing email confirmation:

  • Ensure that content is accurate and reflecting correct information.
  • Content of Email verification code should be properly visible.
  • Confirmation Link/button should be properly working.
  • It should take the user to correct portal.
  • Email should be successfully verified when it reached the application page.
  • It should not take much time to reach application.
  • Appropriate fonts and texts should be used in the email.
  • Application links should be correctly displayed and working.

Hope this information will be helpful for you.

A similar topic to this came up on Slack this week and I saw a few more tool suggestions so figured Iā€™d add them here :slight_smile:

1 Like