I've Made Huge Mistakes in Test Automation, so You Don't Have To with Butch Mayhew

@utchbe is taking to the TestBash New Zealand stage to talk to us about the mistakes he has made in automation so we don’t have to :wink:

If you’ve got questions for Butch about the talk, add them here :point_down:

Remember to :heart: questions that you like to show appreciation for others and help @jamesespie see what questions are popular to ask live.

1 Like

Flaky tests - especially the very slightly flaky that say, fail 1 in 50 times. Keep, or throw out or isolate until fixed, and why?


How do you track the bugs that were found by automated tests? tool?


Great talk! As you mentioned in your presentation, curious to know…how did you measure automation coverage and surface this up to your leadership team?


If you are building a test automation framework from scratch what is the absolute first thing that you build (apart from forming the test/automation strategy)

1 Like

Great talk Butch, what are the key must haves to be part of your test strategy?


Literally pasting these out of the readme on our project. Some of these things are specific to our project.

  • Before you begin, ask the question question if this test is better placed as an unit tests, an integration(api) test, or in fewer cases, an UI test.
  • Every test should be self reliant and able to run independently.
  • All tests should leave the session in the same state it found it in or better. By default UI checks assume an authenticated/logged in state from a staff admin user.
  • All tests should leave the data in the same state it was in.
  • Test automation should be boring. If your test is clever, it is too complicated and should be split into separate tests.
  • DRY is not always the best way. The flow of the test should be readable if multiple steps are needed.
  • Use the DataFactory to create/delete test data.
  • Use the DataFactory to make assertions on the DB along with assertions on the UI.
  • For UI specs every page there is an action or assertion, there should be a page object created.
  • When creating page objects avoid using xpath at all costs (if there is no other way, dev work needs to be done)
  • Only the feature in test should be in the it block. All other steps should be in the before block. (ruby specifc)