I've Made Huge Mistakes in Test Automation, so You Don't Have To with Butch Mayhew

Archive
,
1

@utchbe is taking to the TestBash New Zealand stage to talk to us about the mistakes he has made in automation so we don’t have to :wink:

If you’ve got questions for Butch about the talk, add them here :point_down:

Remember to :heart: questions that you like to show appreciation for others and help @jamesespie see what questions are popular to ask live.

1 Like
2

Flaky tests - especially the very slightly flaky that say, fail 1 in 50 times. Keep, or throw out or isolate until fixed, and why?

2 Likes
3

How do you track the bugs that were found by automated tests? tool?

3 Likes
4

Great talk! As you mentioned in your presentation, curious to know…how did you measure automation coverage and surface this up to your leadership team?

4 Likes
5

If you are building a test automation framework from scratch what is the absolute first thing that you build (apart from forming the test/automation strategy)

1 Like
6

Great talk Butch, what are the key must haves to be part of your test strategy?

4 Likes
7

Literally pasting these out of the readme on our project. Some of these things are specific to our project.

  • Before you begin, ask the question question if this test is better placed as an unit tests, an integration(api) test, or in fewer cases, an UI test.
  • Every test should be self reliant and able to run independently.
  • All tests should leave the session in the same state it found it in or better. By default UI checks assume an authenticated/logged in state from a staff admin user.
  • All tests should leave the data in the same state it was in.
  • Test automation should be boring. If your test is clever, it is too complicated and should be split into separate tests.
  • DRY is not always the best way. The flow of the test should be readable if multiple steps are needed.
  • Use the DataFactory to create/delete test data.
  • Use the DataFactory to make assertions on the DB along with assertions on the UI.
  • For UI specs every page there is an action or assertion, there should be a page object created.
  • When creating page objects avoid using xpath at all costs (if there is no other way, dev work needs to be done)
  • Only the feature in test should be in the it block. All other steps should be in the before block. (ruby specifc)
2 Likes