When to Start Automation?

(Heather) #1

In a #TestChat we had about automation @rosie asked:

When is a good time to start test automation? (I often get people wanting/asking it as a priority and before any other testing has been done before)

This is a question I’ve struggled with in the past a lot.

In my first company I fell into the trap of “We can’t start automation because the product isn’t stable”. I then proceeded to toe a really fine line between the product being stable and there being too much of the product there. I was a lone tester so automating too late was going to be a time sink. As a lone tester though, automating early could have really helped me to use automation in the right way and potentially free up my time. (This was my first venture into automation).

In the next company, I was brought on pretty late in the development process (nobody to blame, it just happened that way). There was an automation suite that had been written months before the (now released) product had gone live. It, naturally, no longer worked. I decided to scrap it and try to start again because it was so outdated. It was then that I realised, automating at this late stage was pretty pointless. The product was massive! We also had unit and some API level checks.

In your experience: When is a good time to start automation? Why?

(Valek) #2

Just found an interesting article that may help to answer your question.

The challenges can be daunting, but the conventional benefits of automation are obvious: tests can be run faster, consistently, and they can be repeated as many times as needed with minimal overhead. But there are some other, subtler benefits worth bearing in mind.

Improved velocity – As stress levels rise towards the end of development, automation can provide peace of mind about the code quality. It acts as a kind of safety net. And when developers are confident that code changes won’t introduce lots of errors, they’re free to really focus on each sprint goal.

Greater coverage – Early automation can save a great deal of time down the line and result in much more comprehensive test coverage. By shortening the feedback loop, test automation boosts your chances of finding serious defects earlier and fixing them before they become embedded. It also frees up experienced testers to explore new areas of functionality and focus where they can provide the most value.

Higher reliability – Today’s business environment demands high quality software. Early test automation is the most effective way to build confidence in the software being developed, and the more comprehensive the testing is, the greater the quality. Automation also drastically reduces the risk of human error, ensures that key business workflows are fully covered, and minimizes the risk of critical bugs reaching production.

It may not be easy to implement, but test automation can deliver tangible benefits, and represent an excellent return on investment for any organization.

No Pain, No Gain: Early Test Automation Is Worth The Effort

(Chris) #3

Tool assistance is useful from before the product has finished an initial design until well after the last build has been shipped, up until around the last user dies.

If we’re talking strictly about large check suites then in my experience because they swell from the drippings of mindless responsibility they are often pointless even if the code is clean and the architecture is well designed, because they don’t check for anything of value, or check in a way that offers the value they claim to provide. “TestLoginPage”. Yeah, right, prove it.

But if you’re going to design worthwhile checks, not including checks just “because” but because you’re a professional making smart decisions about checks including their purpose and cost and value and expiry date, then it’s useful to start considering it before the first line of code is written. You may end up designing an internal API or other layer of interpretation to hook into to make the check suite worthy of its cost. You may change the technology you are using for your main product to suit your check suite product. What skills you have in your teams may influence your decisions. It’s more difficult to plug these things in downstream.

Starting a check suite project also needs a bunch of administrative and social payments. You may need a new environment, or a change to your systems that IT or Ops need to make. You may need a new monitoring system. You might have to get tool budget. You may need training for teams to get them used to the new tools. They’ll have to fold it into the build process or a CI system. Things will go wrong while people get used to the new systems and technologies and environments and how to use them effectively.

This administrative swamp scales with the size of its project. That’s why smaller tools doing specific things to make the product more observable or controllable can be as or more effective an investment - log viewers, alerting systems, database difference tools, scripted installers, VMs with snapshots, and more.