When to Start Automation?


(Heather) #1

In a #TestChat we had about automation @rosie asked:

When is a good time to start test automation? (I often get people wanting/asking it as a priority and before any other testing has been done before)

This is a question I’ve struggled with in the past a lot.

In my first company I fell into the trap of “We can’t start automation because the product isn’t stable”. I then proceeded to toe a really fine line between the product being stable and there being too much of the product there. I was a lone tester so automating too late was going to be a time sink. As a lone tester though, automating early could have really helped me to use automation in the right way and potentially free up my time. (This was my first venture into automation).

In the next company, I was brought on pretty late in the development process (nobody to blame, it just happened that way). There was an automation suite that had been written months before the (now released) product had gone live. It, naturally, no longer worked. I decided to scrap it and try to start again because it was so outdated. It was then that I realised, automating at this late stage was pretty pointless. The product was massive! We also had unit and some API level checks.

In your experience: When is a good time to start automation? Why?


(Valek) #2

Just found an interesting article that may help to answer your question.

The challenges can be daunting, but the conventional benefits of automation are obvious: tests can be run faster, consistently, and they can be repeated as many times as needed with minimal overhead. But there are some other, subtler benefits worth bearing in mind.

Improved velocity – As stress levels rise towards the end of development, automation can provide peace of mind about the code quality. It acts as a kind of safety net. And when developers are confident that code changes won’t introduce lots of errors, they’re free to really focus on each sprint goal.

Greater coverage – Early automation can save a great deal of time down the line and result in much more comprehensive test coverage. By shortening the feedback loop, test automation boosts your chances of finding serious defects earlier and fixing them before they become embedded. It also frees up experienced testers to explore new areas of functionality and focus where they can provide the most value.

Higher reliability – Today’s business environment demands high quality software. Early test automation is the most effective way to build confidence in the software being developed, and the more comprehensive the testing is, the greater the quality. Automation also drastically reduces the risk of human error, ensures that key business workflows are fully covered, and minimizes the risk of critical bugs reaching production.

It may not be easy to implement, but test automation can deliver tangible benefits, and represent an excellent return on investment for any organization.

No Pain, No Gain: Early Test Automation Is Worth The Effort


(Chris) #3

Tool assistance is useful from before the product has finished an initial design until well after the last build has been shipped, up until around the last user dies.

If we’re talking strictly about large check suites then in my experience because they swell from the drippings of mindless responsibility they are often pointless even if the code is clean and the architecture is well designed, because they don’t check for anything of value, or check in a way that offers the value they claim to provide. “TestLoginPage”. Yeah, right, prove it.

But if you’re going to design worthwhile checks, not including checks just “because” but because you’re a professional making smart decisions about checks including their purpose and cost and value and expiry date, then it’s useful to start considering it before the first line of code is written. You may end up designing an internal API or other layer of interpretation to hook into to make the check suite worthy of its cost. You may change the technology you are using for your main product to suit your check suite product. What skills you have in your teams may influence your decisions. It’s more difficult to plug these things in downstream.

Starting a check suite project also needs a bunch of administrative and social payments. You may need a new environment, or a change to your systems that IT or Ops need to make. You may need a new monitoring system. You might have to get tool budget. You may need training for teams to get them used to the new tools. They’ll have to fold it into the build process or a CI system. Things will go wrong while people get used to the new systems and technologies and environments and how to use them effectively.

This administrative swamp scales with the size of its project. That’s why smaller tools doing specific things to make the product more observable or controllable can be as or more effective an investment - log viewers, alerting systems, database difference tools, scripted installers, VMs with snapshots, and more.


(Doug) #4

If possible begin with happy path/smoke tests as soon as possible.


(Joe) #5

In the context of writing automation to support the development of APIs, I advocate starting as soon as there is code available if not sooner. APIs are a special breed of software products that just scream for the right level of automation.
In a recent API project, we were able to use SpecFlow to automate the GWT. The completed tests could be used by developers to guide them as well as help them know when they are done. Over time, the automation was used as the regression suite. Supplemented by a healthy set of unit tests, automation results (both unit tests and the SpecFlow tests) provided the team with high confidence that our products operated as expected.


(Elysia) #6

In an ideal world, I start automating as soon as a feature is ready. (I work on Agile teams, Waterfall makes this more difficult.) That isn’t to say I automate every feature as soon as it’s ready. I have a ranking system that helps me classify whether or not a test is a good candidate for immediate automation, and I automate the high-priority ones first.

Now, I know this isn’t always possible, as you touched on. In the case where I have a live product that I want to support for future bug fixes, hot fixes, and releases, I tend to focus on building a smoke test suite. There can be a lot of value out of a suite of 10 or so tests that at least verifies the pages are loading and users can authenticate. Then you can go from there.

I think the biggest challenge is actually not one of “when do I automate?” Your biggest challenge is getting them to see quality and testing as something that starts as soon as the product is thought of, all the way through development, and into production. If organizations aren’t looking at quality holistically, and what they can do process-wise to catch bugs as early as possible (as in bad requirements, too-large features, etc.), you’re going to have a very difficult time getting any meaningful automation to take hold. It has to be an organization-wide effort.


(Jason) #7

I’ve been in similar traps. Eventually, I concluded there is no bad time to start automation efforts. Even if those efforts are invisible, as you build up a set skills, techniques and tools, it is worth it. At the end of the process, in addition to being in a position to automate more quickly, your manual testing will also be more productive since you’ll have a stronger technical skill set to support your efforts.


(Gintare) #8

I would say, as soon as there are some requirements and coding has started. It doesn’t have to be end to end testing right away, but there are plenty of activities which doesn’t require stable product:

  • you can help and guide developers with unit or integration testing.
  • you can prepare infrastructure for future automation - test environments, test data.
  • you can gather a knowledge of technology used to build a product, select tools for your automation and gain knowledge of those, prepare strategy for your automation efforts.

(Heather) #9

100% agree with this! My first team I was very lucky, they were all on board with me being in on the entire design process from the very beginning.


(Dave) #10

I was literally just thinking about this very topic and then saw this post. Part of the problem is a little thing called ROI. How do you know what automation is valuable? In some ways you need to know something about the product under test to answer that question, so I think we certainly can start automation too early. There is also the problem of automation ‘locking’ you into certain approaches and processes. Starting on the wrong track can be hard to correct once again pointing to possible problems when starting too soon.

So when to start it? I don’t know. Too many thoughts rattling around in my head right now - I should probably organize them into a blog post :-p


(Heather) #11

That would be great! Share it here if you do blog about it please :slight_smile:


(Dave) #12

Will do :slight_smile:


(Dave) #13

Took a while, but I did finally get around to writing the post :slight_smile: