In a #TestChat we had about automation @rosie asked:
When is a good time to start test automation? (I often get people wanting/asking it as a priority and before any other testing has been done before)
This is a question Iāve struggled with in the past a lot.
In my first company I fell into the trap of āWe canāt start automation because the product isnāt stableā. I then proceeded to toe a really fine line between the product being stable and there being too much of the product there. I was a lone tester so automating too late was going to be a time sink. As a lone tester though, automating early could have really helped me to use automation in the right way and potentially free up my time. (This was my first venture into automation).
In the next company, I was brought on pretty late in the development process (nobody to blame, it just happened that way). There was an automation suite that had been written months before the (now released) product had gone live. It, naturally, no longer worked. I decided to scrap it and try to start again because it was so outdated. It was then that I realised, automating at this late stage was pretty pointless. The product was massive! We also had unit and some API level checks.
In your experience: When is a good time to start automation? Why?
Just found an interesting article that may help to answer your question.
The challenges can be daunting, but the conventional benefits of automation are obvious: tests can be run faster, consistently, and they can be repeated as many times as needed with minimal overhead. But there are some other, subtler benefits worth bearing in mind.
Improved velocity ā As stress levels rise towards the end of development, automation can provide peace of mind about the code quality. It acts as a kind of safety net. And when developers are confident that code changes wonāt introduce lots of errors, theyāre free to really focus on each sprint goal.
Greater coverage ā Early automation can save a great deal of time down the line and result in much more comprehensive test coverage. By shortening the feedback loop, test automation boosts your chances of finding serious defects earlier and fixing them before they become embedded. It also frees up experienced testers to explore new areas of functionality and focus where they can provide the most value.
Higher reliability ā Todayās business environment demands high quality software. Early test automation is the most effective way to build confidence in the software being developed, and the more comprehensive the testing is, the greater the quality. Automation also drastically reduces the risk of human error, ensures that key business workflows are fully covered, and minimizes the risk of critical bugs reaching production.
It may not be easy to implement, but test automation can deliver tangible benefits, and represent an excellent return on investment for any organization.
Tool assistance is useful from before the product has finished an initial design until well after the last build has been shipped, up until around the last user dies.
If weāre talking strictly about large check suites then in my experience because they swell from the drippings of mindless responsibility they are often pointless even if the code is clean and the architecture is well designed, because they donāt check for anything of value, or check in a way that offers the value they claim to provide. āTestLoginPageā. Yeah, right, prove it.
But if youāre going to design worthwhile checks, not including checks just ābecauseā but because youāre a professional making smart decisions about checks including their purpose and cost and value and expiry date, then itās useful to start considering it before the first line of code is written. You may end up designing an internal API or other layer of interpretation to hook into to make the check suite worthy of its cost. You may change the technology you are using for your main product to suit your check suite product. What skills you have in your teams may influence your decisions. Itās more difficult to plug these things in downstream.
Starting a check suite project also needs a bunch of administrative and social payments. You may need a new environment, or a change to your systems that IT or Ops need to make. You may need a new monitoring system. You might have to get tool budget. You may need training for teams to get them used to the new tools. Theyāll have to fold it into the build process or a CI system. Things will go wrong while people get used to the new systems and technologies and environments and how to use them effectively.
This administrative swamp scales with the size of its project. Thatās why smaller tools doing specific things to make the product more observable or controllable can be as or more effective an investment - log viewers, alerting systems, database difference tools, scripted installers, VMs with snapshots, and more.
In the context of writing automation to support the development of APIs, I advocate starting as soon as there is code available if not sooner. APIs are a special breed of software products that just scream for the right level of automation.
In a recent API project, we were able to use SpecFlow to automate the GWT. The completed tests could be used by developers to guide them as well as help them know when they are done. Over time, the automation was used as the regression suite. Supplemented by a healthy set of unit tests, automation results (both unit tests and the SpecFlow tests) provided the team with high confidence that our products operated as expected.
In an ideal world, I start automating as soon as a feature is ready. (I work on Agile teams, Waterfall makes this more difficult.) That isnāt to say I automate every feature as soon as itās ready. I have a ranking system that helps me classify whether or not a test is a good candidate for immediate automation, and I automate the high-priority ones first.
Now, I know this isnāt always possible, as you touched on. In the case where I have a live product that I want to support for future bug fixes, hot fixes, and releases, I tend to focus on building a smoke test suite. There can be a lot of value out of a suite of 10 or so tests that at least verifies the pages are loading and users can authenticate. Then you can go from there.
I think the biggest challenge is actually not one of āwhen do I automate?ā Your biggest challenge is getting them to see quality and testing as something that starts as soon as the product is thought of, all the way through development, and into production. If organizations arenāt looking at quality holistically, and what they can do process-wise to catch bugs as early as possible (as in bad requirements, too-large features, etc.), youāre going to have a very difficult time getting any meaningful automation to take hold. It has to be an organization-wide effort.
Iāve been in similar traps. Eventually, I concluded there is no bad time to start automation efforts. Even if those efforts are invisible, as you build up a set skills, techniques and tools, it is worth it. At the end of the process, in addition to being in a position to automate more quickly, your manual testing will also be more productive since youāll have a stronger technical skill set to support your efforts.
I would say, as soon as there are some requirements and coding has started. It doesnāt have to be end to end testing right away, but there are plenty of activities which doesnāt require stable product:
you can help and guide developers with unit or integration testing.
you can prepare infrastructure for future automation - test environments, test data.
you can gather a knowledge of technology used to build a product, select tools for your automation and gain knowledge of those, prepare strategy for your automation efforts.
I was literally just thinking about this very topic and then saw this post. Part of the problem is a little thing called ROI. How do you know what automation is valuable? In some ways you need to know something about the product under test to answer that question, so I think we certainly can start automation too early. There is also the problem of automation ālockingā you into certain approaches and processes. Starting on the wrong track can be hard to correct once again pointing to possible problems when starting too soon.
So when to start it? I donāt know. Too many thoughts rattling around in my head right now - I should probably organize them into a blog post :-p
Well, the sooner the better, Iād say. That of course depends on what do we mean with automation in the first place.
Iād start by automating repetitive tasks Iāll have to do myself, at first. Unit tests are part of automation, but most likely more to be done by the developers.
One of the crucial things is to get visibility for the quality and to automate that kind of things is worth doing. I mean gathering data from different parts of applications and showing the results. How many errors we get to see and on what occasion, how much memory we use, whereās the cpu usage going etc. The nice thing about that is that it actually reveals a lot on the software under development and also forces teams to consider output standardisation etc.
From there you then can start targeting to different parts of the software.
Just my two cents. Iāve been working as tester in my Scrum team for nearly three years. I do not work in IT but in Product management and together with the product owner, I represent āthe businessā. As I work in enterprise tech, I did not have access to the code base and thus no access to all automated integration and functional tests that the developers are writing. Now finally I do have access and I will start a Java programming course next month. In short: Iām up for a steep learning curve which is also what I want. as I want to grow in my role and possibly further. The only ādrawbackā is that the tool they have been using for the functional tests (Selenium driven tests that start by using the UI, going to the database and back) have all been written in JBehave. Yet all I see on conferences is Cucumber, Specflow, etc. etc. Will start reading all your posts as this may help me get a good head start.
Automation should start as early as possible, considering that the tester can get involved in unit testing and integration with developers, and as the project matures, functional automations start, and so on.
