So Iām part of a small dev/test team who have previously chosen a non-code automation solution, where so far the licensing costs have been relatively low.
Our solution has had a good start; despite relatively little time to invest as a test team, we have reasonable coverage of our mobile and web apps (and APIs), and this continues to prove to be effective, having picked up a number of issues typically within 24 hours of code being checked in, and the potential for wider coverage is good and mainly only limited by time.
However, the licensing costs are about to rise significantly (from around EUR 1,500 annually to EUR 5,000) and now - despite the stage weāre at and what I would consider to be the success weāve had, I now find myself having to justify the increased spend (which Iām not saying is unreasonable).
I canāt be 100% sure but I beleive one aspect of all this is that itās perceived to be expensive because developers in our team all use a free IDE for their development work, and so the question being (notionally) asked is something like āhow can we justify spending EUR X for test if we spend nothing on [what is probably perceived to be] the equivilent tooling for dev?ā
I personally feel like this is comparing Apples to Oranges, and it feels a bit like asking why is an electricians toolkit more espensive than a plumbers, even though a plumber might sometimes be able to charge more (okay this may not be a perfect analogy but hopefully you get the idea!).
My point (that I will be making) is that if you want working and effective test automation, you have to pay for it somehow, whether that be with time and (perhaps specialised) resource if you went (for example) for the Selenium/Appium code-based solution route etc., or service costs if you go a route that sees you using a BrowserStack or Kobiton type of solution, or license fees if you go the TestComplete/Ranorex/Katalon route (the latter of which we have taken, which I think has worked out particularly well as itās enabled us to get up and running quickly and effectively with great scope to scale up very easily as and when required).
I just wonderedā¦if anyone else has been in a similar situation and how youāve managed it please? Any suggestions (or better analogies than I can come up with!) would be greatly appreciated!
A few years back promoting a test automation solution was part of what I did. Not currently though, but perhaps what I can remember can help.
The key results of various computation examples and projects were two things: With test automation, we can perform/run test automation more often - not only once or twice, as a person would do within a release. And we can have suites that have more coverage, in the sense that we can easily add more variations.
In one project we had 100 tests running every month, but running it was free in āhuman eyes and handsā. Performing the same āby handā would cost 150 hours. ā¦ at letās say ā¬75/h. 12 times a year. That would be ā¬135.000 pr. year. These numbers are not the correct numbers but it can tell if thereās a break-even in your tool budget. You can do it the other way too, how much āexecution muscleā do you get for ā¬416/month at your current hourly rate?
Some automation solutions require hosting and other running costs, that need to be considered. But what is the alternative without a tool? Will you be able to keep up your delivery cadence without automation? will you need your own device test lab instead of simulators - at the cost of maintaining them?
The first one is often used be those selling automation, unfortunately it tends to contain a lot of fallacies of argument and in reality is often based on your starting point of doing really wasteful time consuming activities that automation can then empower you to do these same activities quicker and faster. Dave would spend 100 days doing testing across these 10 devices and 5 OSās sort of thing and once automated we can do them in a day so every time we save 99 days.
This argument tends to carry a lot of BS unfortunately, nobody in reality is going to do that level of testing without automation and they probably should not be doing scripted testing without automation at all. Stopping doing manual scripted testing and replacing it with something else is often a better saving than automating a bad practice. The argument does however sell well and can get passed a lot of managers.
Perhaps a very small element of the above can be used very validly though but often not enough for the licence fees.
You are often better removing the cost savings argument almost completely and focus on the value the automation brings.
What important things has your automation caught that would be missed otherwise, has it empowered development to go faster, does it provide faster feedback loops to the team and what is the team value of that. There are many other things automation can bring to the table.
If you can take cost savings out of the picture and still make a strong argument you are usually in a good position, if you need to resort to cost savings then it can be hit and miss depending on the audience, some will jump on it whilst others may see big holes in the argument.
I had to help motivate for a TCMS last year, and despite the senior team saying we had run out of budget last year they still found enough license money a week before Christmas. 2 months on, and yes although I could have bought 2 entire teams a nice lunch every month for the subscription fees. It feels worth it. It does however mean that we have to use the tool hard, and make it work. BUT. Going from 1.5K to 5K, would also make me tempted to ditch the tool, so I, would, threaten to ditch the tool openly. In a brinkmanship effort to get people to wake up and get dirty on both sides.
For out TCMS purchase I had to explain to the QA group/chapter that they have to make the best possible use of the tool in ways that suit them and save them time and effort. All before we bought it, and that has worked. We have still, like yourself @kevinm , to push the tool and find ways to get more value from the tool that we currently do, especially if the bill rises. And it will for us at some point. I, would then be arguing for scaling up and scaling out how we use the tool, thatās what I would do. If your teams cannot find more use for the tool, then probably worth quitting, scaling down and building your own test solution. Also āitās not your babyā, itās not on you alone.
This insight alone gets you a little . If the automated regression tests are not finding bugs early, then it might be cheaper to pay someone in China or India to run manual tests for you. Many managers think that throwing money at a thing make itās quality higher, and tool vendors rely on that.
This sir!!! This is an argument I have used many a time. Its great to hear someone else say it
Depending on the application and your release cycles etc - it might be more cost effective as you say to have offshore manual testing. Or even have some juniors, or part timers etc more local rather than paying to develop automation.
This is an incredulous increase and one that would make any manager think thrice before accepting the terms, so I understand the concerns.
There are excellent responses by others above me, and I agree with most. Iāll just add my own 2 cents: create several plans of action. Eg:
Plan A: we go as-is, pay 5k annually
Plan B: we ditch the tool and do everything manually. Calculate cost of person-hours etc (see jesperās answer)
Plan C: find different, cheaper solution. Calculate cost of research and time to switch, etc
Plan D: something else that I didnāt think about.
With each plan, take into account advice from others, like calculating opportunity cost, risk analysis for what happens with quality of the product youāre testing, think about what kind of effect each plan would have on a development/project team as a whole.
At least two things you can unearth with this:
you might find out that you can actually decrease cost and increase quality with another solution - yes that can happen! (donāt get me wrong here, but I have a feeling you are holding a bit for this current solution cause itās in your comfort zone)
others might realise that your current solution is actually the best, given all the parameters (and it will give you a great victory parade! )
Good luck! And definitely report back with the outcomes !!
No, no, no, absolutely not what I was suggesting. Dumping these scripts completely is what I was suggesting, not getting someone else to do these.
Having two good testers with solid investigative, discovery and experimentation skills that switch the focus from basic scripted verification to more of an exploratory approach can offer a massive amount of value change. Often two good testers can do things quicker, find more important things than an offshore team of ten or twenty purely focused on a scripted approach.
Thatās a change Iāve implemented before with a considerable value add difference.
Another tactic is to go and find very specific tool features that your team sorely would like to have to make their jobs much faster and let you scale up more. Take them to any meetings, as backup ammunition when negotiating starts to go against your position.
This is why I like to engage the community forums on things when I think a tool should help me do a task and it does not. Sometimes the community can help you with a workaround that help you to work smarter or more confidently instead.
Start with the problem statement. your goal is to express it in the financial terms understood by upper management.
āCurrently the problem X is being handled by performing Y tasks by Z resources costing A moneysā
āby automating a percentage of tests, we reduce the test cycle time for regression testing. Once we have achieve FOO% of automated regression tests, we will be saving the cost of the license by not paying engineering salary in performing manual regression tests, freeing those resources for other activities like extending the automation or testing new features. if we implement the tool/framework and spend appropriate resources to utilize it we estimate the cost to benefit ratio turns in (estimate of time)ā
that kinda thing. Its not a magic recipe and there are other factors (cost of building the automation, maintaining, etc) But the point is to express it in terms that are understandable by management and sets achievable milestones to ROI
My angle may be a bit different. Or perhaps it is more a different way of phrasing the same thing. But I would first take a step back and make quite sure I know what is the business value of the automation. Not as a team perceives it but as management would see it. Then you can relate everything to that.
Keep in mind that this value is not, for example, speeding up the regression test. Nobody cares about that. That is merely a means to an end. Keep asking why until you end up with real business value. I know only five things that make business sense:
lower cost (rare as focus these days)
higher quality
higher speed (in 3 flavors):
efficiency (how fast you get one item to prod)
productivity (how much value you create per sprint/ā¦)
fast feedback to the team
Only when you know what real value of automation to the org is/should be can you target it effectively. But also you will know how to explain (or sell) it to decision makers. So then you think about what is needed to create that value. And in the context of this question you consider how the higher license cost compares to the value that is delivered. And then you have a ābusiness case.ā
Example: If time to market is all the business cares about then the real goal of automation is productivity (or efficiency, such as when single features are immediately pushed to prod). Then the business case can be phrased in those terms, and you would probably not talk about RoI because cost is not the main factor anyway.
Hi all, thanks for your responses - Iāve read and digested all of them but just donāt have the time to reply to each one individually, but I really appreciate your thoughts on this and every single reply has given me something of value to ponder and perhaps use.
I particularly like your ideas @martingijsen about trying to see the value from the businesses perspective - as you say, thereās certain detail theyāre just not interested in but if I can focus upon what I think is important to them (even if itās less important to me) then I think thatās a great way to view things and that will be my first way to approach this (hence Iāve marked this as the solution, even though all the responses have helped me, and there is never a single solution to complex issues like this!).
Just a couple of other pointsā¦
In terms of offshoring, weāre a small test team anyway testing a complex legacy system, so I donāt think this would prove beneficial for us, but I can see the value perhaps for larger teams for stuff like simple mobile app testing, and @andrewkelly2555 yes I think youāve summarised the situation we have pretty much spot on.
In terms of the licensing hikeā¦I certainly can understand the business concerns but the hike actually brings the licensing up to what I would consider the āgoing rateā for what it is, so although itās probably a āshockā, I feel it still represents good value. Part of my case will have to be that weāve effectively had the licensing ādiscountedā for a period of time and that the cost now is the ārealā cost (which I still beleive is the cheapest and most effective option).
Again, thank you so much to everyone (including those I havenāt managed to tag) for all of your input - I will re-read this entire thread again when I come to work on this next week
. If the automated regression tests are not finding bugs early, then it might be cheaper to pay someone in China or India to run manual tests for you. Many managers think that throwing money at a thing make itās quality higher, and tool vendors rely on that.
)