How to create Business Case for expenditure toward Visual Regression Testing Tool?

Situation:
Some teams have UI heavy regression tests that are not automated using visual tools. However, non-visual automation has started to help with reducing the amount of manual testing.

Completed trial with two visual regression testing vendors.
3 teams participated. No other teams in the company expressed an interest.
Company just setup a new tool evaluation to purchase process and I am at the point of getting approval from the architects. I feel like a guinea a pig in this process…

Meeting with architectures didn’t go as planned and the following are required:

  • cost benefit analysis (CBA) for all teams in the company as if they were to undertake visual regression testing.
  • “proper” use case

How have other people approached this?
What do you consider as a “proper” use case?
Any good examples / templates of CBA that you would recommend?
What other information am I missing?

Note: I am QA automation whose not so familiar cooperate finance, selling and all that kind of work.

Your suggestions much appreciated.

Thanks.

4 Likes

Hi @stwoo2019 ,

Thanks for sharing. Good luck with your evaluation and putting forward a cost-benefit analysis.

I remember during a 30 Days of Tools a mega thread appeared about how folks evaluate tools. It’s this one:

A proper treasure trove of handy ideas and experiences. If not on your radar already, I hope it helps.

Perhaps a Pugh Matrix might help with whatever criteria you add to it. :thinking:

1 Like

Plus, I’ve just found this thread (which wasn’t on my radar). Nice! 30 Days of Automation in Testing Day 7: How do you choose which tools to use in testing?

Thanks for information.

Yes, all the technical evaluation has been done and winning tool identified. I need switch to a different skillset to put data in the face of management and convince them to pay with money.

1 Like

Hey @stwoo2019!

I did this on a previous role and the key thing is to present them with a lot of data on how the current test setup doesn’t scale without a visual regression tool. In our case, teams were already using open source tools so visual testing was already part of the strategy.

What I did to convince the leadership team was I presented them with data on the following:

  • how many times a deployment was blocked due to a flaky visual test caused by pixel-by-pixel comparison. Here, I showed them data on time wasted rerunning pipelines again or updating the visual test to unblock the pipeline.
  • time spent by teams maintaining these tests and explaining to them that a certain tool can get rid of a lot of test maintenance. Here, I took an example of one person’s hourly rate and multiplying that by the time it took them to write/maintain a visual test.
  • number of visual bugs missed even if visual tests were added. This was because teams were setting a higher mismatch threshold and so certain bugs can slip.

Hope it helps and good luck!

3 Likes

Hi @stwoo2019 welcome to the MoT

It seems to me there’s some gatekeeping to the architects’ requests - but I don’t know your org. I don’t understand the “proper use case” either - And I have been in many corporate procurement activities. Can you get any help to understand this “new tool evaluation to purchase process”? Is it described and can you ask for templates for the steps along the way? Is there perhaps an example of a “proper” use case for another tool

It sounds to me like you are on the right path though. You have interest from 3 teams and have compared 2 tools. I reckon you have listed a range of comparison criteria or selection criteria. Recently I had these: “Handling of load testing, being a framework not an app, integrated into ADO CI/CD, preferably Open Source” YMWV.

On the math side of things - Cost Benefit, Business case, or Return on investment (ROI). Perhaps you can do at least two calculations. What does a typical regression test effort cost now - with no new tools? Secondly, what could the activities cost in the future using a tool? I found that the primary cost saver was fewer hands & eyes on the execution of tasks - the rest was more or less the same. Compare the two over three years and compare any savings to the tool costs. The question they would be looking for is how this balances - is there actually a cost save?

hope this helps :slight_smile: /Jesper

2 Likes

I’d like to thank the members who responded with great advice.
I’d like to inform you that I managed to secure funding and have bought the license. Now need to get the teams to integrate, identify the snapshots/scenarios that give the greatest value and identify metrics for gauging productivity etc. Then over the course of the year to encourage other teams to adopt the tool where fit.

The process to convince the Solutions Architect that the product is best fit and value was to compare cost, time and effort for manual regression testing against visual regression testing for one regression run/month/year. Identify the savings for the 3 projects.

Failed => Solution Architect wanted to know what it would cost for all IT projects. This was a change of scope.

Repeat with all applicable IT projects as to identify the cost of scaling up. I had to provide training to all the QAs on identifying snapshots/scenarios of value and calculating the number of snapshots. They were all tasks to provide me with numbers. Churn the numbers.

Passed => Solutions Architecture have buy in after the presentation
Passed => IT Steering Group gave approval after presentation with full understanding of cost at initial small scale, bigger picture and enterprise level.

It’s hard to believe so much effort was used to purchase a tool that wasn’t a lot of money compared to Azure related resources. The change in economical climate means that we have be very wary of every penny spent.

Again, thank you for your input. I will be using some of the ideas and massage them a little to capture productivity changes across time. If you have more ideas about monitoring productivity then do ping. Monitoring and metrics is also something that I am beginning to dabble in.

5 Likes

Glad the community helped here. Had to laugh though “Solution architect wanted to know what it would cost for all IT projects.” Well handled on that, you are an asset to your team now. So good to hear how a QA has made a difference.

Mainly I hope you had fun presenting and making the case. I also hope you find he time to make sure that as you add more tests and get more teams using your new tool, that the same kind of continuous change can be built in as you adapt and change how you use the new tool so that it works the best it can.

3 Likes

Congrats. @stwoo2019. That’s great news. :tada:

And thanks for sharing an update here. It’s lovely when someone comes back to their topic thread to provide an update for those who participated. It’s much appreciated. :smile:

Good luck with the next steps.

3 Likes