Guaging Quality impact on Business

Our CEO just asked how do incidents (outages) affect our business growth? Guess same could apply to missed bugs and quality.

Sounds like a reasonably and unassuming question. I have no idea how to answer something like that. Presumably negatively. More negatively than having some must have feature. Is there a magic number?

Anyone know how to measure or estimate something like that?

Is it even answerable?or is it the wrong question. Obviously it’s context dependant. I work in communications so customers hate any interuption in service but no service is perfect. So in commercial terms goal post is probably based on competition and what market will pay for additional quality. Obviously reputation is also important to us but does it have a dollar amount?

4 Likes

Does anybody in your team have access to the traffic of users during different time windows? This is just a metric that could be collected easily from the servers.

With this info, you could analyze how this traffic of users is impacted after incidences are registered or reported by customers. It’s not accurate, just an estimation, but it could be a good starting point.

As with life in general: If you are in fire fighting mode - you don’t have the bandwidth to invest or grow.
Key resources are pulled from growth mode to resolve mode, so it’s mostly about the loss of opportunity. 1 Major incident a month = x hours not spent delivering features - probably by your key staff.

the Devops hand book phrase it as the core chronic conflict of pursuing both: responding to the rapidly changing competitive landscape and providing stable, reliable and secure services. the Accelerate play book (Accelerate | by Nicole Forsgren, Jez Humble & Gene Kim) also has extensive research on what is needed for a high performing delivery team.

One of the key parameters is Time to Resolve, aka Mean Time to repair. If you can easily push a small fix, then you’re not as vulnerable and can react faster to incidents and changing market. Think facebook or twitter - they deliver “good enough” quality and react.

In a competitive market situation Reputation has a value, or rather loss of reputation has loss of sales. If your site is down consumers can so easily shop elsewhere. As @alopezari points out, start to monitor prod. when do your customers usually work your site and purchase orders. Break it down in segments, and you can get a loss of revenue pr. segment if an outage occurs. Would that work for you?

/Jesper

Yeah, I guess the issue we have is that because it’s not a public platform. We have large enterprise customers with licenses for hundreds or thousands of users. So we can see the interruption to users, but they can’t really leave. So any churn or disaffection may not be come clear until contract renewal. You can’t just say the users are up or down. It doesn’t reflect much.

1 Like

Yeah as i just replied to @alopezari our users can’t easily leave. I guess that’s a blessing and a curse. But yeah that’s all really good advice. I suspect our CEO is asking the wrong question. Impact of quality on growth may be implausible to measure. But we can expect growth to improve with improved quality based on evidence from accelerate and others.

1 Like

Hey Matt, good to see you here.

I’ve been working on something similar where I am working. Unfortunately, there aren’t any hard and fast rules for this. I think the first thing I did was consult with stakeholders to discover what mattered to them, and how best to identify the appropriate data and communicate it to them.

Some engineers didn’t think it worth capturing bugs or reported issues from support, as these could be easily gamed to suit a particular narrative. However, live incidents that effect customers are what most of my key stakeholders want to avoid, so in it goes. Likewise, whilst code/test coverage has been low for a while, it’s been improving. Some didn’t see it as helpful for code coverage to be included, as it only told part of the story regarding testing and quality. Other work was going on, but it wasn’t easily measurable.

2 Likes