Issues in current testing approach

Discussions
,
1

Hi

Looking to get some thoughts / suggestions

We have a shared platform across regions. Releases are generally stable, but there are periodic challenges related to: Defects, Rework, Post-release fixes. Multiple stakeholders are involved in UAT and validation before releases. Teams are making efforts to improve quality, but results are not consistently improving over time. There is no clear consensus on where the main issues lie.

How to assess the gaps in the current quality and validation approach?

What metrics would be ideal to track?

Thanks

Chris

2

I don’t think I have a good grasp of the problem, which may indicate that nobody has a good grasp of the problem.

You say that results are not consistently improving over time, and that might be more about how you’re evalutaing what quality is and how it improves. For example if you use simple metrics to measure improvement that don’t actually represent quality, or represent it in some overly simplistic way that gets swallowed up in the error bars of reality, then you are trying to improve something you can’t measure (incidentally this is probably true for all metrics in most contexts). It could be that improvements are not supposed to be consistent. If a product does a lot of work in one complex area that suddenly introduces change risk in a high-risk part of the product. If staff changes, or communication changes, or customer need changes, or the marketplace changes these can reflect on what “good quality” then means, and affect the results of any attempt to measure improvements. A good tester probably can test the means of measurement as well as the thing being measured, and determine where that process fails. So it may have nothing to do with the actual quality of the product, and improvements may actually be showing success. It depends on who’s perspective on success we use, and what “success” actually means, for one thing.

With that murky swamp mapped out with a large question mark the problem then, for me, is to find out who thinks there is a problem and what they think it is. Then I need to find out if it actually is a problem of any kind. The phrase “releases are generally stable” sounds like it’s going okay and “there are periodic challenges” tells me that reality sometimes happens to your project. It may be that you can tie each challenge back to a possible cause, like adding a new person to a team, or needing to learn a new technology, or a rushed design document, or any of the many little things that can affect projects in a large way, especially large ones across regions.

Each periodic challenge could be affected by a particular systemic issue across the company. Or companies. Or whatever. Such as poor management, poor communication, lack of critical skills, lack of flow, horrible deadlines, insufficient planning, premature formalisation, and so on. It could be that each one comes from its own source, and is the result of one particular team project not going well, or some process improvement that hasn’t had time to settle down (or is not working), or a myriad of other things. Without investigating each problem you can’t push back on the “why” of it and try to find a source. If there is one. Or two. Or twelve.

So it’s very hard to answer your question, but hopefully I’ve given some inspiration for how to go about breaking it down into smaller questions. I’m sure others will have other insights. I’d definitely say not to rely on metrics to track improvements, especially if those involved don’t have some understanding of metrology, statistical analysis and scientific humility, but they can be used to trigger questions that might find problems that are stopping you from ascertaining quality properly, or indeed producing higher quality… whatever you’re producing. What you look at may not tell the story someone assumes it will, but can be worthwhile if treated properly, and, unfortunately, depends heavily on what you’re trying to achieve. Numbers going up and down mostly only reliably indicate numbers going up and down.

So in short:

  1. Is there really a problem or is someone mistaken, overstating something, or has an agenda?
  2. Is there really a problem or is the measurement borked?
  3. Is there really a problem or is it within the bounds of the normal chaos of software projects?
  4. If there could be a problem can we describe it realistically and accurately, or are we just sad about the whole thing?
  5. If there is a problem is it actually lots of smaller problems?
  6. If there is a problem can we trace it to a cause, or is it too complex?
  7. If there is a problem can we turn it into a goal we can examine qualitatively?

Hope that is vaguely helpful in guiding the ideas. Or at least figuring out if a problem exists and what it might be to do with. If there’s any specific problems feel free to update, and best of luck

3

There are different quality attributes for examples :

  • Performance
  • Reliability
  • Robustness
  • Security
  • Usability
  • Accessibility
  • Privacy
  • Availability
  • Observability
  • Recoverability
  • Scalability
  • Maintainability
  • Modifiability
  • Reusability
  • Testability
  • Simplicity
  • Understandability
  • Operability

that organisations offer to their customers depending upon features and domains of ur product some might be more important than other , so identify that first is important and focusing on improving that first. Also most people think quality as product quality , but process quality is also very important . for example :

  • Frequency and size of deployments
    • Frequent, small changes reduce risk
  • Time to deliver changes
    • Cycle time and lead time
  • Percentage of time spent on rework
  • Length of feedback loops
    • For example, automated regression tests
  • Success of production deployments
  • Percentage of failures
  • Time to recover from failure

How process quality affects product quality :

Examples:

  • Short feedback loops
    → defects are found earlier
    → improves reliability and testability
  • Frequent small deployments
    → lower risk per release
    → improves stability, availability, and recoverability
  • Less rework
    → team spends more time building correctly
    → improves maintainability and simplicity
  • Fast recovery from failure
    → service is restored quickly
    → improves availability and operability
  • Low deployment failure rate
    → fewer production issues
    → improves robustness and user trust
4

Keeping it simple I’d start with DORA metrics. But metrics highlight issues with outcomes, they don’t tell you where the problem is. That comes from investigating the data behind those metrics to establish the patterns behind them.

5

I would suggest introducing a RACI matrix to bring better clarity and transparency around roles, ownership, and contributions within the testing team.

Clearly defined responsibilities help avoid overlaps, missed ownership, and dependency confusion during execution. It also improves accountability and provides better visibility into who is responsible, accountable, consulted, and informed for each activity.

With better clarity in task ownership and communication flow, the team can operate more efficiently, which positively impacts overall delivery quality.

6

What does “results” mean in this context? It might be useful to define the issue more concretely firstly. Are stakeholders unhappy? Do team members lack trust in automated checks? Are the same kind of issues consistently being missed?

I think once you’ve defined the problem better, you’ll be in a better position to decide what “efforts to improve quality” should look like. It could be that the things the team are doing don’t have the ability to influence the actual problem in the first place, and that’s why things don’t seem to be improving.

I also wonder whether you have a test strategy in place? This could give you clues as to the types of testing / techniques you’re missing, quality aspects which aren’t being assessed / addressed, etc. You might also want to try a Quality Radar workshop: Increasing visibility and closing gaps with the Quality Radar (workshop)? My team ran one of these a couple of months ago, and it really helped us to visualise what our quality efforts are going in to.

Keep in mind as well that if “releases are generally stable, but there are periodic challenges”, that might not necessarily be an issue. You probably already know that you can never eliminate all issues / challenges. Things will always happen sometimes, and there could be good reasons for them. Maybe you could analyse past challenges and see whether you can spot any recurring patterns, then see what you could do to address those.

Good luck!