Iām a consultant so sometimes I arrive at projects which are already ongoing for years. What I often notice is that they have a huge application with loads of bugs in the backlog. (Non-Blocking bugs)
They only solve them when the customer reports them. Most of the bugs are only solved when the customer reports them and not when a QA finds them on the test environment (only during new feature development & blocking bugs)
How do you guys deal with these backlogs? (Iāve seen backlogs of 6000+ open bugs)
I sometimes am allowed to introduce like 2h-4h-1d/week time slot to focus on bug fixing, for those that āweā do kind of feel important but a lot of these bugs are not valid anymore either (we most of the time canāt afford a QA re-checking all these bug reports). So there has to be a better way to deal with bug backlogs and Iām looking for tips & tricks here.
How would you deal with joining a new project and noticing there are 6000+ open bugs?
I guess some are not blocking at all and probably will never be noticed but is it worth then having them in our backlog as open? Do you maybe have a specific status for it like āwonāt fix-until client noticesā?
Your backlog is almost like digging a big whole, at some point you have added so many things that time makes irrelevant that you cannot get out that hole or you waste so much time and effort pondering about how to get out the whole.
If you are lucky and fairly early in a products lifecycle its much easier, simply do not dig that hole at all.
I had a recent project which was one of the best projects I have been on in years.
We only had two bug priorities Critical and Major. If they were at this level they were important enough to do something about either that day or at least within the current sprint.
We had other bugs that were either fixed the same day by the developer often within 10 minutes of being highlighted or they would deemed as non essential clutter.
If its not important now, its unlikely to gain importance in a couple of weeks and if we added it to the backlog it would be creating waste and distraction from more important things.
My test session notes were in the system so if suddenly it did become important we could track down those for insight, but they were not on backlog distracting the team.
This model worked really well for us, keeping bug backlog to zero or there aboutās empowered the team to focus on bring value.
Should note, all developers were adding regression risk coverage with each new feature so in reality we did not get a lot of bugs due to regression.
If you are already in a fairly deep hole though, does it make sense to keep digging, perhaps landfilling it may be a more productive option even if someone is worried we may just lose some important information hiding in that hole at some point in the future.
@kristof : I faced same issue in my one of my project.
I bucketized the backlog as:
based on Priority
based on date ticket created (for example need to prioritize ticket in backlog that was raised X days / Month / Year back , where X depend how old is the project)
Rather then re-checking or verifying tickets again, small meeting with old team members (dev + qa) that can comment during meeting and either close or take an appropriate action item.
Remaining considered as fixed as those must be either reported again if they are of any concern
We log all bugs, no matter how minor. Each sprint I allocate a large percentage of the time to roadmap (including bugs introduced by the current round of roadmap work) and the remainder to āotherā. āOtherā includes third line support, self-training, technical debt (all that stuff that we all have in software thatās over a few years old and is using third party components that could be updated, tweaks weād like to do to areas of the code when we get time, tooling etc) plus bug fixes. Primary concern is to fix anything reported by a customer as soon as itās reported no matter how minor it is as that improves customer perception of the company as well as the product. Secondary concern is reducing the bug list.
Just a few years ago the bug list was in the thousands (a 20 year old product base and concentration on new code rather than fixing had taken its toll). Some of the bugs on the list were even for discontinued areas of the software so no one was ever going to encounter them (some for software that had never been released) so they were closed off immediately. We now aim to pick at least a couple of bugs off the backlog each sprint (at least one per dev) and through re-testing and fixing weāve now got the list down to under 30 for the first time in at least 10 years - 15 years ago every release went out with no known bugs although I canāt see that happening in the near future as the remaining bugs are either obscure and low risk to customers or in areas weāre not updating for a while.
My advice is actually to start with the quick wins - these donāt have to be massive amounts of work, sometimes tidying up a few cosmetic bugs that only take a few minutes to resolve will make a lot of difference to a userās perception (no one likes typos or non-standard labels etc) and will make you feel that you are making in-roads into the problem. If you can, allocate some time each week to re-testing the backlog starting with the oldest first or areas that you know have since had a bit of a make-over - I found a number of the problems no longer existed (probably had been double-logged or changes to that area had resolved the problem) and being able to close them off really helps. I also regularly publish the bug numbers to key stakeholders; seeing the progress helps them to understand why allocating time to non-roadmap work makes a big difference to the product base.
PS an approach a previous dev manager took was to simply close the entire list and see what re-appeared - virtually everything was re-logged by the Support team whereas if the list had been left open theyād just have tagged another report to the bug and it would have risen up the priority list simply because of the number of customers reporting it. Ignoring the list doesnāt resolve the problem.
Weāre busy dealing with this exact problem at the moment.
We prioritise bugs P1-5. P1s and P2s get sorted this sprint. P3-5 get put into a backlog that without someone really driving this, can become a backlog of 100+ issues very quickly.
We recently did a bug smash which so far has worked really well. We grouped up all the bugs in the backlog logged by other departments in the business and got them to help us close off anything that isnāt a big issue and we realistically wonāt fix. With the remaining issues they wanted fixed, we got them to give us a priority so we can fix them in order that makes the biggest impact to them (instead of us prioritising).
Next we did the same with the exploratory bugs we as QAs/Devs logged and we closed off tons of P4s and P5s.
Now we are down to an achievable number of issues which we are pulling 1-2 bugs into each team per sprint and chipping away at it in priority order.
Ditto on some of the above - weāve swapped in a bug smash sprint for a regular sprint and thatās helped a lot. Itās also a great way for engineers to learn about features they may not work on often and it can be a fun morale boosting exercise for the whole team. Who doesnāt love seeing bugs get closed? Note that there is substantial prep work that goes into it. The QA team grooms the backlog and tags tickets that are good candidates for a smash so that when the smash sprint starts, thereās a real dev-ready list to pluck from.
Another strategy is to have regular QA bug grooming bashes. We had a pile of old bugs sitting for a long time and once we took the time to review it we found that many of them were n/a - the product had changed a lot since the bugs were reported and either they were fixed or the feature changed so much the bugs just went away. I try to empower teams to look critically at low priority bugs - if there are a lot of bugs on a feature that is likely to be sunset, teams should feel comfortable withdrawing with a note that fixing the bug doesnāt align with the product roadmap and thatās ok.
Thought of one more thing - if you have leadership buy in, it can be good to make bug tracking and reporting a part of your culture. Eg. we have a segment during sprint demos where the QA team shares the bug trends for the last 2 weeks and we also have dashboards where individual teams can see how their own bug backlogs are trending. We have a bug SLA so having reports and visibility on bugs and bug goals are how we work to hit our SLA.
In my current team we use Kanban, as itās a relatively small API-based product. There was no tester in the team until I arrived, so being a nosy tester I reported a lot of defects in the past few months. Most of them were accepted as valid after being triaged by the whole team, a few disregarded ones were related to API which were planed to be decommissioned and replaced, so there was no economical reason to fix them.
As they keep pilling up, the test summary report Iām sending at the end of each sprint, started having more and more of those red charts! So, we came to an agreement, for each sprint the developers would take a few of those bugs and move them from to-do to in-prep columns, so we tackle few of them each week. After two month of doing that the reports started to look more favourable and the upper management was very happy and encouraged us to keep doing that.
These small wins are very helpful and important, especially if you have a lot of more time consuming stories (or blocked ones) in the development pipeline to get some gratification and a sense of progress. Simon Stewart referred to this as āDopamine Driven Developmentā, more details are available in this presentation, in case anyone is interested.
We have the strategy - and it works pretty well for us - to close bugs(or at least talk about and consider it) if they are open for a certain amount of time unblocked and without progress.
For me that makes perfect sense. Just consider a bug which is open for one year without progress. Seems like the developers donāt care about it or have more important stuff to do because otherwise they would have put a least some effort into it. Seems also like the project manager/ team leader does not care because otherwise she would shift priorities so the developers have time to fix it. Also the stakeholders of the product do not even realize the bug or see it as not important enough to force some activity on that bug.
So if no one really cares why should you keep it open?
Of course there are some scenarios where it makes sense to keep it open even though there was no progress in the last year. Thatās why it is important to have a short discussion about it before closing.
Also keep in mind closing is not forever. If for some reason you later realize it was a mistake to close it - just open it againš
I feel this issue is mainly about getting the bugs into a state so they are consistent enough to group them and manage. First problem usually being too many inconsistencies in the process and the bugs raised. Soon as you are able to accurately group them, you can analyse the bugs to get an idea of the problem.
Some includes,
different teams raising bugs differently so makes it difficult to manage them
people not understanding the bugs workflow. Status and fields set incorrectly
a lot of bugs coming from one project,
a lot of bugs coming from a functional area,
a lot of bugs pending re-testing etc
Once youāve got the data about the bugs, that will help drive how to tackle them. Your approach to reduce them
So few things you could do:
Clean up the data: Assign someone to groom the bugs, ensure they are tagged/labelled consistently so they can be grouped /filtered, and they all make sense and are clear. NB: Ambiguous or unclear bugs canāt be fixed, so either needs updating by who ever raised them or closed. Someone might have to go to those that raised them to re-review those issues
Make sure people understand what the different status mean exactly(clarify each status and share it) and the different fields. Best option is having drop down for as many key fields as possible. Make the process simple and easy to understand.
have someone dedicated to monitoring and following the issues up. Someone who follows up for updates, gets etaās assigned, ensuring they are assigned to the correct person at each time. ie someone that can actually do something about the issue
Iād come up with an agreement to close any bugs older than X months eg 2/3 months depending on if its a fast moving project or slow moving ie what are the chances of a 3 month old bug still being relevant?
Have a session to review the remaining items, main focus being to get them assigned to someone. So either its clear and is being assigned for a fix, its fixed and being assigned for retest, or needs more info and is being assigned for additional info, or assigned to a PM/scrum master to look after and address but each ticket should be assigned to someone.
Share with the team progress so people know something is being done with the tickets they raise, if using a tool, use comments to share updates.
Any defect review meetings should be focused, discussions should be left to the end for issues that need them.
Have a bug fixing sprint, if there are a lot of Very important bugs that need fixing asap
If focus is on getting numbers down iād work on High/important bugs + quick wins first as someone has already said.
Soon as they are down to a manageable amount , make sure thereās a process to stop them getting big again. ie if the team knows they wonāt fix it close them asap instead of a backlog. See Augusto post as Simon shared
Archive the lot. They are no longer valid bug reports, ignore them, itās historical reference only. A bug report nobody is going to do anything about is a massive cost for no benefit.
Written bug reports are now a last resort. All bugs should be communicated in person and fixed in situ, preferably before or when they are written.
Either deal with the bug or ignore it completely. If the problem is considered okay to shovel out to customers then donāt spend money thinking about it and writing it down.
Deformalise all testing to stop pointless and repetitive bugs being made, and concentrate on problems that matter. If the testers donāt know what matters then thatās where to start.
Examine the testing and programming processes to find out why there are so many bugs coming out of it.
The aim is to clear down and stop the pile from growing again. A bug mountain is a symptom of something broken in the project, chase that down and fix it or youāll be bailing it out forever. The system clearly is permissive of buggy software, so make some decisions about what you want that software to be and how much youāre willing to spend to make it like that.
Itās a really long post by Angelo, My takeaway was just one piece
āā¦Forgetting about it forever also means that we wonāt carry a stinky dead fish in a defect management tool. We wonāt have to waste time re-discussing the same dead fish forever in the future and our customers are happy we are not wasting time but working on new features.ā
Iām a big fan of having a bug bucket that I call file 13. Itās the folder at the back, with all the bugs that only the auditors or insurance company care about, bugs that are unlikely to cause customers major grief and loss, and bug, which we suspect might go away if we ever do that planned component re-write. Any unfixed bugs that are age>2 years belong in this bucket, reality is they will never get fixed. Any bug that survived 3 major releases definitely belongs in file 13. Quickly rejecting them with wont-fix and a triage meeting that captures why you wont fix it (like for example planned obsolescence of a feature) removes it from the radar for good. Time saved, be honest with yourself about your ability to fix some things. Customers are waiting for that shiny new feature.
Yep, agree bug-grooming bashes are a great fun activity to do post-release of a major release. And a good way to improve communication between teams. We should all read the blog link Simon posted, because my experience has been that having bug-mountains ends up deflating your test team when they still see last years ābug-bashā bugs are still there in the backlog, perhaps Iāve been doing this gig too long too.
