For our third talk of TestBash Home 2021, @melissafisher takes to the stage to explore how we can better understand our production environments and learn from the bugs we find there.
Weāll use this Club thread to share resources mentioned during the talk and answer any questions we donāt get to during the live session.
Matthew Churcher > Glastonbury sells out every year and makes huge profits despite all their IT issues. How do well they or we would succeed more with higher quality investment?
Roman Segador > What is your definition of a bug? When something stops to be work in progress and starts being a bug?
Kashyap Mehta > As a Manual tester, how do you do root cause analyse of a bug if you donāt have enough coding knowledge?
Atiqah Azlan > How often or when you usually review regression test suite
Risko Ruus > When you started testing on feature branches with the devs, how much did that testing differ from the testing you did once the branch got merged into master?
Steven Devonport > Great talk Melissa, how do you balance the activity of preventing bugs versus delivering into production as soon as possible?
Jonathan Hope > Would you create a branch for each defect reported in production?
Conrad Braam > Bugs, is āPriorityā versus āSeverity/Impactā, for managing bugs better? Or if you have to spend time prioritizing, have you maybe just got too many open bugs?
Paul Naranja > Can you give more details on the workshop āWhere your bugs come fromā? How did you show that many root causes of bugs are before the actual ātesting stageā?
Ellie Lock > At what stage do you start considering whether there is a problem in the development (coding) process as opposed to just QA process?
Matthew Parker > Can a bug become waste?
Olly Fairhall > Did you have to change the culture single-handedly when getting bugs in production fixed or was the team happy to change?
Ben Dowen > What is the best bug that was found, but had insufficient information so was left until it was way too late?
Becca Batchelor > How do you convince your business to pay attention to bugs when they dont cause $$ to come in such as new work for new customers?
Kashyap Mehta > I noticed that when bugs been reported in production, they are rarely included to test suite once reported. How do we encourage QA team to add such edge cases?
Louise Barnes > What testing improvements you implemented, didnāt work?
Penny Howard > Follow up: What bug metrics are you using?
Simon Rigler > Do you have a process for managing āurgentā Prod bugs that have come up when sprints have already started etc?
Julia Shonka > Considering āhow many are too many bugsā, donāt you have to also take into account what kind of application it is? Will the bug endanger human life?
Flashback Express Recorder FlashBack Express - the best free screen recorder
Club thread How do you prioritise bugs?
Iām going to get ON and tackle these questions for the next 30 minutes! We all have our own experiences, perspectives and knowledge to share, so come join on this thread. I invite you to answer the questions or bring in your perspective to this problem.
A great question! Yes, unlikely to affect Glastonbury. That festival has a reputation that goes beyond everything. EPIC festival. Highly recommend going if you ever get a chance. Who it might affect is the live streaming company that provided the service. It was a company called drift live. That live streaming company is affected in terms of reputation. It was ALL over the news. Iām sure they will be able to bounce back, however, if I was looking for a live streaming service, Iād probably look for other options and likely go elsewhere. Glastonbury live stream technical problems overshadow star-packed show | Ents & Arts News | Sky News
I see a bug basically a result of a problem that wasnāt thought about. So, for example, you have a form, you submit it, and it goes through. However, there was a problem that wasnāt discovered here. There were mandatory fields there that the user had to fill out. The user didnāt fill them out because they werenāt forced to fill them out. What would be your definition of a bug @romansegador ? For the second question, probably when you start implementing. Often we donāt think enough about what weāre doing before we start building. However, would like to know your perspective and others on this?
Great question, Kashyap. With the cause and effect technique. What you need to do is understand the consequences of what you are building. So, if you touch this area of the code, the result could be that it affects another area of the product. I would ask the question āwhat introduced this?ā to your team. Was it a result of new feature work? Did they fix a bug, that then caused a knock on effect? Iād also suggest that you ask developers to talk you through the code and whatās happening. They can really help you. Be that Question Asker and dig a bit deeper. Team members are in it together!
All the time! I often ask the question, are they adding value? What have I recently learnt? Do I need to adjust the regression suite? Do you have any view on this @atiqah
Great question @risko Testing on branches was a deep dive to prevent as many issues as possible. Testing on master was more of a light touch, but Iād explore any integration issues with other work that was going on.
Hey @steve_devonport Glad you could make the talk! This is a great question and Iām not sure I have the complete answer. I know itās all a balance. However, I do fundamentally believe that this whole ādeliver fastā approach causes us problems. We are always rushing. Not slowing down. As a result, we are introducing too many bugs. Perhaps the answer is to move away from fast delivery and towards thoughtful-delivery. More deep thinking is required. What do you think? Whatās your experience?
Hey @jonathan.hope1 Great question! No I wouldnāt. Also, it depends what it is. Perhaps you have a number of defects in the same area, so you could create a branch and dix them all there. I am curious why you have asked the question. Is branching strategy something you are exploring at work?
@conrad.braam !!! SO glad you could make the talk. Do you think the severity/impact could feed into the priority? Yes, Iād totally agree on your second point. You probably do have too many open bugs with prioritisation. However, what Iād argue is that what you touched on is important. The impact to the customer. If we fix this, does it add value to the customer? If it doesnāt add value to the customer, Iād question spending time over it. Do you have any further thoughts on this?
Hey @melissafisher we recently introduced feature branches to our process. So for example we would create a branch to develop and test a āfile uploadā feature. And once dev and testing and automation (unit+e2e) tasks are complete, and all issues are resolved we would merge to develop. Basically just allowing us to keep develop branch clean so we can release often. However we have discussed whether we should create branches for customer reported defects. I like your suggestion of grouping defects that occur in the same area and fixing them on the one branch, perhaps grouping them via triage. That is worth exploring, so I will suggest this to the team, thanks for the tip!
Nothing to add. I think you did go where I was keen to . Only fix bugs that impact the most customers, so know your customer, gauge the impact and fix those first, then look at the remaining bugs.
P1=showstopper
P2=workaround is possible so we can just document this for now and do P1ās in this hotfix
P3=stuff we should fix
P4=stuff we are unlikely to fix, but impacts the quality perception of the product
P5=stuff we wont fix in the next few sprints - so make sure people agree, this will probably NOT get fixed in reality.
Itās well known that Jira tries to route you to use priority and not to use impact, be aware of the bias it will bring when triaging work.
Hey @melissafisher! I canāt see this talk on the list of TestBash 2021 recordingsā¦ will the recording be available soon?
Definitely missing Iām sure one of the minions will sort it shortly. @friendlytester may have run out of hard drive space?
Hi Simon, Iām sure it will be available soon. Great that you want to watch it! Let me know what you think.
Hi Paul, It was a number of years ago I did the workshop. I think I remember asking the team where in the cycle was the root cause, then we added sticky notes and grouped the bugs under there, like requirements, design, build etc. Lisa and Janetās agile testing condensed book has a number of good diagrams in there that can help with this. You could, for example, draw out the SDLC and get the team to put the bugs against the area.
Hey @elliel Do you have an example of a problem you have come across that has been hard to distinguish? If so, what was it?
Hey Matthew! I canāt find your name when I try and @ you. Do you mean the bug has no purpose? Iād be interested to hear your thought further.