For those that don’t know, TWiQ is where we gather every week on an audio stage to talk about all things quality and testing. The recording then gets published as a podcast. It’s a great way to dip your toes into the MoTaverse, as a listener or as a participant on stage.
Do you have a scary story to share?
Feel free to drop it in here in thread, we hope to share some of these on stage.
You are most welcome to join us on stage and share them there too.
I instantly go to this story. Still feel uneasy about it.
That reminds me of a time I fell into the regression testing complacency trap. I wonder if there’s a bias name for it.
The sales team rushed into our office to point out that ad revenue had suddenly taken a dip. Turns out myself and my fellow testing colleague had signed off a release without realising all the AdSense had disappeared. Both of us were so used to running regression that we no longer used the regression test checklist. We literally didn’t see what wasn’t in front of us – the missing AdSense. Was a big learning day for us both!
We resolved the issue quickly and through a post-mortem, it actually led to a push for automated checks for the essential features and a renewed energy of not taking things for granted or becoming complacent with regression tests. On reflection, it was very likely the start of moving towards a whole-team approach to quality instead of relying on QA folks to sign off a release.
One organization I worked for would use QA as the scapegoat whenever they could. Quality was not owned by the team. I can’t remember the issue exactly, but I had discovered a defect before it was released to the public. I wrote up the issue and made sure the developer working on it was aware of the issue. The issue was determined small enough not to be a blocker. On release, however, it created lots of trouble for clients. I think at least 4-5 people came to my desk that day asking if I’d written up the bug before it was released. I showed them the issue, the date it was written up, and the response of the team. I felt terrible because it felt like people were wanted to blame QA, but they couldn’t and I had to explain myself over and over, even though I did what I needed to do.
I can’t get scarier than this. Prepare yourself for a tale of horror and despair ….I shall begin.
10 years ago we built a Police Custody, Case, Crime and Intelligence management system that went to production for 2 customers.
The system was built on Microsoft Dynamics so we could use the workflow functionality and we built a front end in Silverlight because we thought we could get in cheap as the Police had loads of Dynamics licenses they weren’t utilising (that was the direction the CTO sent us in anyway). Once we had gone down this path, there was no going back.
Testers found bug after bug for years. Myself and colleague were automating the tests and there was no scenario with what we couldn’t cover other than the vast functionality we needed to get across. But it was difficult to find a scenario that would actually pass.
The team kept growing to tackle the growing backlog I managed the test team that grew from 4 to 11, the devs eventually grew from a few to 40 strong located on 2 floors. We built up a bug backlog eventually of more than 2000 bugs. Our automated set of core end to end tests had a 40% pass rate at best, it was usually in the 20’s.
One of the bugs the automated tests found was when a record was saved, the screen changed to a completely unrelated area of the system for a record that someone else had committed at the same time!!!
Did that stop us? No we ploughed on deploying it to 2 customers. Quality didn’t matter, the deadline ruled, we’d gone too far down this road with a flawed architecture. Just as a reminder this was for the Police!!! So only one decision was left…one by one….we RAN!!!
Fortunately, it didn’t live in production for long or exist as a product. This product was the reason the company was acquired by a corporate giant. So they shrunk the company and moved any remaining resource to another division. The building is now derelict on an industrial estate.
It’s said that when you walk past that building….you can still hear the screams of testers….sleep well
UAT approved it, but nobody knows who UAT is??
It is Schrödinger’s backup if we haven’t tested it
The bug appears in production every full moon
The fix worked, but everything else is broken
Scary in a different way but I was investigating a support case and asked for them to change a setting and then get fresh logs. I was advised that wasn’t possible as the kit was now hidden in a person’s house. Did not know we did that.
(I worked in video security)
I’ve been there too. The sinking feeling when you or someone else spots something you’ve missed. Worse when it’s something blindly obvious.
Complacency bias is a real risk in testing when you’re familiar with a system. For manual regression testing, I rely on the previous regression test plan, spend some time trying to ensure it covers everything that’s been added or removed since the last release and then disengage brain and follow the test plan like a robot. I liken it to driving long distance, you kinda go into auto pilot until something out of the ordinary happens. I’ve spotted so many issues this way.
Gather round as I tell the tale of my first day in tech support and FIRST call for the company , the user got and I kid you not “Catastrophic ERROR" . Picture it, Texas, 2011.
The system was for video and audio equipment for law enforcement. So naturally this was going a wild ride. After calling my boss, the QA at the time and other firmware QAs to try and replicate this issue, over the course of 4 hrs as an officer is trying to just get his video.
We got the dev who actually wrote the program for this product in to look at it. She takes one look and is like " Oh he’s missing the processing file to indicate it’s ready to be uploaded.” Me, my boss, the QA and everyone else who’d gathered to see this all look stunned. “…. So why use Catastrophic, if it’s so simple?" The developer looks to my boss and the speak in Mandarin for a minute and he drops his head into his hands. I said “What’s wrong ? “ He said still facepalming, " She used Google Translate to rewrite the error in English.”
I came as that error to the company Halloween costume party later that year and won.
….I shall begin.
