Failure #1
My first proper job after university was testing chemical analysis equipment. On my first day I was told to test a £10,000 (which was two and a half times my salary in 1982) analyser, which was a one-off custom build. I plugged it in to the mains supply and it exploded. Lesson #1 - check if your company builds equipment for countries with different mains voltages than ours.
Failure #2
The next one isn’t test-related and it wasn’t me, but one of my staff. I sent them to install a intruder alarm at a launderette. Later that day, the client called to ask when the installer was coming. I called my installer and he said he had nearly finished the installation. It turned out that he had installed the alarm system in the dry cleaners next door to the launderette. Lesson #2 - never underestimate human stupidity.
Failure #3
In the mid-1990s I was testing the software for a microprocessor-based intruder alarm system manufacturer. I gave the new build the green light and it went into production. A couple of months later, we started to get returns. A trickle at first, then a flood. The new software was writing to the event log every minute instead of every hour, which it had done previously, and this massively exceeded the write cycle limit for the non-volatile memory chip, which eventually died.
By the time we diagnosed this, we had shipped 3,000 control panels that were now screwed to house and factory walls across Europe, and every one of them was going to fail within months. Because we sold into distribution channels, we didn’t even know who had bought the equipment so we couldn’t advise installers to do pre-emptive chip changes. The control panels were not networked, so every fix required a site visit and a new ROM and NVM (thank goodness they were socketed). Then the whole system needed to be reprogrammed because the original program was in the dead NVM chip.
It cost the company a fortune in replacement parts and compensation. Lesson #3 - ask the developers what they have done that was over and above what you asked them to do. Lesson #4 - some things can’t be tested via the user interface, so get under the hood if you can. Lesson #5 - don’t work on systems that can’t be fixed over-the-air (someone has to, but it doesn’t have to be you).
There are so many more I could tell you about, but these will do for now.