He told stories of failures he has been contributing to, like his mistake making the newspaper on temporarily making secret phone numbers public; and deleting database tables from production by selecting 2/3 SQL lines and running deletion for more than you bargained for.
This left me thinking: a lot of my career is about failures I prevent hitting production. What are the stories of me failing, other than messing up communications and human-to-human relations. What are yours?
We started off the conversation in a small group, and I wanted to bring it here to continue. My big ones are:
Pulling an all nighter after accidentally upgrading some millions of computers - could have been bad but was not, turned out the opposite.
Not finding all the bugs I was expected to find in start of my career, and having QA of QA find them, and hitting a contractual clause of a discount because I just did not understand that test cases mean exploratory testing. Always.
Wanting to show ânaughty stringsâ test helper list in a conference where google search results did not end up filtered. Taught me to write ânaughty strings githubâ ever since.
In the group we had great examples of mixing up test and production, and even a second hand story of accidentally making oneself a fugitive running away from police in production system by creating test data where it did not belong.
Have I just forgotten all the bad things Iâve caused or am I exceptionally lucky? Do everyone has a major story to tell?
You shouldnât use the big list of naughty strings though. At least not the security strings in there, lots of them have stuff like drop table instead of a sleep or simple select, which makes it illegal or malicious. just an fyi :d
But yea I love to fail too and fast! You learn the most from failures, and expressing them.
I once used a credit card to test a payment system but it âapparentlyâ wasnât a test credit card but a real company credit card and they got billed quite a lot
Failure #1
My first proper job after university was testing chemical analysis equipment. On my first day I was told to test a ÂŁ10,000 (which was two and a half times my salary in 1982) analyser, which was a one-off custom build. I plugged it in to the mains supply and it exploded. Lesson #1 - check if your company builds equipment for countries with different mains voltages than ours.
Failure #2
The next one isnât test-related and it wasnât me, but one of my staff. I sent them to install a intruder alarm at a launderette. Later that day, the client called to ask when the installer was coming. I called my installer and he said he had nearly finished the installation. It turned out that he had installed the alarm system in the dry cleaners next door to the launderette. Lesson #2 - never underestimate human stupidity.
Failure #3
In the mid-1990s I was testing the software for a microprocessor-based intruder alarm system manufacturer. I gave the new build the green light and it went into production. A couple of months later, we started to get returns. A trickle at first, then a flood. The new software was writing to the event log every minute instead of every hour, which it had done previously, and this massively exceeded the write cycle limit for the non-volatile memory chip, which eventually died.
By the time we diagnosed this, we had shipped 3,000 control panels that were now screwed to house and factory walls across Europe, and every one of them was going to fail within months. Because we sold into distribution channels, we didnât even know who had bought the equipment so we couldnât advise installers to do pre-emptive chip changes. The control panels were not networked, so every fix required a site visit and a new ROM and NVM (thank goodness they were socketed). Then the whole system needed to be reprogrammed because the original program was in the dead NVM chip.
It cost the company a fortune in replacement parts and compensation. Lesson #3 - ask the developers what they have done that was over and above what you asked them to do. Lesson #4 - some things canât be tested via the user interface, so get under the hood if you can. Lesson #5 - donât work on systems that canât be fixed over-the-air (someone has to, but it doesnât have to be you).
There are so many more I could tell you about, but these will do for now.
I unwittingly deployed the wrong codebase to one of the sites owned by a large financial regulator bringing it down for about 10 hours. Took a couple of devs a few hours to bring it back up. I never did that again
No, I should not show nude imagery in a conference. I should show big list of naughty strings on GitHub, and I should know what the things on that list do and in general not be blind about following any advice on copy pasting things to my apps.
On talking about actual failures, I also have made mistakes leading to company being billed. Mine was with Datadog Synthetic Monitoring, I did not realize the kind of bill it could generate me.
I donât love to fail, not I fail fast. I love to learn. And I accept that to get forward in a world of uncertainty, both success and failure will teach me things where as not doing things wonât take me anywhere.
