Your failures, not failures you've prevented?

Discussions
,
1

There was a lovely talk in a conference last week in Finland where Anssi Lehtelä talked about failure and declared he is a “failure specialist”.

image
image1920×1080 327 KB

He told stories of failures he has been contributing to, like his mistake making the newspaper on temporarily making secret phone numbers public; and deleting database tables from production by selecting 2/3 SQL lines and running deletion for more than you bargained for.

This left me thinking: a lot of my career is about failures I prevent hitting production. What are the stories of me failing, other than messing up communications and human-to-human relations. What are yours?

We started off the conversation in a small group, and I wanted to bring it here to continue. My big ones are:

  1. Pulling an all nighter after accidentally upgrading some millions of computers - could have been bad but was not, turned out the opposite.
  2. Not finding all the bugs I was expected to find in start of my career, and having QA of QA find them, and hitting a contractual clause of a discount because I just did not understand that test cases mean exploratory testing. Always.
  3. Wanting to show ‘naughty strings’ test helper list in a conference where google search results did not end up filtered. Taught me to write ‘naughty strings github’ ever since.

In the group we had great examples of mixing up test and production, and even a second hand story of accidentally making oneself a fugitive running away from police in production system by creating test data where it did not belong.

Have I just forgotten all the bad things I’ve caused or am I exceptionally lucky? Do everyone has a major story to tell?

4 Likes
2

You shouldn’t use the big list of naughty strings though. At least not the security strings in there, lots of them have stuff like drop table instead of a sleep or simple select, which makes it illegal or malicious. just an fyi :d

But yea I love to fail too and fast! You learn the most from failures, and expressing them.

I once used a credit card to test a payment system but it “apparently” wasn’t a test credit card but a real company credit card and they got billed quite a lot :rofl:

So always double check!

1 Like
3

I missed a y2k bug, cannot remember the exact details but I remember the feeling of htf did I miss this.

Thankfully we were on 24 hour support at the time and it was a small couple of lines fix and I believe we got it out before major markets opened.

Interestingly pretty much all of the competitors and similar tools missed it too so had the reminder I’m human after all.

Definitely learned a lot from it and I still use it as a learning story.

1 Like
4

Failure #1
My first proper job after university was testing chemical analysis equipment. On my first day I was told to test a £10,000 (which was two and a half times my salary in 1982) analyser, which was a one-off custom build. I plugged it in to the mains supply and it exploded. Lesson #1 - check if your company builds equipment for countries with different mains voltages than ours.

Failure #2
The next one isn’t test-related and it wasn’t me, but one of my staff. I sent them to install a intruder alarm at a launderette. Later that day, the client called to ask when the installer was coming. I called my installer and he said he had nearly finished the installation. It turned out that he had installed the alarm system in the dry cleaners next door to the launderette. Lesson #2 - never underestimate human stupidity.

Failure #3
In the mid-1990s I was testing the software for a microprocessor-based intruder alarm system manufacturer. I gave the new build the green light and it went into production. A couple of months later, we started to get returns. A trickle at first, then a flood. The new software was writing to the event log every minute instead of every hour, which it had done previously, and this massively exceeded the write cycle limit for the non-volatile memory chip, which eventually died.

By the time we diagnosed this, we had shipped 3,000 control panels that were now screwed to house and factory walls across Europe, and every one of them was going to fail within months. Because we sold into distribution channels, we didn’t even know who had bought the equipment so we couldn’t advise installers to do pre-emptive chip changes. The control panels were not networked, so every fix required a site visit and a new ROM and NVM (thank goodness they were socketed). Then the whole system needed to be reprogrammed because the original program was in the dead NVM chip.

It cost the company a fortune in replacement parts and compensation. Lesson #3 - ask the developers what they have done that was over and above what you asked them to do. Lesson #4 - some things can’t be tested via the user interface, so get under the hood if you can. Lesson #5 - don’t work on systems that can’t be fixed over-the-air (someone has to, but it doesn’t have to be you).

There are so many more I could tell you about, but these will do for now.

2 Likes
What mistakes have you made that were painful at the time, but made you a better person, tester, or leader?
5

I unwittingly deployed the wrong codebase to one of the sites owned by a large financial regulator bringing it down for about 10 hours. Took a couple of devs a few hours to bring it back up. I never did that again :sweat_smile:

1 Like
6

No, I should not show nude imagery in a conference. I should show big list of naughty strings on GitHub, and I should know what the things on that list do and in general not be blind about following any advice on copy pasting things to my apps.

On talking about actual failures, I also have made mistakes leading to company being billed. Mine was with Datadog Synthetic Monitoring, I did not realize the kind of bill it could generate me.

I don’t love to fail, not I fail fast. I love to learn. And I accept that to get forward in a world of uncertainty, both success and failure will teach me things where as not doing things won’t take me anywhere.