Bloggers Club October/November 2020: Most embarrassing test fails

🗄️ Archive
,
1

This months Bloggers Club topic suggestion comes from @conrad.braam

Tell us in a blog, about how hard it is to be a test engineer, not every bug you find is in the product sometimes it’s in your approach. Tell us about some test automation project that blew up and what you learned not to do again. Or, tell us about the bug that was in the environment, not in our product code, but it still wasted your whole day. Not necessarily the most grizzly fails, but the ones that made you change your working techniques.

The blog can be as short or as long as you want it to be.

How to get involved?

  • Write a blog on the above topic before November 20th :writing_hand:
  • Share a link to the blog here :eyes:
  • Receive lots of love from the community :heart:
  • Maybe even get a shout out from the Ministry of Testing Twitter account :wink:
2

I don’t have a blog to promote, but in an effort to assist others in being comfortable with sharing their failure.

I was working on a web based intranet product. One particular feature allows a user to upload an MS Office document. This file will then be converted to a PDF to allow embedding within a webpage. We were a MS based company, using MS based products, and we tested within that box.

One day we received a Mac Mini for testing. I had zero experience with it, but we needed to validate a customer issue that was happening in Safari. Pop in, validate the issue (on production), and I’m about to move on with my day when I get a thought.

“Mac and Linux machines allow filename characters that are invalid in Windows. What would happen if…”

So, I rename file.doc to be ..\..\..\..\..\..\..\..\..\..\file.doc and feed it into the (still production) system.

And the fun begins.

I’m sitting there waiting for the file to process. As far as the system is concerned, the conversion process has started and I’m given a processing prompt in the page. Refresh. Nothing. Refresh. Nothing. Refresh… It’s been a long time and I’m still getting nothing.

Head over to a back end monitoring system and it’s reporting the same thing. Even worse, none of the files that have queued up after are being processed. This is a parallel system and should be able to handle multiple files, but nothing is happening.

Convinced I’m about to be fired, I head over to the VP of dev and explain both what I did and what I’m experiencing. He laughs, essentially calls me an idiot (in a friendly, you should know better way), and we head over to IT.

IT tries to log in to the VM supporting this service and nothing. The machine is completely locked up. I’m sent on my way. IT will deal with it. I’m not to worry about it.

The resolution required IT pushing up a new VM, which luckily was templated/scripted and easy, and rerouting to the new machine. This particular service was isolated within it’s own VM, so the damage was limited and easy to hot swap. Within 15 minutes everything was back up and running again. I wasn’t fired, just lightly ridiculed for awhile.

Here’s what I’d actually done.

The service we were using to convert files was a third party, off the shelf application. It worked in Windows only and expected admin permissions. How it worked was that it would create temporary folders, based off the source file name, to store the source files and all converted files. When it was complete it would recursively delete the folders.

When I fed it a file with directory traversal in the filename it actually honoured it. Instead of recursively deleting its temporary work folder, it recursively deleted everything on the system from the root directory.

And that’s the story of how I ran a denial of service attack against my own company in the middle of the work day using a misnamed file.

10 Likes
3

I don’t know why but this line gave me a giggle! Thank you for sharing your story @davidshute :grin:

1 Like
4

As promised, I have drafted up my blog post to invite zero sympathy on one of my own automation failures. After all I asked for this topic.

I’m sure there are more failures than the one I have drafted up, but since I’m a slow blogger, I’m just going to add an obligatory graphic, some references to make it look like I’m not the only idiot who does this, and then hit publish once I’m over the tears. Before I run out of coffee, …flour, pasta, or toilet tissue?

Update: Here we go

2 Likes
5

My entry for this month. After the initial wave of “aren’t I a silly goose”, it was a fun writing task. I’ve not catalogued all my fails. But there are a few to enjoy.

1 Like
6

It’s amazing how easy it is to get forgiveness for a mistake. Aaron you have taken this seriously, and as per my intention with this topic shared a few key giveaways. Great writing style and layout.

2 Likes
7

Thanks @conrad.braam, really appreciate the feedback, and great topic. As you say mistakes happen, and 99.9999% of the time we can move on without much fuss. Failure no matter how small or big is a great teacher. I hope that being seen making mistakes and moving on, helps people not beat themselves up too much.

1 Like