I don’t have a blog to promote, but in an effort to assist others in being comfortable with sharing their failure.
I was working on a web based intranet product. One particular feature allows a user to upload an MS Office document. This file will then be converted to a PDF to allow embedding within a webpage. We were a MS based company, using MS based products, and we tested within that box.
One day we received a Mac Mini for testing. I had zero experience with it, but we needed to validate a customer issue that was happening in Safari. Pop in, validate the issue (on production), and I’m about to move on with my day when I get a thought.
“Mac and Linux machines allow filename characters that are invalid in Windows. What would happen if…”
So, I rename
file.doc to be
..\..\..\..\..\..\..\..\..\..\file.doc and feed it into the (still production) system.
And the fun begins.
I’m sitting there waiting for the file to process. As far as the system is concerned, the conversion process has started and I’m given a processing prompt in the page. Refresh. Nothing. Refresh. Nothing. Refresh… It’s been a long time and I’m still getting nothing.
Head over to a back end monitoring system and it’s reporting the same thing. Even worse, none of the files that have queued up after are being processed. This is a parallel system and should be able to handle multiple files, but nothing is happening.
Convinced I’m about to be fired, I head over to the VP of dev and explain both what I did and what I’m experiencing. He laughs, essentially calls me an idiot (in a friendly, you should know better way), and we head over to IT.
IT tries to log in to the VM supporting this service and nothing. The machine is completely locked up. I’m sent on my way. IT will deal with it. I’m not to worry about it.
The resolution required IT pushing up a new VM, which luckily was templated/scripted and easy, and rerouting to the new machine. This particular service was isolated within it’s own VM, so the damage was limited and easy to hot swap. Within 15 minutes everything was back up and running again. I wasn’t fired, just lightly ridiculed for awhile.
Here’s what I’d actually done.
The service we were using to convert files was a third party, off the shelf application. It worked in Windows only and expected admin permissions. How it worked was that it would create temporary folders, based off the source file name, to store the source files and all converted files. When it was complete it would recursively delete the folders.
When I fed it a file with directory traversal in the filename it actually honoured it. Instead of recursively deleting its temporary work folder, it recursively deleted everything on the system from the root directory.
And that’s the story of how I ran a denial of service attack against my own company in the middle of the work day using a misnamed file.