Deadlocking an application, what's your experience?

What’s a deadlock?

Deadlock is a state in which each member of a group waits for another member, including itself, to take action. An example:

Sample B is the deadlock, where 2 people wait for each other to perform an action

Sharing my story but I like to hear yours also!

I was just new on the project and so I had to test/explore this application when most of our team members (all devs) had holidays and I had access to the only “admin” account on the test environment. I was playing around with permissions, and yes you can see it coming, I deleted my own admin permissions. :man_facepalming:

At this point a deadlock occurred and I was forced to logout.

Later that day… they were mad at me for doing it… they asked me why I did it?
So I reversed the question to “why did you allow me to do it?”

So yea there are a lot of types of deadlocks, what is your experience deadlocking an application?
Perhaps you can give me some new idea’s :wink:

4 Likes

That is an unusual deadlock. I never understood that one until recently having this same question/requirement surfacing in a new project.

I’m not really following the animation very well, I assume it’s just a random graphic you found. But in my previous life as a coder, it was the source of a substantial number of bugs. Mainly caused by using low level languages that don’t have concurrency built in and then using that language to do highly performant parallel pipeline processing. So the deadlocking at an application level is a new one on me. Mostly I do hit it at the app layer when licensing enforcement comes into play.

  1. User consumes a license point,
  2. license point not returned because the resource lost network connectivity
  3. After this happens a few times, user is unable to “use” a resource due to the quota of license points being all used up
    Hard to find bug in testing since test environments often have unlimited license allocated - instead, get your devs to allow setting license size to 1 #TestAbility
1 Like

Actually I don’t see why a user shouldn’t revoke their own permissions. But can see why that tripped up the application. That is a typical deadlock scenario.

So yeah, I see them all the time. Any multi-threaded application can deadlock two-threads can deadlock if they are processing the same objects in different orders.

But more typically these days we see web API(s) which update tabes and the order is inconsistent then deadlocks occur. Same with distributed locks using dynamo or redis. If app 1 has lock A and wants B and app 2 has lock B and wants A then boom. Can be very hard to identify at scale.

2 Likes

I agree but in my case I would have to promote another user first before I could revoke my own permissions.

Yup from the wiki, basically it shows in (B) that 2 users are waiting for each others actions.

I don’t know 100% if this counts as deadlocking the applications, but some 6-7 years ago, at my first job I accidentally did something little problematic. It was an e-commerce app with asp.net Storefront as backend, and if you logged into the fronted with an admin account, in the same browser session you would also get logged in as admin in the storefront part.

I was testing something with a particular admin account (which had a unique ID granting it some automatic discounts) and changed something on the front-end part in the account profile sections (for testing needs, I don’t recall exactly what it was) but, shortly afterwards other people started complaining that they can’t login to storefront using any account! I was scared shitless! :grimacing:

They eventually restored the admin accounts from backup, but I don’t think anybody ever bother to get to the bottom of it or to fix the root cause.

1 Like

@mirza That sounds like you merely triggered a lockout, which is a kind of deadlock in my books, but just the lockout was on the wrong object, the app, not the account.

1 Like

For many years I worked on an application that worked on a hierarchy of customers and accounts (like a hierarchy of folders and files in an operating system). In order to prevent deadlock, there was the rule that whenever code wanted to do something to the hierarchy it took out locks first. The relevant bit of the hierarchy had to be locked top to bottom, and then in customer / account number order.

If you don’t do this then you can get interesting problems. For instance, user X is trying to transfer something from account A to account B, at the same time as user Y is trying to transfer something from account B to account A (i.e. the same accounts but the opposite direction). The code dealing with user X locks account A then tries to lock account B. The code dealing with user Y locks account B then tries to lock account A. Oops.

It’s easy to have this kind of bug where you lock things based on the direction of some operation, which can vary / be in opposite directions. Instead you need to lock things based on something that can’t change so easily.

1 Like

Going back to your point @kristof, this points to the lack of requirements, or poorly implemented requirements. It seems like no-one thought there should be requirements like these:

  1. An admin user must be created as soon as possible
  2. There must always be at least one valid admin user

Ow you have no idea :stuck_out_tongue: They didn’t even had proper analysis :stuck_out_tongue:
I’m wondering now if they even knew what acceptance criteria/requirements were

2 Likes

Oh, wow, I am just so thinking you can create a large recursion type lockout where you lock all the accounts at once, in a long chain of them, and then walk away and wait till a developer gets so agro they fix the design.

2 Likes

3a5

2 Likes