Hugops to everyone involved in dealing with the global outage related to Microsoft/Crowdstrike.
Thanks for already sharing the following, @ipstefan:
What else have you read about the outage? What information can we share to learn from? What information can we share about business continuity testing, disaster recovery testing, resilience etc?
Share links and your thoughts on this thread as the story unfolds.
Y2K has arrived? 
This is basically what we were all worried about with Y2K, except it’s actually happened this time !
— Troy Hunt
A helpful unbiased high-level summary so far from Reuters.
From @w4dd325 on the MoT Slack:
From what I have read, CrowdStrike automatically sent out a faulty channel file. This has caused PC’s running Windows (mainly Win 10 I think) to get stuck in a boot loop / BSoD (Blue screen of death).
There is a workaround to remove the faulty file (but if you have bitlocker you might not be able to get in via safe mode to do the workaround).
CrowdStrike have halted all updates until this is resolved, so if you haven’t been hit by it already, you should be ok…John Hammond explains it here:
Your horoscope, not found.
Cash only at the shops, don’t get caught out!
Struggling to find helpful updates on microsoft.com.
Nothing on their news page and nothing on their LinkedIn page.
And a keyword search for “Crowdstrike” on their help pages returns nothing relevant.
So we know that the BSOD occurred after a reboot of Windows that was done because the update of Crowdstrike required it. This meant at boot, Windows had a conflict with a Crowdstrike corrupted file or its contents.
So to test it they just had to do the same. Or am I missing something?
What Crowdstrike has done was stop the updates as soon as they found out what they caused.
I heard from people with Crowdstrike installed that their company’s IT department sent them an email to not accept the update if allowed to be installed.
I am wondering next, as a company is it wise to keep auto-update ON for all the employees’ computers for any tool? Or should a check be done first by IT for each update of any essential tool?
As far as I could find using the jobs published online they do their testing in India, for the marketing websites with QA Analysts(write and execute test plans, and cases), and with SDET(automate things) for the apps. An example that might reveal their focus:
They are currently looking for broader solutions as far as I saw.
The renaming of the Crowdstrike corrupted/faulty file might not work for everyone. On top, it’s required generally to be done by IT hands-on for each employee, together with a reboot of the machine.
There were two issues so it would be good to be clear which one we’re referring to, although both are related to Microsoft:
“The CrowdStrike flaw was not the only problem facing Microsoft. On Thursday, some Microsoft clients in the central United States, including some airlines, were affected by an outage on its cloud service system, Azure.”
https://www.nytimes.com/2024/07/19/business/microsoft-outage-cause-azure-crowdstrike.html
The conclusion in the NY Times article was clear:
"The outages underscored an uncomfortable reality that software companies face few liabilities for major disruptions and cybersecurity incidents. The economic and legal penalties for such significant outages can be so minimal that companies are not motivated to make more fundamental changes. While a car manufacturer would face stiff penalties for faulty breaks, a software provider can often issue another update and move on.
“Until software companies have to pay a price for faulty products, we will be no safer tomorrow than we are today,” Mr. Parenty said."
that’s a powerful visual!
About the CS fix:
“The issue has been identified, isolated and a fix has been deployed.” - written by lawyers who don’t understand the issue. The missing part is “fix has to be applied manually to every impacted system”
Did CS just cause the biggest impact in the history of software?
Apparently “Crowdstrike customers account for 298 of the Fortune 500…”
And then to understand some context, I’ve gone through several people comments:
You can read more here: Reddit - Dive into anything
Crowdstrike has provided a technical report should you be interested.
Really nice article.
Just one thing - Please check the link for this article, it seems it is wrong
I don’t see the value of that comparison:
Can people not have surgery, use their bank account, fly, or call emergency services if Meta is down?
