A line of code should have read “0000291.” Instead, it read “0000000.” That was it. (Sort of.) Within hours, millions of computers worldwide were “borked”*—plunged into Microsoft’s “blue screen of death (BSoD).” The victims included Windows machines in the Black Hills, and All Net Connect techs spent a couple long days and a long night bringing hundreds of machines back to life.
The bad code was in an update of a cybersecurity program called CrowdStrike Falcon, which All Net Connect recommends. CrowdStrike has prevented untold numbers of disasters, and it continues to do that.
See “Dave’s Garage”
Still, the crash was a calamity. How could three bad zeros “bork” the world? For a clear and darkly entertaining answer, we recommend a YouTube channel called Dave’s Garage. Dave Plummer is a retired Microsoft software engineer who knows things. Dave doesn’t talk down to his audience, so prepare yourself for sentences like this: “The only things that go in the kernel mode are things that have to, like the thread scheduler and the heat manager and functionality that must access the hardware, such as the device driver that talks to a GPU across the PCIU bus.”
You might relish those details if you speak IT. If not, hang in there. Dave’s Garage has payoffs, even for civilians.
For example, you’ll learn the important difference between “user mode” (where mere applications run) and “kernel mode” (the sacred home of operating systems.) You’ll also learn how CrowdStrike provides faster protection by booting up immediately, alongside the Windows “boot driver,” which resides in the holy domain of the kernel mode. That means CrowdStrike’s protection starts instantly, which is good—except when it crashes. Then, says Dave, “…your system is completely borked.”
Directions to Dave’s
Dave offers a couple of videos about the crash:
- CrowdStrike Explained by a Windows Developer (13 minutes), describes what happened.
- Crowd Strike: Latest News, Lesson Learned from a Retired Microsoft Engineer, (17 minutes), provides context, such as:
- Who’s to blame (not Microsoft, but it’s complicated).
- How to prevent similar crashes (even more complicated).
- Wild conspiracy theories (debunked).
And one more thing
Dave’s deadpan delivery also is salted with nuggets of cyber-gallows humor. For example, describing how to eliminate the faulty “0000291” code, Dave says: “You have to find the corrupted Channel 291 Update File in the CrowdStrike folder and delete it and reboot. And so, that’s where we’re at—a whole lot of techs standing around with their discs in their hands waiting for safe mode to boot eight million blue-screened Windows machines.”
But don’t try that at home. We’ll also add that All Net techs are doing a little more than that. They’re monitoring the CrowdStrike recovery, and they’re always looking for the best ways to protect our clients.
*PS: The verb “to bork”
Merriam Webster says it comes from the harsh criticism of Judge Robert Bork, who was nominated for the Supreme Court in 1987. His nomination failed after a bitter fight, and “bork” was later adopted by gamers and programmers to describe sudden, epic electronic failures.