A massive recovery effort continued Monday after a defective software update from CrowdStrike led to a global IT outage that impacted about 8.5 million devices using the Microsoft operating system.
The outage, which struck less than 1% of Windows devices, impacted critical infrastructure providers across the globe, leading to the cancellation of thousands of commercial flights since Friday. The outage temporarily shut down 911 emergency services in several U.S. states and major hospitals had to cancel surgeries.
CrowdStrike CEO George Kurtz apologized for the incident, saying a defect in the Falcon content update for Windows hosts led to the outage. Systems using Linux and Mac operating systems were not impacted.
“Nothing is more important to me than the trust and confidence that our customers and partners have put into CrowdStrike,” Kurtz said in a letter sent Friday. “As we resolve this incident, you have my commitment to provide full transparency on how this occurred and steps we’re taking to prevent anything like this from happening again.”
The incident is already raising questions over the vulnerability of critical infrastructure systems and government insight years after the Sunburst attacks led to a far-reaching supply chain attack on SolarWinds.
Mark Montgomery, senior director of the Center on Cyber and Technology Innovation at the Foundation for the Defense of Democracies, said federal officials were once again caught off guard by the major cyber disruption.
“The White House has told us a few times over the last three or four years that they have it all under control,” Montgomery said. “I’m not sure what their mechanism of control is.”
CrowdStrike said the outage was the result of a sensor configuration update on Friday that triggered a logic error, causing systems to crash and leading to blue screen errors. The defect was not the result of a cyberattack, according to CrowdStrike.
Recovery roughly underway
Cybersecurity and IT experts said users are having major difficulties in recovery efforts, despite workarounds and guidance released by CrowdStrike and Microsoft.
Delta Air Lines CEO Ed Bastian apologized to customers as more than 3,500 Delta and Delta Connection flights were canceled on Friday and Saturday, with additional disruptions later in the weekend.
Bastian said the outage impacted a number of applications using the airline’s Windows operating system, particularly a tool used to help track flight crews.
Microsoft on Saturday released an updated recovery tool to help users expedite the process of getting their systems operational. Users have two repair options: recovery from Windows Preinstallation Environment, which involves production of boot media; or organizations can use safe mode and log in with local admin privileges.
Microsoft said the WinPE recovery is the preferred option and does not require local administrative privileges. Users may still need to manually enter a BitLocker recovery key and then repair the system.
Microsoft has sent out hundreds of engineers and other experts to help customers recover from the outage. The company is also working with Google Cloud Platform and Amazon Web Services to share information with Microsoft users operating on those platforms.
CrowdStrike on Sunday said it has tested a tool to accelerate the recovery process and is working on a process that will allow affected users to opt in.
The Cybersecurity and Infrastructure Security Agency warned that opportunistic hackers were taking advantage of the crisis to target users under the pretense of offering recovery services. CrowdStrike said malicious actors were specifically targeting Latin American customers with a ZIP file that contains a HijackLoader.
The outage is having residual impacts on major infrastructure providers across the globe. U.S. Transportation Secretary Pete Buttigieg said the department is getting reports of continued flight disruptions and customer service problems Delta, noting that hundreds of complaints have been filed with DOT.
