On July 19, approximately 8.5 million Windows machines were blocked causing flight cancellations, banking disruptions and media outages around the world. Major US airlines, including American Airlines, United Airlines, and Delta had to cancel flights due to communication problems. Banks and stock exchanges, including the London Stock Exchange, Lloyds Bank, and South Africa’s Capitec, faced similar problems. The failure also affected the Visa and Mastercard payment gateways, according to DownDetector data.
The outage led to serious financial distress. For instance, the cancelation of almost 7,000 flights by Delta could cost the company from $350 million to $500 million. According to some estimates, the total direct loss facing the US Fortune 500 companies, excluding Microsoft, was $5.4 billion. The healthcare sector has been hit the hardest, with projected losses of $1.94 billion, followed by the banking sector with $1.15 billion in estimated damages. The airline industry also experienced significant disruptions, leading to an estimated $860 million in losses. Fortune 500 companies alone could incur direct losses of $5.4 billion.
What went wrong
The outage was caused by errors in an update of the Falcon security platform by information security solutions provider CrowdStrike, as the company later explained. Interestingly, the update was successfully tested on March 5, but the error could not be noticed due to a bug in the diagnostic software.
CrowdStrike also noted that it usually provides security content configuration updates in two ways: one through Sensor Content, which comes with the Falcon Sensor component, and the other through Rapid Response Content, which flags new threats using various behavioral pattern-matching methods. The latter was the one that contained the previously undiscovered bug.
Why did this mistake lead to blue screens around the world? The reason lies in the relation between this kind of endpoint protection software like Falcon and operating systems: there is no way to limit such software from controlling the operating system, as doing so would open the possibility for a virus to take over. This scenario would negate the very purpose of having a security application in place, as it would allow malicious entities to bypass the protection measures entirely.
Gradual upgrades and regular backups
Despite the significant impact of the recent incident on companies and organizations, it is unlikely that there will be a widespread abandonment of CrowdStrike products. Solutions like Falcon are deeply embedded in IT infrastructures and have been developed and refined over decades. Replacing them is time-consuming and costly. In addition, there is no guarantee that alternatives would not lead to the same troubles.
However, this incident shed light on some burning issues in the tech industry. One of them is the lack of diversity. Nowadays the market is dominated by just a few major vendors, and this concentration of control is precisely why the impact of the incident was so widespread. To mitigate such risks in the future, it’s crucial to develop and invest in alternative solutions, including cloud-based options. This is the key takeaway we should derive from this situation.
Furthermore, while accountability for the accident rests with CrowdStrike, businesses also need to incorporate new approaches to security. One of them is to constantly back up their data. Companies that do that regularly probably were also less impacted by this outage, in my opinion. Some system software usually updates itself over the night or in the morning. If something goes wrong, the firm can just roll that out. So another suggestion for business, and we’ve been saying that again and again for decades, is that you should have some backup procedure applied, running, and regularly tested.
I also think that companies that keep their infrastructure in the cloud, coped with the consequences of this outage quicker than others thanks to virtualization and API-based scripts. For AWS-hosted and Microsoft Azure-hosted virtual machines, the instructions are usually published in a matter of hours. Moreover, it does not take much time to imply those instructions compared to doing that for a full park of bare metal servers. Therefore, probably more firms would switch to cloud-based solutions. If 20% companies would do that, it would be a fantastic win for our industry. But I believe, only 5-15% would actually go for that.
Future updates
In addition, future updates are also better deployed gradually. It means first upgrading a small subset of systems, then monitoring their performance, and extending changes to a larger group of systems. With this strategy in place, it would take more time for businesses to update everything, but it would help them to avoid such massive damages as we have seen today.
There are some steps that regulators could take too. Many companies create a risk model to assess potential threats and choose appropriate cyber defense solutions. However, regulators sometimes mandate specific cybersecurity measures without considering if all businesses truly need them. For example, they might require the installation of antivirus software without verifying its necessity for every company. As a result, some businesses end up purchasing cybersecurity solutions just to comply with regulations, rather than based on their actual needs. It’s likely that from 50% to 90% of affected companies would not have been impacted if they had not installed CrowdStrike or other EDR and XDR software products solely for compliance reasons in the first place.
Overall, I hope that the situation will bring more positive change to the industry and help transition to safer cybersecurity practices.
We’ve featured the best business cloud storage.
This article was produced as part of TechRadarPro’s Expert Insights channel where we feature the best and brightest minds in the technology industry today. The views expressed here are those of the author and are not necessarily those of TechRadarPro or Future plc. If you are interested in contributing find out more here: https://www.techradar.com/news/submit-your-story-to-techradar-pro