Even minor IT glitches can cost millions, technology leaders say

This audio is generated automatically. Please let us know if you have any feedback.

Dive short:

The average annual downtime due to high-impact IT outages is 77 hours, with hourly costs of up to $1.9 million, according to a Tuesday report published by NewRelic. The observation company commissioned ERT to interview 1,700 technology professionals in April and May.
IT teams spend an average of 30% of their time resolving disruptions, respondents said – the equivalent of 12 hours per 40-hour work week. The top causes of unplanned outages reported over the past two years include network outages, issues with third-party services, and human error.
Major disruptions, such as the global event caused by a flawed CrowdStrike Windows system update in July could bring operations to a standstill, said Nic Benders, chief technical strategist at New Relic. But small problems can also have a snowball effect. “It doesn’t have to be that way CrowdStrike so it would be a three-alarm fire,” he told CIO Dive. “With a relatively minor technical problem you can disable the business function of IT.”

Diving insight:

All it took was an automated software update sent shortly after midnight EST on July 19 to crash millions of Windows computers around the world. The CrowdStrike update was live just over an hour but the effects were felt for days, like several major airlines rushed to restart workstations and restore operations, grounding thousands of flights.

“The CrowdStrike incident is in a league of its own because it had a disproportionate impact on some of the largest companies in the world – it was a poison pill that those companies had to fix on their own,” Benders said.

As executives assessed the losses, they mounted $5.4 billion among Fortune 500 companies and Delta Air Lines cost $500 million in just five days, IT resilience and recovery planning were the focus.

“When something like an outage occurs at a cloud provider, it’s rare that the problem is initially apparent,” Benders says. “Your alarm goes off, support tickets light up and you’re in chaos, but in that first step you’re just trying to characterize the nature of the problem.”

While major supplier outages and cyber events often make headlines, scenarios involving thousands of cuts and smaller outages are much more common. The average number of annual outages among respondents was 232, with more than half of businesses experiencing low-impact disruptions every week.

Costs can be difficult to estimate, especially for low-impact issues. But the minutes or hours it takes for engineering teams to identify and fix even minor IT glitches add up. Over the course of a year, teams spend approximately 134 hours – the equivalent of nearly six full days – resolving IT outages at all levels of business impact.

“It all comes down to dollars,” Benders said. “If there were no costs involved, I would tackle a thousand incidents a week. That is not an incident at all.”