Reducing Downtime Costs In Automotive Manufacturing With Improved Cyber Protection

A combination of legacy IT systems on the factory floor, lack of skilled local IT support, and growing data volumes driven by emerging technologies like smart sensors and IoT devices is driving up downtime costs in automotive manufacturing. The future of the automotive manufacturing industry depends on its ability to improve business continuity and reduce downtime costs by deploying data-center-quality cyber protection at the plant level.

By James R. Slaby

The automotive manufacturing sector faces a complex assortment of technology and security challenges as it evolves into an ever more data-centric business. Downtime costs in manufacturing are among the highest of any industry: thousands of dollars are lost for every second that production lines are stalled by IT issues, like process-control server or workstation failures. The automotive manufacturing industry has also become a favorite target of cybercriminals looking to steal sensitive intellectual property or extort cash by locking up critical systems via ransomware attacks. Further, the amount of data that the industry accumulates and processes continues to surge with its adoption of cloud services, advanced sensors, Internet of Things (IoT) devices, artificial intelligence (AI) / machine learning (ML), and self-driving capabilities.

Today and for the foreseeable future of automotive manufacturing, there is new urgency to reduce downtime by improving data protection, including backup and disaster recovery, as well as cyber security to defend against an increasingly sophisticated array of attackers, from hostile state actors to online criminal gangs to malicious insiders.

With some modest manufacturing downtime tracking, it becomes easy to understand why IT reliability issues are so costly to the automotive industry. The modern manufacturing process is an intricately interconnected web of people, IT servers, workstations, software, robots, networks, and far-flung supply chains. The unexpected failure of a single server can bring a production line to a halt, and every minute of downtime has cascading effects across the organization and its supply chain. The sector loses an estimated $50B each year in unplanned downtime costs, with 42% of losses attributable to simple asset failures, like a computer hard disk drive mechanically malfunctioning.

Tech industry researcher Aberdeen Group estimates the downtime cost in manufacturing at $10,000 to $250,000 per hour. The problem is widespread: nearly every factory loses at least 5% of its productive capacity from downtime, and many lose up to 20%. By conservative estimates, the typical automobile manufacturing plant loses $22,000 for every hour of downtime.

The costs of cyberattacks on the industry are also significant and growing. In a 2019 study, Accenture Security noted a 48% jump in cybercrime costs to the automotive sector (from $10.7B in 2017 to $15.78B in 2018), placing the total economic value at risk in the industry at $505B. Tech analyst firm Juniper Research estimates that a single attack can cost automakers as much as $1.1B, and the trend toward connected vehicles has led it to predict that overall costs to the industry from cyber-attacks could reach $24B by 2023.

While automotive manufacturing technology must evolve, a common source of issues is legacy IT systems like older process control servers. It is not uncommon for such systems to use application software that is old, stable and unchanging, and thus needs to run on older versions of the server operating system (OS) like Windows Server 2003. The need to maintain this environment in its steady state presents several challenges to data protection and security, both of which are critical to reduce downtime in manufacturing.

• In the event of a server hardware failure, it can be difficult to restore the system from backup to a new server that does not have an identical hardware configuration.

• Backup and recovery are likely managed using simplistic legacy data protection tools that slow down the recovery process. The limited features and slow performance of such tools can make it difficult to perform backups with adequate frequency to meet the organization’s recovery point objective (RPO, the time since the last backup was performed) and recovery time objective (RTO, the time it takes to complete a recovery operation after a failure). On an aging server, the overhead of backup management can also be an excessive burden on system performance.

• The security vulnerabilities presented by an unpatched, outdated revision of a server OS increases the likelihood of a successful malware attack. At present, the two most prevalent such malware threats are cryptojacking and ransomware. Cryptojacking, which steals server resources to surreptitiously mine cryptocurrency on behalf of distant cybercriminals, is the less urgent threat, as it adversely affects server performance more than uptime. However, the wear-and-tear it inflicts on the system by consuming CPU, memory, power and cooling resources (even during otherwise idle periods) shortens the system’s hardware life. The more grievous malware threat to the sector is ransomware, which encrypts all files on an infected server or workstation and then demands an extortion fee to unlock them. This brings the system and its associated manufacturing processes to an abrupt halt, and can spread over a local network to afflict other servers and workstations.

• The process of recovering from a server or workstation hardware failure or malware attack can be hindered by the lack of skilled IT staffers on the factory floor. Any data protection regimen which is not highly automated and simple enough to be operated by plant engineers with limited IT skills will delay server recovery and prolong the downtime event. This issue is exacerbated by the fact that some systems like process control servers may operate in standalone environments on the factory floor, with neither local nor wide-area network connections. This means that any recovery operation must be effected locally and cannot rely on remote access by centralized IT staffers and systems.

In light of these challenges to maintaining high uptime, and the inevitability of periodic system failures, it’s important to shore up the ability to quickly restore systems. The first step to establishing a business continuity plan for automotive manufacturing companies is employing easy, efficient and secure backup – a simple, proven strategy to reduce downtime in manufacturing and improve overall factory floor IT reliability.

Broadly speaking, the typical automotive manufacturing IT environment has the data protection, security infrastructure and staffing characteristics of an edge IT environment: a small, remote data center rather than a large, centralized one. It has greater security exposures, less up-to-date systems and tools, and lower IT skills than the typical large data center.

Meanwhile, the unique data protection challenges of legacy IT systems combined with their increased exposure to cyberattacks at the plant level require the kind of sophisticated data protection and cyber security tools used to improve overall reliability and uptime in large data centers. Of course, they need to be delivered at a cost, scale, and ease of operation appropriate to edge environments.

Looking forward, the challenges will only get harder to reduce downtime in manufacturing. The adoption of new technologies like IoT, connected vehicles, and AI/ML will soon result in data volumes and downtime costs at the plant level that approach the scale of many large data centers. Closing the reliability gap between the two – by endowing the edge-like IT environment of the factory floor with the cyber protection (data protection plus cyber security) tools of a large data center – will be a key tactic to reduce downtime in manufacturing (and reduce downtime costs in manufacturing) due to hardware failures, software issues, configuration and operation errors, and malware attacks.

To reduce downtime in manufacturing and its associated costs, automotive manufacturers must address current IT gaps by investing in cyber protection solutions with the following characteristics:

As a leading technology partner to the manufacturing industry in general and the automotive manufacturing sector in particular, Acronis offers a variety of resources to help industry IT pros address their uniquely complex and growing cyber protection challenges:

• For a quick glance at Acronis automotive manufacturing disaster recovery and backup solutions, check out this overview.
• To learn how a leading European automaker is protecting its factory floor systems with Acronis Backup, read this case study.
• For a deeper understanding of the threat of ransomware to the automotive industry and how to combat it, read this article.
• To access an ongoing series of articles on how Acronis cyber protection technology is being deployed by cutting-edge developers of automotive technology in the motorsports industry, visit the Motorsport Technology website.