What The U.S. Consulate Database Crash Teaches Companies About Data Protection

Most travelers trying to get U.S. travel papers processed during the past couple of weeks know about the U.S. consulate database crash that impacted 200,000 or more visa-seekers worldwide. The culprit, a simple software patch, caused the U.S. State Department’s central passport and visa system to crash on July 19. While the data was backed up, the system was not, causing visa processing to grind to a halt during the slow recovery process. 

There's a data protection lesson from this situation that all companies can learn from: Quick, reliable recovery depends on backup strategies that protect all company data, including configuration data and application code. 

The Consular Consolidated Database, or CCD, is the central repository for all U.S. visa applications and integrates with other federal intelligence databases. It's one of the largest Oracle-based databases in the world, storing more than 75 million photos and 100 million visa files, according to The Register. During the outage, those files were inaccessible while IT restored the system. The downtime created a backlog of visa applications, causing headaches for those dependent upon a visa for school, work or travel. The State Department's Consular Affairs office says it has made significant progress and that the majority of nonimmigrant visas have been issued, yet applicants may still experience one-week delays on top of the normal processing time. 

Even now, 24 days later, officials are still working through the application backlog. Some Chinese students studying abroad in the U.S. likely won’t arrive in time for the start of their term. Irish citizens with flights booked for a vacation to the U.S. may not receive their temporary visas in time. American parents who just adopted a baby are unable to get home with their new child. And the list goes on. 

The good news is that the government didn’t lose the visa application data. The bad news is that recovery is slow because the system wasn't backed up. The situation, and the lengthy recovery process, is a reminder that systems fail and accidents happen. IT's job is to ensure the company has a data protection strategy in place to get back up and running quickly. 

New-generation data protection methods, such as image backup, speed up recovery times and give IT more control over company data. This is important given how expensive downtime can be. About 80 percent of IT professionals estimate that downtime costs their business at least $20,000 per hour, and more than 20 percent estimate that it costs at least $100,000, according to IDC research. 

In our data-driven world, companies are highly dependent upon their data. To protect against data disasters or costly downtimes, companies need reliable protection technologies that cover all of the bases, including application, file and system configuration data. The State Department learned that lesson the hard way, but it's experience reminds companies worldwide that recovery and backup must be a priority.