Scary DR Stories 2023 Recap
On Halloween, we hosted our 8th annual Scary DR Stories where we heard from real customers about real IT nightmares in their past. This year we had three spine-tingling tales that covered everything from hardware failures and human errors to ominous outages, monstrous migrations, and a blindsiding bioterrorism attack! After customers shared their stories, we delved into the depths of how Zerto, a Hewlett Packard Enterprise company, makes a difference for their organizations and why you should build a resilient, ransomware-proof solution with HPE and Zerto to keep your organizations safe from the most terrifying threats haunting us today.
Horrifying Hardware Failure and Human Error
Our first scary DR story winner was Proby Patel, storage team lead at Sensus, a Xylem brand. He shared an eerie tale of total data loss:
In our DFS environment, disaster struck twice. First, Server 1 met its demise in January. With Servers 2 and 3 still running, we pressed on. A replacement was ready but sat idle. Then, on April 30th, Server 2 faltered. User data, synced only between 1 and 2, was at risk. Backups from January were restored to 3, but the Robocopy job sought Server 1. Four months of data were lost. Recovery efforts followed, but much was corrupted. Pre-January tapes became our lifeline. Lessons learned: vigilance in adversity, question assumptions, prioritize precision over speed. Incidents demand collective planning. Beware false comforts; verify backups thoroughly. When burnout signals appear, intervene promptly. Trust your instincts and provide needed support.
Among today’s threats, hardware failures and human error account for a major portion of downtime events. In fact, 74% of breaches involve the human element and include social engineering errors or misuse—or misplacement of a company device—as the point of a cyber-attack.
Zerto’s continuous data protection (CDP) helps thwart these ransomware attacks with real-time detection and protection. The near-synchronous replication and granular journaling technology detect data encryption. This helps customers mitigate data loss and downtime, as they can now detect an anomaly within seconds, then protect and recover within just a few minutes.
Proby’s team at Sensus sees these types of results every day. Currently, they’re meeting various SLAs across multiple sites for clients and getting an average two second RPO across 95 VMs. Proby also shared that his team is “…required to do an actual DR exercise—not a test- twice a year, and that it usually took an hour and a half to two hours. Now, that takes only about 15 minutes, and everything just works!”
Ominous Outages and Monstrous Migrations
Real-time replication and the unique journaling technology are two key components of CDP, but let’s talk about how the third key component, and some other benefits of Zerto can help in scary events like a full-on outage of your data center or a massive migration. Khan Mohammed, system engineer at CVS Health shared his ominous experience from long ago:
In a windowless data center, the IT team embarked on a disaster recovery exercise for Linux servers, databases, and applications. However, a power outage and backup generator failure led to a communication breakdown. With flickering lights, they navigated the server room, encountering technical anomalies with strange images and cryptic symbols showing on the screens. Then they found a mysterious, unknown server rack which caused huge concern. A dense fog began to envelope them, and panic set in since they had no phones, and the radios went dead. Suddenly, lights returned, generators restarted, and communication was restored. It was a puzzling event, attributed to series of unfortunate technical glitches. This inexplicable experience still haunts that team today.
This outage story that Khan shared with us was spooky alone, but it was even more alarming to hear, was that it happened while his team was handling a large migration project that took two years to complete!
Zerto delivers multi-cloud mobility and application centric protection and recovery to easily move and recover apps as one cohesive entity, and to mitigate any disruption, unplanned or planned—like a migration. This past year, we helped American Airlines with a migration project. With their previous vendor, the project was estimated to take 9-12 months, but with Zerto, they completed it in about a month’s time with zero issues in the process.
Large enterprises, like American Airlines and CVS, are using Zerto to seamlessly protect 1000s of VMs to, from, and between the cloud. Zerto is making these big projects easier to tackle and helping to meet SLAs for an organization’s regulatory requirements. Khan shared his favorite part of Zerto was the non-disruptive testing that has reduced DR testing from hours down to minutes every quarter.
“Our disaster recovery testing is so easy now. It only takes us 12 minutes, and it works perfectly every time.” – Khan Mohammed, system engineer at CVS Health
Blindsiding Bioterrorism
Even scarier than these disastrous events in the virtual realm, is when disaster is lurking in the flesh. When it’s been looming in your office, and you have no recovery option. An anonymized customer shared a bioterrorist attack he experience long ago. He told us that letters that contained anthrax had been inside the building for some time before being discovered and getting a few employees sick. Fortunately, there were no fatalities and our secret survivor walked away with the scariest DR story of 2023.
There was a time in the 90s that anthrax seemed to always be on the news. Imagine the FBI coming through your office in hazmat suits and masks and taking everything—including your servers! That was this customer’s reality— “It was like we had a fire” because everything had to be shut down, and at that time, there wasn’t a DR site setup, so they had to rebuild from scratch.
Over the years, companies have become more prepared by at least having a DR site, but now it’s time to ensure you’re truly ready for these worst-case scenario events.
Cyber Attacks and Recovery with the Vault
Bioterrorist attacks, ransomware attacks, malware, oh my! These cyber threats are those worst-case scenario events, and they are getting more targeted and more strategic year over year. These bad actors even offer ransomware as a service now! IDC reported that 61% of DR events were triggered by a ransomware attack or malware last year, and this is now a board-level challenge. Zerto’s Cyber Resilience Vault is the solution. It is a clean room with zero trust architecture that is completely air-gapped, isolated, and provides immutable data copies. According to Gartner, “immutable data vaults (IDVs) provide the highest level of security and recovery against insider threats, ransomware and other forms of hacking.”
The Cyber Resilience Vault is purpose-built for cyber recovery and for our customers who have been asking for this last resort of recovery option. As demands for standardized compliance and regulations grow, an immutable vault will meet that market need.
Building Resilience with HPE and Zerto
Zerto built the vault with other industry leading solutions from HPE such as ProLiant, Alletra and Aruba networking. HPE’s acquisition of Zerto has led to powerful data protection creations like the Cyber Resilience Vault, as well as a new SaaS-based DR solution—HPE GreenLake for Disaster Recovery, which is built using Zerto’s CDP technology and is offered as a subscription service on a cloud platform.
Now, just add HPE GreenLake for Backup and Recovery to complete your data protection portfolio, which is a SaaS-based, long-term retention solution. From a single SaaS console, you can seamlessly manage these DR and backup solutions with global, multi-site management for hybrid cloud.
HPE and Zerto want to help you build data resilience for all workloads, all tiers, and all SLAs—from edge to cloud. We are here to help you prevent your own scary DR story from happening in the future.
If you missed the webinar, get the replay here: Scary DR Stories 2023 Webinar Replay
Kyleigh FItzgerald 
