The whole SimpliVity team has been eagerly watching HBO’s Silicon Valley, since our OmniCube solution influenced the Pied Piper Box that will be featured on Sunday’s episode. As I was binge watching to catch up with the team, one episode that really caught my attention is the one where Richard Hendriks has the night sweats from the stress of trying to run Pied Piper.
The idea of loosing sleep due to being worried about the business is something that many IT people can relate with. While it pretty much never comes down to the entire company being deleted with one line of bad code (that story was a hoax), every IT professional who’s been around for more than a year or two probably has a IT nightmare story to tell.
My personal story came early in my career when I was still a web developer. I was working on a commerce website testing new code in our development environment and wanted to test the code on a clean database. Having done this dozens of times, I logged into the database, and cleared all rows out of the main table. When I refreshed the webpage I was working on, it still showed the old data. Then it hit me. The last time I had logged into the database, it was to troubleshoot a Production problem. Sure enough, I had wiped out the Production database.
I then performed the walk of shame, performing short little prayers that we had good backups of the database. Lucky for me, the database administrators had a rigorous backup policy in place and were able to restore the Production database with only a few hours of lost data, less than an hour of downtime, and very little teasing.
Knowing — or at least hoping — I wasn’t alone in having such a horrible IT day, we asked a few of our customers and internal solution architects for other IT nightmare stories: what catastrophic incidents do they fear, which have they actually experienced, what did they do to recover from it, and what they have in place to prevent it. And as they used to say on Dragnet, I’ve held back the names to protect the innocent.
One customer had a story that was very similar to mine, only with slightly more impact. After accidentally overwriting a production database on their proprietary AIX server in the middle of the day, they found that restoring the database was not so easy. Since it was one of their core systems, employees had to revert to keeping paper records during the day, and spent the evening keying the data back in after the database was finally restored and business had ended for the day. This resulted in a higher than normal error rate and a significant pizza bill.
While these nightmares will inevitably still happen — this is a judgment free zone — SimpliVity can help make the recovery much easier. Whether that’s restoring individual files, entire databases, full virtual machines, or a complete data center recovery from a disaster, SimpliVity has the built-in data protection that will help an IT professional sleep better at night.
Another customer shared a story where they accidentally deleted a critical virtual machine, but this one had a much happier ending because SimpliVity hyperconverged infrastructure was deployed. Not having any local backups configured, the administrator feared that the entire company was in jeopardy. He then realized that SimpliVity was taking remote backups of the virtual machine. Now his fear was slightly relieved as he sat down to wait for the remote restore to complete. Three minutes later, the restore process was complete and the VM was powering back on.
While SimpliVity’s technology saved this story from escalating to a complete nightmare, disaster recovery is still a nightmare I have, even though I don’t directly support a production data center. This nightmare came to life for one of our customers. They had a replication setup for their legacy stack environment that never fully worked. This led to a disaster recovery situation that was extremely complex due to incomplete replication. The ease of restoring an entire data center was one of the driving factors when they later chose SimpliVity for their core data center and disaster recovery infrastructure.
Whether it’s a BitLocker failure, a remote data center with really old equipment, a server that doesn’t come back from a hard reboot, or a simple bottle of Tres Comas accidentally sitting on the delete key — only three of which actually happened in real life — every IT infrastructure has its risks, so make sure you have the proper data protection strategy in place, and an infrastructure that will make it easy to implement.