I started my morning reading yet another article about a business in serious trouble because their backups were no good. In this case it had to do with GitLab, but the particular company doesn’t really matter. What matters is at least once a week I read a similar article where some failure or security breach can’t be adequately dealt with due to a lack of good backups and a poor or non-existent disaster recovery plan.
I find this incredibly frustrating. It is 2017, not 1999. There is simply no reason for this anymore. The state of the art for backups and disaster recovery – even at the personal workstation level – has advanced so far there is no excuse for not having verified, reliable backups. Being caught in a situation where you can’t recover is no longer something you can blame on difficult to use software and unreliable tapes and hard disks. It’s simply gross negligence.
Insurance companies see it this way too. A few years ago I was brought in to help a company recover from a hardware failure, only to find out they had absolutely no reliable backups. The cost for a data recovery firm to try and recover data from the crashed disk drive was more than the business owner could afford. His business insurance provider denied his claim – completely due to having no reliable and tested backups. The resulting data loss put the company out of business.
In my experience, there are two reasons companies/organizations find themselves in this situation:
- Their IT staff or service provider doesn’t take it seriously. Here’s a dirty little secret from the IT industry: most engineers don’t have any idea how to ensure a backup system is configured and running the way it should be. Worse, they are entirely uninterested in the subject. They want to believe they have a system they can “set and forget” because dealing with backups is boring. It’s drudge-work. So, they turn something on and then just assume it’s working, with no follow-up and no checks.
- Senior leadership doesn’t make it a priority. In many cases this is due to a lack of understanding, and deliberate misinformation the receive from their IT staff or service provider. Business leaders aren’t (usually) specialists in backups and disaster recovery, and it isn’t appropriate to expect them to dig into the technical details. They may get bad counsel from the people they hired to take care of this aspect of their business. However, data protection and integrity is at least as important today as physical security and fire prevention is to a business. I’ve worked with businesses who would freak out over an unlocked door but who would never think to set up a regular testing schedule for their backups, and who won’t fund the creation of a disaster recovery plan.
So, what should you do? How can you, as a business leader or owner, be sure you won’t get caught out? Fortunately, backup and disaster recovery have come a long way in the last few years and it’s easier than ever to protect yourself.
- Understand backups and disaster recovery require your attention and oversight. Your data is valuable. Protecting it requires at least as much of your attention as physical security and fire prevention.
- Require your IT department or IT service provider to demonstrate and report on their ability to perform restores of files and entire servers/systems quarterly. Any exceptions or problems found during a test restore should be corrected and addressed with a follow-up report explaining what corrective actions were taken.
- Conduct a disaster recovery drill at least annually. Again, demonstrate that the systems most critical to your business are capable of being restored and functional within a realistic time period. Ideally, you’ll have a written plan and procedure for making this happen, with the plan being revised based on findings from your test. Annual tests are a bare minimum!
- Learn enough about your company/organization’s backup system to be positive regular, verifiable backups don’t require any human intervention. If a backup process requires human intervention it will fail. This means if you are still relying on some kind of tape backup system, you’re in jeopardy.
- Be sure your backup system in keeping at least one copy of backed up data offsite, and again, does not require human intervention to make this successful. When testing backups and disaster recovery, be sure to test these off-site backups, and don’t just rely on local copies. The local backups may be gone in the case of a disaster.
- Require your IT department of service provider to review the backups ever day to be sure they completed successfully. They should be able to provide evidence on demand demonstrating the backups are functioning correctly.
At Brightworks, we are deeply experienced with backup and disaster recovery. We take it very seriously and have developed procedures to ensure our customers can depend on their backups. Our BackupWorks service is designed to take the worry – and the humans! – out of the backup process to make it verifiable, repeatable, and easy to manage. If you’re concerned about your data and how safe it is, contact us for a backup assessment and to learn more about how we can help you protect it.