When Disaster Strikes, Will You Be Ready?

Broken Egg

Photo by stevendepolo

This is a guest post by Holly Plumley West (@hplumley), a disaster recovery & business continuity professional & triathlete.  Enjoy!

When Disaster Strikes, Will You Be Ready?

Disaster recovery (DR) sites are like insurance: most people make their monthly / annual payments and hope to rarely (if ever) use them. Knowing that a DR site exists provides comfort to the business, although it can be a false sense of comfort. If you have a DR site you should ask yourself a few questions to help avoid creating additional disasters for your company during an actual emergency.

Who Turned Out The Lights

Have you ever tested your DR site? After the initial test of the environment do you make sure to test it at a regular interval? The answer to both of those questions should be yes. One client implemented a DR site and for 18+ months could not make time to test the site although the initial test of the environment would have only required an hour (or less) of his time. As luck would have it a storm blew through his state and took out power before the initial DR site test took place.

Everyone crossed their fingers, hoped for the best, and the engineers activated the site. From a technical standpoint the site worked flawlessly, although this does not always translate to the user experience working flawlessly. Luckily the user requirements had not changed much from the time the site was implemented to the time it was activated. The functionality the client needed to add to the environment was quite simple and we were able to add it in the matter of hours. The client was lucky because the new functionality was a standalone application that did not require data to be replicated from production, so they avoided potential data loss.

After this situation the client has made an effort to test their DR site more regularly, which is a relief to everyone.

The Fine Print

Have you made sure all employees are installing applications and saving critical data to a location that is covered by your DR site? A new client told us that the list of applications included in their DR contract included all applications they used on a daily basis. They had reviewed the contract many times and discussed it on several calls with their Account Manager. As we worked on the implementation of their environment the client became angry when they realized their three most critical applications were not listed in the contract.

How did this happen? The client used a separate third party vendor for the three critical applications and both the client and vendor failed to tell us (their primary IT provider) about the applications.

How can this be avoided? Make sure all your vendors are in communication. It is not safe to “assume” that everyone is aware of the changes employees make to their computers unless monitoring solutions are in place on individual computers or access is restricted. Many companies still allow their users to be local admins on their computers and do not monitor everyone individually, which can lead to DR issues.

Sticky Notes Don’t Count

Do you have a business continuity plan (BCP) to supplement your DR site? Is it one you have written? If so, has it been reviewed by a professional? Another client that I worked with for a couple of years swore up and down that his firm had a BCP (he had written). Despite recommendations that it should be reviewed by a professional he declined. Guess what?

His building happened to have a power outage while he was on vacation. Did any of his employees know what to do? Or how to get the DR site activated? No. They were “dead in the water”because his employees did not know to read the post-it notes on his monitor for information on activating the DR site. He had not reviewed BC procedures with his employees so they were clueless how to react and respond during a disaster.

Even after this situation he did not learn from his experiences. It was not until about ~3 years later (after he had departed the company) that they are FINALLY having a professional BCP written (with training and maintenance for ongoing support). They are lucky that his lack of planning did not result in bankrupting the company during a simple power outage.

Conclusion

Like the fire drills and bus evacuations everyone had to do as a kid, it is always important to take DR and BCP seriously. Always read your documentation and maintain an open dialog with your employees, clients, and vendors. If you have unanswered questions or holes in your plans make sure to actively address them with your employees and vendors.

About Holly Plumley West

Originally from New Hampshire (Enfield), I moved to Boston in the fall of 2000 to attend Northeastern University. I graduated in September of 2004 with a Bachelor of Science in Business Administration with concentrations in Marketing and Management Information Systems. After graduating I worked in Marketing at Kallmann, McKinnell & Wood Architects, which was one of the companies through which I had done a co-op. In 2005 I was recruited by Eze Castle Integration (www.eci.com) and have been working there ever since. Since December of 2007 I have worked as a Project Manager in the Disaster Recovery Department. In my “spare” time I like running, triathlons, snowboarding, renovating my house in Boston, reading, volunteering, and spending time with friends and family.

Advertisements

About Kit Merker

Product Manager @ Google - working on Kubernetes / Google Container Engine.
This entry was posted in Business Continuity, Disaster Recovery, Downtime, Technology, Uptime and tagged , , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s