"Having a comprehensive plan that's bigger than just IT is key, but often IT can be the forcing function to get you started."

I recently had a chance to interview Jamison West of Arterian. Jamison, who founded the company that is now Arterian in 1995, envisions a future where every small to mid-sized company will have an IT partner become a vital part of its core operations team keeping them free from disaster and flourishing.

SoftwareDisastersBlog: How do you help your customers prevent and prepare for IT disasters?

Jamison West: We see with our customers that reliance on connectivity is higher than it’s ever been for businesses to execute and support their customers. People now expect email to work like instant messaging, sent and received as fast as they type it.  We try to prevent IT issues  by adding redundancy to make sure that if there are problems — natural disasters or bad weather like we had recently in Seattle  — our customers are still up and running at least for critical operations.

Another key area for us is setting up backup solutions.  This is how often we take copies of files off your server, how long do we retain them, how long do we want to be able to take a file and restore it.  Backup solutions are a simple and effective way to improve resiliency.  For instance, we had one customer whose primary server went down and we were able to bring the snapshot and spin it back up as a virtual machine in about 45 minutes. This was a very inexpensive solution for the level of continuity they achieved.

SDB: Companies are becoming more reliant on IT systems being up and running.  How does this affect how IT departments are run?

JW: My experience is working with companies whose IT departments aren’t keeping pace with technology.  This creates a great opportunity for us to come in and help provide solutions that are less expensive and more agile than what they experienced before.  Agility in adapting to new technology trends and how people actually use IT systems to achieve business results is the key to how a successful IT team is run – whether that’s in-house or outsourced.

Business continuity handles situations where hardware fails, but we need to workaround these problems so you can continue the business. For example, we’ve set up systems where we take virtual servers and run them so we can get things running immediately. Being able to temporarily virtualize on local devices or in the cloud means you can solve todays problems quickly.

Because we’re looking at the risks and thinking about uptime, we’re able to raise issues about factors outside of IT.  You might need to look at alternative office space or facilities, or your telephones might go down.  The lines are continuing to blur between business infrastructure & IT.

Having a comprehensive plan that’s bigger than just IT is key, but often IT can be the forcing function to get you started.

SDB: What’s the hardest part of improving resiliency and continuity of businesses?

JW: I’ll start by saying what’s become easier is that some vendors have gotten good at making the technology simpler to set up and maintain.

What’s gotten harder is justifying the budgetary requirements and establishing their risk tolerance.  A lot of resilience is like insurance; if nothing went wrong they might not see the value of it.  But when things do go wrong it becomes very vivid and the value is clear. Many of our customers get why they need backups, What most people aren’t willing to spend on is business continuity, because they aren’t sure of their risks and the value of mitigating those risks upfront.  Until it’s too late.

Operationalizing resiliency is tough, and doing the planning is very challenging.  Many people hope for the best, and cross their fingers.  There are lots of disparate systems across our clients, and there are lots of different technology.  If certain servers go down, a small business might be OK for a day, but don’t take down their phones for 5 minutes!

We are starting to see our medium-sized customers starting to invest more in building resiliency for the back-office operations. Depending on the industry, if a facility goes down, you have serious challenges and the whole business has to stop. In those situations, they might not even have staff available to come into work.  But they want to make sure that their admin staff can do the core day-to-day operations — answer phones, send email, bill customers – even if they can’t provide their product or service.

SDB: Can you talk about a recent example of a disaster or major IT problems you encountered, and how you recovered from it?

We had a huge mess with Seattle Snowmageddon 2012!  Every client we had with redundant internet connectivity were able to have remote workers during the disaster period. Those that had a single connection were for the most part not able to function through the storm.  For my company, roughly two-thirds of our staff had power at home, and some went to Starbucks to use the internet so we were up and running.  And we tell our clients that if they were set up the same way, they could be too.

Another  recent example of a small IT disaster is this morning we had a client-server crash and burn. We had recommended them to use a HW RAID 5. They chose to save a buck and bought a cheaper server that used software based raid that is much less reliable.  Now they can’t run their business today.  We have to educate our customers to help them make the right decisions about where to make things redundant and invest.

SDB: What are the most prevalent risks that you see in maintaining continuous uptime?

JW: The number one thing we see is server hardware failure. Servers are made of millions of tiny pieces that run hot it’s just a fact that they fail.   If you’re willing to spend the money, you can get redundancy, but it adds up fast.  When designing redundancy, we look at mean time to failure, and the question is how far back can you afford to lose data, and how long can you wait until it’s back up and running.  If you can afford to go a week without losing the data, the solution can be cheap.  But if you can’t live with that, you need to buy a better solution.

Cloud helps you get rid of servers, so you can change the dynamics of your infrastructure risks. We’re not afraid of the cloud because anything that adds value to our customers and helps us become a more trusted advisor we see as progress.  We see our opportunity to be the ‘last mile’ for the larger service providers, and people will always need help demystifying and operationalizing systems for their businesses, and help making purchasing decisions.

For example, every small business used to take tapes offsite. Now with cheaper storage and bandwidth in the cloud, we can reduce the human risks of tapes getting dropped or broken. This is a huge example of how backup and recovery has changed with cloud.

What we see coming is the ability to really virtualize an entire IT system in the cloud.  I can use IaaS environments today and not have to worry about it (except to the extent that I trust the service provider).

Data is more complex than its ever been, systems are spread all over the place, you have multiple cloud providers, so the complexity is where we can help our customers.  Some people think that if they’re in the cloud they don’t have to think about it anymore.

Photo from csmonitor.com

On Friday January 13th, the Costa Concordia had a disaster – running into rocks off the shores off Italy’s western coast and eventually rolling onto its side in the water.   The toll on human life is tragic – several are dead (the number still growing), more missing, and everyone involved went through a traumatic experience.  

The saddest part of the story is that it appears it could have been prevented.  And if not prevented, could have been handled better.

As I’ve been following the story and reflecting on it, a few things have jumped out that I think we can learn from. 

Human Error

I don’t know the details of the timeline of events that caused the accident, but I’m guessing there were a few design problems at play.  For example, I’d people were ignoring early warning systems because they seemed like false alarms.  It’s not whether the alarm goes off, but how seriously you take it that matters. 

Also, I’d wager that the “designed” social structure on the ship prevented junior staff from questioning senior staff.  This was evidenced in a video I saw where the crew would not release the lifeboats until the captain ordered abandon ship.  And unfortunately it was too late for a smooth exit at that point.

Emergency Readiness

Another issue that keeps coming up is the readiness of the crew to deal with an emergency.  This video shows the descent into chaos.  It definitely raises questions about what is the right way to train a team to deal with bad situations.  How often should this be done?  How would the training or simulation be performed?  Is it really worth it?

There are two levels that I think about with emergency training.  The first is “What information will be needed and what procedures will need to be performed?” This is relatively domain specific, and needs to be thought through carefully.  What knowledge must be kept in the head vs. being readily available at the time of crisis?  And how can you reduce and simplify all these procedures down to the absolute minimum to ensure success? 

The second level of preparedness is more general – “What mindset should an emergency responder be in and how will they perform under pressure?”  If you know all the procedures but choke under pressure, you will not be successful in handling a crisis.  So any training program you have must include a “field” portion where people are asked to experience something that feels like a real crisis – confusing circumstances, time pressure, and dire consequences for non-performance. 

If you can combine the two readiness techniques into a safe, comprehensive, repeatable training program, you can be ready for a disaster.

Prevention

One assumption of how the ship operates is that the captain is in charge and knows what he’s doing. In this case, that assumption turned out to be wrong. How could the design of the system have been improved to avoid it?

Why did the ship go off course in the first place? Where there any systems on board (computerized or human) to notice and raise the issue? We don’t know yet, but hopefully will learn when the full review is completed.

They may have had computerized mapping systems could watching the route and complaining when things changed.  Sensors below the ship could also have been watching for rocks and debris and alerted the crew when something was approaching. And the crew might have been aware that something was wrong. 

Ultimately whatever detection systems they had either didn’t work or were ignored.  Sometimes too many errors makes you numb.

Conclusion

We’ve been looking at this in the context of emergency procedures on a cruise ship, but the exact same principles apply to your software systems.  If you can think about human error as flawed design, improve your signal the noise ratio, and make errors OK, then you can improve the resiliency of your systems.

How has this tragedy made you think differently about disaster preparedness and recovery? How ready are you?

Photo by Creativity103

You might think that new year’s resolutions are made to be broken.  Whether it’s to exercise more, chew fingernails less, or other clichés, they are hard to follow through on.  Witness the packed gym in January that becomes empty before March.

When it comes to keeping the systems that run your business humming along, the new year is a good time to pause and reflect on what you can do differently. And  you also have the energy to take action to make it a reality. But don’t let your well-intentioned resolution become lost in the shuffle of forgotten promises of self improvement.

1. “I will learn from last year”

Think back to last year and all of the downtime-related issues that happened.  Maybe you have these carefully catalogued, or maybe you just have juts been trying to block out the memories.  Consider what you can learn from these experiences.  What could you have done to prevent it all together?  What could have been done to be more prepared?  Would it have been worth it, or was it actually better to let the downtime happen because prevention would have been-cost prohibitive?

2. “I will make sure that my mission critical system hits 99.95% uptime”

You know which system matters most.  Maybe it’s email.  Or point of sale.  Or perhaps your website.  In any case, focus your energy on making that one system resilient and reliable.  Don’t try to improve everything, you’ll just waste time and resources on systems that don’t matter.  Set an uptime goal, and track how well you’re doing.

3. “I will keep my crisis contacts up-to-date, and always have someone on-call”

A simple and effective way to handle the unexpected crisis with your systems or customers is to always have someone who will drop everything to handle it.  There are many ways to do this and it’s easy to get it started.  But make sure that everyone knows what’s expected, who to contact, and what to do.

4. “I will improve the signal-to-noise ratio, and I will listen to the signal.”

I read a quote from a reformed burglar that said something to the effect that “If I had a $100 for every time I was right outside a house I was about to rob and the owner told their dog to quiet down, I’d be a millionaire!” 

If you’ve got systems to alert you that somethings going wrong, the biggest problem is they are always telling you something is wrong.  Resolve to improve the number of signal needles in the noise haystack.  A good strategy to do this is to treat every alarm as real.  You’ll have no choice but to look to how you can improve the quality of your alerts because otherwise you’ll spend all your time responding to false-alarms

5. “I will have a plan, and I will test it.”

If you don’t have a plan for your software or business continuity emergencies, then resolve to build one.  But don’t leave it on the shelf, make sure your team knows where to find it, what’s in it, and why it matters. 

Think about how you can try out the various disaster response instructions in a controlled environment, and take the time to actually do it.  Your team may not want to pretend they are in crisis mode, but without practice you won’t know what to do when there’s additional stresses and time pressures.

Conclusion

As we begin 2012, we are more dependant on software than ever before.  We owe it to ourselves to be become resilient to errors & mistakes, and to be prepared for problems outside our control.  This is the key to keeping business running and customers happy.

What were some of the lessons you learned in 2011?  What will you do differently this year?  What other ideas do you have for resolutions and how to keep them?

Photo by Todd Huffman

“It’s not rocket science.” – Some Expert

“One test is worth a thousand expert opinions” – Wernher von Braun (Rocket Scientist)

I am a firm believer that reducing software & IT risks is not easy, but it is imperative.  It takes a strong will, the right attitude, and acceptance of responsibility for your web service or application.  You have to really understand the realities of how software works (or doesn’t) and the impact a disaster would have on your business & livelihood.  But accepting may be the hardest part.

It’s sort of like the 5 Stages of Grief.  Behold, I present to you my 5 Stages of Disaster Preparedness.

Denial

“I don’t need to prepare for an IT disaster. My site is running fine now, customers are happy.  I have bigger things to worry about. This isn’t even on my radar.”

If this sounds like you, your IT systems may be at risk of sustaining downtime that could impact your customers and your business.  You may need to start with the basics and make sure you take the first step and admit you have a problem.

Anger

“I thought modern software was supposed to just work! Didn’t they tell me cloud would solve all my problems?  Why should I have to waste money on preparation?”

It’s easy to get upset about the fragility of networks, IT infrastructure, backup technologies, etc.  There are lots of new products & technologies out there that will help you reduce many of the resiliency and recoverability risks in your software.  But the 2nd law of thermodynamics applies to your software as much as to the rest of the universe, and eventually your software and infrastructure will decay and expose you to risks of failure.

It’s nothing to get angry about.

Bargaining

“Maybe I can get away with putting this off a little longer.  If I just add some backups that should be enough.  My team knows what to do if there’s an emergency, so I think we’re sort of prepared.”

It’s true that you can and should prioritize your mission-critical systems for resiliency and disaster preparation, and make a calculated decision on where to invest in uptime improvement.  But be careful what you choose to ignore as you focus on top business priorities.  Your online reputation is more important than ever (regardless of your klout score), and your customers will notice if your website or application goes down.  And your employees will be much less productive if they are unable to use email and business software.

Depression

“There’s nothing I can do to make any of this better.  Disasters are going to happen, that’s life.  I can’t stop them.”

You can’t stop disasters, but you can lessen their impact on you.  You can learn how to stop preventable disasters, and you can learn to cope with the inevitability of software failure.  I can’t think of a tough downtime or failure situation I’ve been in that couldn’t have been improved by better planning, testing, and execution.

Acceptance

“If we’re smart, we can reduce our risks and improve our uptime.  There have to be some simple changes to make things better.”

“Disasters may come, but we’ll be ready.”

Identify your biggest risks, create a plan to address them, implement solutions, and (most importantly) test them.  Prevention is not buying expensive tools that promise silver bullet solutions, it’s good engineering, self-critical analysis, and expecting the worst.

Preparation is not a big fancy document that sits on a shelf, it’s having a team that’s been through it before.

Conclusion

Don’t wait any longer, come to grips with how you’re going to prepare for downtime & disasters.  Don’t get stuck in denial or angry at the problem.  Take the first steps to reduce your risks and keep your customers happy.

What’s stopping you from investing in preparation?  Is it something you know you “should” do but you’ve found a good reason to put it off?  What helped you make the decision to invest in preparation?

Photo by dannynorodo

The biggest holiday risk on your mind might be electrocution by lawn decoration or having to suffer through a eggnog induced story from your brother-in-law. 

However, there are a few risks to consider for your online business or website that are holiday specific.  The best present is peace of mind as you’re able to confidently ignore your business for much deserved downtime.

1. “Low-Risk” Updates & Changes

The last few weeks of the year area great time for system admins and developers to get caught up on work because there usually aren’t a lot of people around the office. But changes to your environment create risks that under normal circumstances would be small, but during the holidays can be more severe. because you may not have a full staff or traffic to your site, a problem in your site might go unnoticed until after new years. And you may not be able to quickly handle a problem.

The best bet is to have a defined lock-down period for changes that pose downtime or recovery risks during the holidays. Critical and low risk fixes get through, and everything else gets shelved.

2. Who’s got the ball?

We all have shopping to do, events to attend, family to see, and carols to sing. Giving your employees flexibility to do all this fun stuff during the holidays is a great perk and makes work a lot more enjoyable. But if people are offsite frequently or pre-occupied with other (arguably more important) things, they may not be at the ready to notice and respond to an issue with your online business or website.

My recommendation is to have a published rotation of responsiblity for any production site issues that occur. You can even give extra incentives for folks that take the calculated risk of being on rotation for Christmas day or other holidays. They might get a treat for free, or they might have to do some critical work. At any rate, they shouldn’t have to do it two years in a row.

3. Seasonal Peaks & Valleys

Depending on your business, you may experience dramatic changes during the holiday period.  Obviously, if you’re on online retailer, you need to be ready for shoppers from Black Friday on.

On the other hand, if you sell to other businesses or provide services that get less use during the holidays, you may actually get a dramatic reduction in usage of your system.  The biggest issue this creates is unused capacity in your data center, leaving you at risk of paying for computing power you simply don’t need.

One approach is to track your demand over multiple years and pay attention to seasonal patterns.  Make sure you have a plan in place to grow or shrink your capacity based on expected demand, plus a margin of headroom for unexpected spikes.

Conclusion

The holidays are a special time for businesses too.  There are unique risks that are posed and should be mitigated to ensure smooth continuity during the holiday season.  

How are you planning to keep your business humming during the holidays?  Are you experiencing a spike or drop in demand?

Scott Hamlin - "If your data center burns down, it’s easy to walk in and determine exactly what the problem is and create a plan for recovery."

Scott Hamlin is a local business owner and IT professional that has been serving IT customers since the early 90′s. I recently had the opportunity to ask him a few questions about PacketDrivers and his views on improving software resiliency.

The full interview follows:

Can you give some background about yourself and your company?

I started PacketDrivers in 1997 with my business partner Bob Benshoof. We originally created the company with the intent of providing “network” services to small and medium size business. At the time it was very technically focused hence the name “PacketDrivers” because we drove the packets on your network.

As our company matured we began to see our role as a trusted advisor to our client base and realized in truth we were a company that was uniquely positioned to not only give “business” advice as it related to technology, but that we could be the IT Department for those same organizations. We evolved to where we are today primarily playing the role as the IT department in a fixed-fee IT outsourcing role.

What’s your job really like? How do you spend your time?

We are still a relatively small organization of 13 people. Although I spend time thinking strategically about our business I still serve a CIO type of role with many of our clients. In our monthly or quarterly business reviews with clients we discuss the primary opportunities and challenges our clients have not only directly with IT issues, but overall business issues. This allows me to provide direct input on how IT solutions may be able to help with these challenges, enhance the opportunities and in some cases provide input that IT is not the best or most cost-effective solution.

What advice do you have for IT managers or business owners that want to improve availability and resiliency?

The most important step is to locate the systems and applications that the entire organization relies on to produce their product or service and what the costs of downtime are. Although most organizations continue to increase their reliance on IT systems there are still plenty that produce much of their actual product or service without direct reliance on IT systems.

It’s easy to falsely assume that the biggest systems and applications are the ones that will cost the most in both real dollars and opportunity costs if they fail, but that is not always the case. Assess first the business impact and then you can focus your preparedness to handle outages and disasters.

Another common mistake is to plan for what we think of as a traditional disaster when in fact most outage and down time is related to issues around logical problems with upgrades to operating systems, applications etc. If your data center burns down, it’s easy to walk in and determine exactly what the problem is and create a plan for recovery.

It is much more difficult when the users in the organization start reporting that the data that is coming from remote retail stores into the central system is causing the distribution management to suddenly make no sense and the organization cannot ship product. Where is the real problem? The scanners at the retail stores, data coming from registers, an export/ import process from the local store databases, a database corruption etc.

Can you give an example of being prepared?

An excellent example of properly identifying the systems that represent the greatest risk exposure can be seen in a manufacturing company that we have worked with. The organization has a manufacturing system to manage much of their operations. They use virtualization with failover options, system backups, redundancy in many places, UPS’s etc. and have spent some time thinking about those issues. They also have a single PC’s out in the manufacturing plant (a location that is hot, dirty etc.) that is used to do calculations for melted steel.

It turns out that if you start the process and suddenly cannot utilize the PC to control the process there are 10’s of thousands of real dollars (for electricity, material etc.) that are completely wasted in the process. Conversely most of the manufacturing process can continue even if the entire server room were to burn down. In this case there was an actual failure of the PC during the process (in the middle of the night) and the process had to be aborted costing the organization plenty.

The solution was to not only have a spare completely configured with the all of the necessary software but to also train the staff to swap the machine if necessary. A quarterly test and swap of the workstation along with the necessary training was also implemented. This naturally led to an evaluation of other single points of system failure in the plant and recovery plans for each.

What’s the toughest part of making software systems resilient to failure?

Software is much more difficult to manage in terms of resiliency and failure than hardware. If a hard drive fails we know exactly what happened and what we need to do. Applications on the other hand can be considered more of an art than a science. When a hard drive fails 99.9% of the time you put a new hard drive in the system and the hard drive is complete functional. Restoring the system may be more complex, particularly if you have to move to a new server.

Drivers may be incorrect, how the new server connects to the infrastructure for input from other systems and other factors make that portion of the recovery much more difficult. Applications today are often integrated with many other applications to exchange data. What if some of the data is corrupt? How do you identify the actual problem? With the input of data into an application from other applications how do you ensure that other components of the application can function without that input?

What have you learned in the last year that’s made your customers’ systems more reliable?

When we are strictly talking about applications themselves the most important factor is great documentation on all of the systems but most importantly documenting how they are integrated. We have clients that have one important line of business application that essentially stands on it’s own.

On the other hand we have clients that may have 6 separate applications and each relies on data from some or all of the other applications some of which may be hosted at the client data center and some are Software-as-a-Service (SaaS) such as SalesForce.com. In this case we have found that it is essential to have diagrams of the critical components for data flow and identifies the point of failure that will not only effect a single application but has a cascading effect. This is a great tool for beginning the business impact analysis and eventually planning the recovery process.

If I am concerned about the resilience of my software systems, what should I do first?
Take the time to begin the process of a business impact analysis. There are plenty of resources online to at least provide the initial framework for a plan. That being said, historically the industry has had a focus on hard systems and major disaster planning. Failures that do not represent a total failure or actual physical disaster are much more common.

Do an analysis of the systems with an eye for more subtle failures such as data corruption, internal information disclosure, hackers, etc. and then test how well prepared the organization is for these scenario’s – they are more likely to happen than a regional disaster such as an earthquake (an event that likely involves your customers who will also be preoccupied with their own issues anyway.) If you are not comfortable doing this yourself, consider looking for outside expertise to provide assistance.

Photo by austinevan

“Plans are nothing; planning is everything” – Dwight D. Eisenhower

I was chatting with a couple of disaster recovery/business continuity folks yesterday over happy hour, and I listened as they talked about some of the challenges with disaster recovery planning.

One of the comments that stood out in my mind is that biggest challenge with disaster recovery plans is getting people to test them.  Companies will pay a bunch of money for disaster recovery plan, and then let it sit on the shelf and collect dust.

I remember a team I worked on that was trying to build out a disaster recovery solution – basically just active/passive failover.  Someone proposed that we switch which data center was active every few months.  Everyone groaned at the idea.

Testing a disaster recovery plan might feel like a big waste of time because there are so many factors that won’t match the real disaster. From my experience testing, I think one of the most important factors in testing is designing the right test.  You can spend a lot of energy “testing” something that really doesn’t matter. 

So what are you testing for?  Are you trying to see what happens when your servers fail?  Or maybe that your alerting system works properly?  Or that your team is trained to handle an external event, like an earthquake or fire?  Do your backups restore without corruption? Depending on what you’re looking for, you need to design a test that exercises the process, and measures the important outcomes.

Just having a disaster plan is useless.  Planning can help you think through your risks and what to prepare for.  But testing the plan and putting your system and teams through the paces will help you improve resiliency and your teams capability to handle disasters.

Photo by ˙Cаvin 〄

“The dose makes the poison.” – Paracelsus

I had coffee with a friend who runs a small IT service company on Friday, and we were talking a bit about preparing for disasters. He’s been doing this a long time and seen a lot of failed software systems.  One of the things he told me is that in his experience, IT systems are brought down from a series of small “disasters” creates more frequent & annoying dowtime than the big disasters. Yet many IT pros he works with spend their disaster recovery energy preventing and preparing for “The Big One.”

Why do we have a tendency to think of the big disasters that occur infrequently and neglect the smaller everyday problems?

I think it’s easy to look at everyday failure as annoyances that aren’t that bad.  We can work around them, and it’s not worth preventing them from happening again.  If stopped at looked at them in aggregate, you might see the ROI in preventing them through improved uptime.

If you have a frequent number of small outtages and downtime, you’re probably at risk of upsetting more customers & employees than during a large catastrophe.  If there’s a major flood, hurricane, or fire, your customers are likely going to be concerned about their own safety & property.  And they are more likely to excuse a hiccup in your website or service (if they even notice!)

I’m not suggesting you shouldn’t take the necessary precautions for a major disaster event- backup, redudancy, etc.   What I am saying is that reducing the risk of small frequent failures in your system may actually yield better results than just focusing on large scale disasters.

And the best approach would be to invest in IT and application redudancy that will help with both situations if possible.

Where do you spend your disaster recovery money?  Planning for the Big One? Or making your system more resilient to smaller failures?  Or both equally?

Playstation downtime.

If your software goes down, I hope it’s for accountants.

Insider data breaches soaring.

Half of UK laptops vulnerable?

Data loss risks – perception vs. reality.