5 Tips for Avoiding IT Outages During a Disaster

We are deep into the throngs of hurricane season. Following the catastrophic events in Houston from Hurricane Harvey and the looming threat of the now category five Hurricane Irma approaching Florida from the Atlantic, thousands of people and businesses are either preparing for the worst, or recovering from absolute disaster.

For IT teams, keeping systems running and avoiding communications network outages is critical during these emergency weather situations. A recent model estimates that nearly a million people were left without power due to Hurricane Harvey. When disaster strikes, it’s very important that organizations stay proactive, and have contingency plans in place, and work to the best of their ability to stabilize the network and keep services running.

We already know that unplanned outages are costly to the business. Just look to Amazon, which lost approximately $66,240 per minute when they experienced downtime back in 2013. Network Computing estimates the cost of information and communication technology (ICT) downtime is roughly $700 billion (with a “B”) for North American companies alone when factoring in things like employee productivity, revenue lost, and maintenance costs.

So what can you do to avoid these costly IT outages when facing hurricanes or similarly big disasters? Plenty! Just as citizens are told to take preventative measures to minimize damage to their homes during a storm, IT organizations can follow suit to prevent network and system downtime.

Here are just a few line items that may help to prevent or quickly resolve IT outages during a disaster recovery situation.

1. Use Preventative Best Practices

One of the best things you can do to avoid outages is to do your due diligence and take every day preventative measures on your system and network infrastructure. Hardware is a big one; be sure to regularly schedule maintenance of systems and perform regular upkeep on old, or problematic equipment that may have a high probability of failure during a storm. You should also verify system redundancies, perform regular system health checkups, and make any upgrades, patches or fixes to known bugs before it’s too late. These minor items can add up if not resolved ahead of time, and cause unnecessary hiccups when facing a disaster recovery situation.

2. Establish ITIL Change Management Processes

ITIL change management best practices are critical for energy and utility companies that face stringent compliance audits mandated for the purpose of avoiding downtime in a hurricane or disaster scenario. However, you don’t need to be in a heavily regulated industry to maintain strong governance of IT change practices. In fact, implementing an ITIL change management solution can support disaster recovery efforts by helping to reduce human errors via the automation of critical system changes that may affect dependencies, or help teams react and make changes faster via CAB meetings, automatic change scheduling and pre-approvals.

On top of that, auditing changes as they happen is important for IT teams for historical purposes, meeting compliance requirements, and for the ability to retroactively rollback changes at a moment’s notice during a disaster recovery scenario.

3. Use Monitoring Tools to Maintain Network Visibility

Disaster situation or not, one of the best ways to reduce chances of downtime is by implementing a real-time network and application monitoring solution within your IT infrastructure. This way, IT gets a bird’s eye view over their organization’s most critical dependencies, so they can take proactive measures the moment they spot abnormalities coming the from their data centers, configuration item assets, network and hosting services, etc.

During a high-impact weather or emergency situation, monitoring real-time performance is very important if IT teams hope to take quick action regarding things like servers or data centers starting to show signs of trouble.

4. Have Disaster Recovery Plans in Place

One of the best things you can do is to create a plan of action during a natural disaster. Create a checklist, hold meetings with your team, hold mock scenario drills and do everything you can to be ready during a time of crisis.

Here just a few things you can do to prepare before disaster strikes:

1. Take stock of all recent, planned and upcoming changes before the emergency event hits your affected area. An IT Change Management solution will help here.
2. Update, review, practice and be ready to act on all safety procedures set by your organization in case of a natural disaster
3. Secure any and all back-up media, hard drives, servers, etc., to reduce data loss and speed up the recovery process. You may want to take any backup copies off-site as well as a necessary precaution
4. Print out and save hard copies of critical configuration assets in order to save time if you need to reference them later
5. Check and review your emergency and/or backup power setup to safeguard against potentially catastrophic damage caused from power surges
6. Review and assess the physical location of where configuration items are housed and decide if you need to take any precautionary steps such as re-locating switch/applications to minimize damage caused from exposure

5. Practice Smart Communication

Like any bad situation brought on by a natural disaster, people must work together in order to overcome big challenges during a crisis. Your IT organization is no different. Your team must act fast and quickly communicate with each other in order to identify and resolve issues either as they happen or before they get worse. Having a clear, defined communication strategy ready for natural disasters is a big first step you can take to making sure everyone knows what to do, and who to talk to when it counts.

Be sure to also check and test your means for direct-line communication. That means utilizing cell phones, landlines, backup network services…. even walkie-talkies. You never know what will happen, so preparedness is critical.

That covers some of the high-level steps and precautions IT teams can take to avoid outages in a natural disaster. Have anything to add? Drop us a line in the comment section below!