Those Who Work in the Networking/IT Field - Advice

[IBleedMusick]

For those of you who currently work in this field, how do you typically deal with worst case scenario's when they actually come true? Hurricane Sandy managed to bring my company to it's knees in a matter of minutes and as prepared and backed up as we are we still weren't prepared for it to cause this much chaos. With most of my servers down my clients have no access to Eclipse or Concordance (Platforms I utilize on a daily basis) and with no email I spent a better part of my night answering a barrage of phone calls from pissed off people well in to the morning mostly why are you not responding to any of my emails. While most of our younger more computer savvy clients were understanding of the fact that my hands are tied our older larger clientele was more like "I don't give a shit why can't I access anything?". I am intertwined into a network in New York City everything I am capable of doing in any of my Florida sites is based on how that office is running (from a networking standpoint). If that office goes completely off line I have no link to them meaning I cannot access anything whatsoever. With the severe weather I couldn't even have my backups sent to me via FedEx so that leaves me to wonder how anyone else in this field deals with these kinds of issues. I am asking for the guys who work for larger companies whose infrastructure is based out of key offices in certain cities that control the Network for every other office. I mean who knew so many cases were going active the week Hurricane Sandy decided to hit .

[JamieHoward]

Well at least in my case, our servers are located at a datacenter that is prepared for stuff like this with several bus sized generators that can power everything at the datacenter for days at a time if need be. The last hurricane that came through put power out for everyone basically in a 50 mile radius, but the clients that did not lose power at their respective places of business never got disconnected from our network. I would say that if you are in a lucrative industry with several clients that need continuous, uninterrupted access, you may want to think about biting the bullet and looking into a datacenter that is designed to never go offline.

[Tarball]

You have to have a working and practiced disaster recovery plan:

• Redunant systems
  
• Backup Power
  
• Redunant Redundancies
  
• Off-Site incremental backups
  
• Simulated outages for proof of concept
  
• Backup Plan is all else fails
  
• and most importantly $$$$
  

These are obviously high level concepts but i have no idea what your infrastructure is like nor do I expect a detailed statement on a public forum.

[Ninja_v1.0]

Your infrastructure needs to always include servers that can be powered from off-site. Sometimes it is not in budget but most critical designs have this options. In my case if the main building where to catch on fire I would be able to cold boot our virtual infrastructure from a off-site location and still be able to provide access to our infrastructure. Sometimes there is nothing you can do and for those that do not have the intellectual ability to comprehend what you are trying to explain to them I wouldnt waste my time. Focus on your plan of restoration and what would you do if you had some server that will not come up. You should ALWAYS have a disaster plan in the event of something like this. Execute it. If you can put off talking to customer I would while you are focused on bringing up your network. Get your backups ready, get your plan in place. If you need any help just let me know

[Tarball]

Well at least in my case, our servers are located at a datacenter that is prepared for stuff like this with several bus sized generators that can power everything at the datacenter for days at a time if need be. The last hurricane that came through put power out for everyone basically in a 50 mile radius, but the clients that did not lose power at their respective places of business never got disconnected from our network. I would say that if you are in a lucrative industry with several clients that need continuous, uninterrupted access, you may want to think about biting the bullet and looking into a datacenter that is designed to never go offline.

^^^

And dont forget to charge the customer uninterrpted service is not easy nor cheap.

[IBleedMusick]

Well at least in my case, our servers are located at a datacenter that is prepared for stuff like this with several bus sized generators that can power everything at the datacenter for days at a time if need be. The last hurricane that came through put power out for everyone basically in a 50 mile radius, but the clients that did not lose power at their respective places of business never got disconnected from our network. I would say that if you are in a lucrative industry with several clients that need continuous, uninterrupted access, you may want to think about biting the bullet and looking into a datacenter that is designed to never go offline.

We are actually looking into it for our largest client who also keeps food on the table. Our only issue is security a lot of our clients don't have approved vendors for a datacenter that is managed in a remote location outside of the office. Most of them want to send auditors in and see the servers, hardcopies and backups under constant lock and key. When I say security is a big priority in our company most new clients walk in thinking we're the FBI or CIA lol.

You have to have a working and practiced disaster recovery plan:

• Redunant systems
  
• Backup Power
  
• Redunant Redundancies
  
• Off-Site incremental backups
  
• Simulated outages for proof of concept
  
• Backup Plan is all else fails
  
• and most importantly $$$$
  

These are obviously high level concepts but i have no idea what your infrastructure is like nor do I expect a detailed statement on a public forum.

Oh we do a very good one but what happened last night no one could predict our home office was not warned of the power in the building being turned off because of flooding. Had it not been for one hell of a great IT director, I would've have been doing more than explaining we're down because interrupting the power for our Eclipse Servers causes them to just dump new information and I could've lost loads of Data that would've been very hard to explain. The case here is more so what do you do when your backup plan and contingency plans aren't enough to stop the inevitable. It's more of a how do I explain this to the dummies who don't get that we needed to shut down to save everything and not risk a shutdown. Our backups are off site but not local so even with that I had no way of getting my physical backups lol. It was murphy law cluster fuck plain and simple.

Your infrastructure needs to always include servers that can be powered from off-site. Sometimes it is not in budget but most critical designs have this options. In my case if the main building where to catch on fire I would be able to cold boot our virtual infrastructure from a off-site location and still be able to provide access to our infrastructure. Sometimes there is nothing you can do and for those that do not have the intellectual ability to comprehend what you are trying to explain to them I wouldnt waste my time. Focus on your plan of restoration and what would you do if you had some server that will not come up. You should ALWAYS have a disaster plan in the event of something like this. Execute it. If you can put off talking to customer I would while you are focused on bringing up your network. Get your backups ready, get your plan in place. If you need any help just let me know

Our plan is to keep the essentials online. Priority Grade A clients who will walk if we go off line those people were taken care of. We honestly were just not expecting for the building not to warn us they would be turning off the power so there was no way for our guys to get in and prep for it so when the power went out the back up batts kicked in but they are only there for priority components everything else had to be shut down. We are working on getting up now but its all remotely, I am just glad all of the offices maintained internet and local network access so they can still function day to day. Just need to figure out a way to keep backups for our important cases local without it costing an arm and a leg.

Well at least in my case, our servers are located at a datacenter that is prepared for stuff like this with several bus sized generators that can power everything at the datacenter for days at a time if need be. The last hurricane that came through put power out for everyone basically in a 50 mile radius, but the clients that did not lose power at their respective places of business never got disconnected from our network. I would say that if you are in a lucrative industry with several clients that need continuous, uninterrupted access, you may want to think about biting the bullet and looking into a datacenter that is designed to never go offline.

^^^

And dont forget to charge the customer uninterrpted service is not easy nor cheap.

I wish I had the say so to pull the trigger on this. It has been discussed but still hasn't happened. We have been thinking cloud for awhile but again until a clients auditors approves vendors for this we are limited to working with what we have for now. I am sure Sandy will make things speed up now though.

[JamieHoward]

Well at least in my case, our servers are located at a datacenter that is prepared for stuff like this with several bus sized generators that can power everything at the datacenter for days at a time if need be. The last hurricane that came through put power out for everyone basically in a 50 mile radius, but the clients that did not lose power at their respective places of business never got disconnected from our network. I would say that if you are in a lucrative industry with several clients that need continuous, uninterrupted access, you may want to think about biting the bullet and looking into a datacenter that is designed to never go offline.

We are actually looking into it for our largest client who also keeps food on the table. Our only issue is security a lot of our clients don't have approved vendors for a datacenter that is managed in a remote location outside of the office. Most of them want to send auditors in and see the servers, hardcopies and backups under constant lock and key. When I say security is a big priority in our company most new clients walk in thinking we're the FBI or CIA lol.

Yea ours too. We deal with electronic medical records. Walking into our datacenter requires jumping through a few hoops. The whole building is reinforced concrete with industrial steel doors requiring a key card and pin to get in to the second security checkpoint, which is the key card again and a monitored sign in and THEN a fingerprint scan. They do it up right I have to say.

[IBleedMusick]

Well at least in my case, our servers are located at a datacenter that is prepared for stuff like this with several bus sized generators that can power everything at the datacenter for days at a time if need be. The last hurricane that came through put power out for everyone basically in a 50 mile radius, but the clients that did not lose power at their respective places of business never got disconnected from our network. I would say that if you are in a lucrative industry with several clients that need continuous, uninterrupted access, you may want to think about biting the bullet and looking into a datacenter that is designed to never go offline.

We are actually looking into it for our largest client who also keeps food on the table. Our only issue is security a lot of our clients don't have approved vendors for a datacenter that is managed in a remote location outside of the office. Most of them want to send auditors in and see the servers, hardcopies and backups under constant lock and key. When I say security is a big priority in our company most new clients walk in thinking we're the FBI or CIA lol.

Yea ours too. We deal with electronic medical records. Walking into our datacenter requires jumping through a few hoops. The whole building is reinforced concrete with industrial steel doors requiring a key card and pin to get in to the second security checkpoint, which is the key card again and a monitored sign in and THEN a fingerprint scan. They do it up right I have to say.

That sounds like so much fun lol, one can only wish hopefully this storm makes it happen we definitely need a solution for atleast our important clients who shit bricks when stuff like this happens.

[beaker]

I used to work for the City of Baltimore, and we had 3 different sites that where all replicated in case of an outage. This was around 2003. It all worked great until a manhole fire took down the fiber ring.

All of the backups lost power and connectivity.

Ever since then they layed a seperate fiber ring. Now with cloud computing you can also do it that way, but I'm still skeptical on security.

The company I currently work for is worldwide, and the IT staff in terms of networking and redunancy is horribad. When 3 of our east coast sites went down, the users tried to WFH and very quickly overloaded our VPN (went from 5k users to 12,000 in an hour) and DHCP for the gateway ran out of IP's.

Bottom line is it all depends on the workload, the amount of people, and how much you're willing to spend to have the mythical "Five nines" 99.999% uptime.

[n8ball2013]

there are so many ways to approach this its not funny? all depends on how deep your pockets are. im on tablet but i know a thing or two about this