Those Who Work in the Networking/IT Field - Advice

[IBleedMusick]

there are so many ways to approach this its not funny? all depends on how deep your pockets are. im on tablet but i know a thing or two about this

Honestly the pockets of this company are very deep but our IT committee who is full of a bunch of people who really don't understand the IT field won't approve anything out of the norm that they feel just isn't necessary so I spend most of my time going around and figuring out ways to work with what I have but after this fiasco I am sure that will change quick. Within the first hour of no power the entire team of Forensics/Data/IT guys were on a conference call discussing how it should went down and what should have been done but we know no one will listen to us. The IT director has been begging to try out a cloud setup but as with what beaker said our IT committee doesn't trust the security of it. Only issue I see here is the fact that they feel like since we can do our jobs without it why do we need it? Someone actually said I guess we will have to add more batteries or maybe a backup generator but to me it seems like that is nothing more than a bandaid for awound that needs stitches. I have invested a lot into this company and I am in it for the long haul but if these things keep being overlooked I may have to start second guessing my decision to become a lifer here. While I know this company isnt going anywhere for a very very VERY long time, I want to see it grow into something people know. Hell I want to be able to say I work for XXXXXXXXX and people instantly know who we are. I don't think thats too much to ask.

I would be interested to know what your solutions would be on this N8 because so far I have taken Sensai's, Beakers, and everyone else's suggesstions to my IT director who for the most part had the same idea but no funding to put it in play. We have a Partners meeting coming up to discuss a solution for the Sites not to go down and I plan on bringing it up amongst the panel to see what can be done because honestly next time I am going to start handing out their phone numbers and let them deal with the backlash.

[n8ball2013]

First and foremost. Your data guy that left the backups behind should be canned. See ya.

Your IT committee is to blame for this. They need to understand that. Any of them credentialed IT people? If not. They serve no purpose. But youll also have to present to them a cost benefit analysis. Do you pay SLA penalties for servers being offline? How much did this outage cost all your customers affected? Put a dollar sign to it and show them a quantitative analysis and they may get the picture. Is there a dedicated disaster plan? Is it tested for each account you have? If not why? IF they claim to be so vested in security they obviously don't have a clue. Security is much more than keeping unathorized people from viewing things they arent supposed to.

There are a lot of variables I dont have the answers too and that makes it hard to accurately size. If zero down time is important then a redundant site is necessary. Especially if you are spread out all over the country and each site has their own servers on site (Gag). Id definitely explain the benefits of a datacenter. Id even think paying for dedicated bandwidth at multiple sites is more expensive than a larger chunk at a single site would be one benefit. The other is little to no downtime at a top tier data center. Just to put it into perspective. Monday night I got an email saying they were switching all datacenters to generator power as a precaution. They expected no issues powering the full centers to capacity until power issues were resolved. Let me know if you need a short term consultant on this

[IBleedMusick]

First and foremost. Your data guy that left the backups behind should be canned. See ya.

Your IT committee is to blame for this. They need to understand that. Any of them credentialed IT people? If not. They serve no purpose. But youll also have to present to them a cost benefit analysis. Do you pay SLA penalties for servers being offline? How much did this outage cost all your customers affected? Put a dollar sign to it and show them a quantitative analysis and they may get the picture. Is there a dedicated disaster plan? Is it tested for each account you have? If not why? IF they claim to be so vested in security they obviously don't have a clue. Security is much more than keeping unathorized people from viewing things they arent supposed to.

There are a lot of variables I dont have the answers too and that makes it hard to accurately size. If zero down time is important then a redundant site is necessary. Especially if you are spread out all over the country and each site has their own servers on site (Gag). Id definitely explain the benefits of a datacenter. Id even think paying for dedicated bandwidth at multiple sites is more expensive than a larger chunk at a single site would be one benefit. The other is little to no downtime at a top tier data center. Just to put it into perspective. Monday night I got an email saying they were switching all datacenters to generator power as a precaution. They expected no issues powering the full centers to capacity until power issues were resolved. Let me know if you need a short term consultant on this

Oh no he is in it deep and this isn't the first time he has had a major screw up. This error made a lot of higher ups second guess the cheap labor of our Data team, so much so we may see some of our vets make a valiant return being payed what they deserve (a lot of them left when our competition made an offer I almost didn't refuse)

They are very very much to blame for this and no none of them are know anything but their the ones that make the decisions, they have stake in this company so they are frugal. Be that as it may your idea is not bad at all. The outage itself cost a lot of our clients some money if you think about the hourly rates some of them charge that they couldn't because they had no way of working or a lot of Depo's and Arbitration’s that had to be rescheduled because of what happened this was very costly in the end. I will get some definite numbers when I get into the office on Monday. I will ask my IT director if he is penalized for servers being down. The disaster plan I don't even want to talk about because that was a joke. My only real issue was the fact that one of our newer databases has no setup in another location so the physical backups was all I had and its the one I really need not to be offline for too long. Most of our guys have the ability to pull documents remotely but unfortunately with no backup they can't do anything.

I am hoping real generator power will be considered no reason our NYC office which is where our IT/Data dept is in its entirety so there is no reason it isn't able to run full capacity for extended periods. Just think they say Miami is not a good spot to keep a second set of backups even though we are the reason for 40% of the data in the company. If I would have had my imaged drives I would have been able to locally upload them to a network share drive and the clients would have been given access (within the office) to work from them. Since I am On-Site it would have been so easy to setup and maintain. Thanks for the advice though I am going to talk to him on Monday get some numbers and really sell the idea of actual plans for things like this because this cannot and will not happen again. Now that I am in a position where I have a voice I can actually speak up and bring this to light. I tell you this much as soon as they get in the office I am having every clients database imaged and sent to me to keep a backup with me in my office. As for the cost of dedicated bandwidth (we use cogent) in each office I wish I knew the cost but I know it's not cheap and though I don't know much indepth about servers and networking outside of basic troubleshooting and functions I know it was explained to me that there is a purpose for it.

[n8ball2013]

if none of them are IT people they can not consider this a discretionary expense. Things like this are mandatory. Don't be surprised if lawsuits start coming in for breach of service. Another thing that needs to be addressed is your backup policy. It should never come down to one guy dropping the ball and its a major issue. Tapes should never be stored on site either. Fire or anything of the like and you not only lost servers but the viable backups as well.

If you are hosting services for a client whether its Saas or Iaas you must have generators available. Be careful with the imaged databases. If its not encrpyted you run a high liability of having them. Any breach and they will be looking at you. Especially if they are pushing security and don't have any concept of what it really encompasses.

The bandwidth isnt something youd have access to. Its somethign a guy like me who does infosec would need to request from someone way up top. It really sounds like the only way you are going to be able to get through to them is to show them the hurt. Show them a dollar sign amount and they will get that. Its ridiculous to have a committee with one IT guy on it that gets walked on.

so in short. Show them the total cost of this outage. If you can figure what the total cost of an actual disaster recovery program would be you can even show them the savings. Theres a hwol ebunch of formulas to get into but without the figures its pointless to even attempt to explain all of them

And then push a cloud computing environment. Having everything centralized is so mcuh easier than having remote offices with each having their own servers. Not only does it allow for less techs (saves money) it allows you to set up a backup site if needed among other things.