The Verge: "Post-hurricane, New York's internet industry runs on diesel." Sandy knocked out power to key data centers on which publishers like HuffPo, Gawker, and Buzzfeed relied.

  • Sam Ley

    Hey, that’s my bro they quoted from Fog Creek! Those poor guys have been working non-stop since the hurricane trying to keep their services online – they had everything planned out perfectly until the flood waters fried the fuel pump.

    Just heard that someone has shut down their bucket brigade for some sort of legal reason, and that they have less than 3 hours of fuel left. Power not expected until tomorrow at the absolute soonest.

    • nixiebunny

      It seem like they ought to let it go. You can only carry so many gas cans up 16 flights of stairs before you get tuckered out.

      New York isn’t designed for flooding. There are many coastal cities that can handle it, because it happens often enough that the buildings have to be designed to accept water. But who knew that the ocean would do that?

  • Bad Juju

    And this is news..how?  Should I be outraged it’s not biodiesel at least?
    Generators do have to run on fuel, it’s true.

    When you get a chance, look up the story of what the folks at DirectNIC in New Orleans had to do to keep their hosted sites up through Katrina.

    • Sam Ley

      The interesting part to me is how these disasters show just how fragile even our redundant systems are. This building’s story is not uncommon – they had a great backup generator, plenty of fuel, remote controls and alarms, UPS systems setup to carry the datacenter during the power change-over, etc., and it was all foiled by having the small fuel lift pump get damaged.

      Same thing at Fukushima – the emergency systems were all in place and functional – but were entirely crippled by one set of pumps being damaged.

      You see these single-point failures all over in our redundant systems, which severely reduce their real-world redundancy. We just haven’t had many opportunities to test them, and everytime we do, we learn we aren’t quite as prepared as we thought.

      I also am somewhat amused by the image of software developers and weary sysadmins humping 5 gallon buckets of diesel fuel up 17 floors. Adds a very physical component to what is normally a very abstract pursuit.

  • http://www.zhrodague.net/ Drew from Zhrodague

    Expedient, in Pittsburgh, has the largest 4 diesel generators I’ve ever seen, and several battery rooms. Even larger than that place near the Totten Pond exit in Waltham, MA. I didn’t see InterNAPs generators (Boston), but they did have some big power. For redundancy, it helps to have your stuff in more than one geographic location – just like the requirement for DNS that is often overlooked.

  • http://beautifulsynthesis.com Andrea

    This kind of a thing is a concern where I work – way out in Nebraska. We’ve caught flack from our clients (investment advisors throughout the U.S.) when snowstorms have shut down our office. Our systems stay up, thanks to a diesel generator and data backups, but until last year, we didn’t have a large-scale system in place to allow employees to work from home without compromising data security. We’ve worked that out since.

    We have a whole string of contingency plans, the biggest one being backup servers in a building that’s on a different part of the power grid, hooked up to a different water supply, in a different watershed, and unlikely to get hit by any tornado that takes out our main office.

    This week we had three days when nothing happened because the stock market was closed. It was strange to have “hurricane days” when it was so bright and sunny out.

    It’s probably the one thing that makes me uneasy about our reliance on technology. It is SO easy to take down the system.