Build the Next-Generation Data Center
6 week online course starting in spring 2017

Disasters and Recoveries, Part 2

You wouldn’t believe what your second most pressing problem is when you lose electricity for a few days in the middle of a winter storm: freezer. Being a good engineer focused on redundant solutions, I bought a diesel generator before moving into the hills to keep the freezer at a reasonably low temperature in case of a long-term power loss.

I also thought about using the same generator to run our central heating. As always, I found a huge disconnect between theory and practice.

Untested recovery solutions are useless. Relying on them is stupid.

During the first longer power outage we experienced last Friday I decided to connect the central heating to the generator. Opening the distribution box revealed a huge spaghetti mess of unlabeled cables.

Documentation is mandatory in a well-run IT operation, but it’s crucial during any disaster recovery process. You have to have a well-documented current state and the step-by-step recovery procedure.

I’d explained my generator-run-central-heating idea to my electrician when he’d been installing the wiring, but obviously he never understood me and did things his way (hint: totally useless for my purposes).

Badly implemented redundant design is sometimes worse than a non-redundant one. With a non-redundant design you’re at least aware of the actual (non)capabilities of the system.

Even worse, the electrician decided to make his life easier and didn’t use proper wiring techniques – some cables running from the central heating to the distribution box had a single wire connected to the patch panel (the other two were simply cut off).

Don’t trust subcontractors without verifying their work at least a few times, or you might be in for a nasty surprise at the worst possible time.

Eventually I gave up – there was simply no way to untangle the mess, and get the central heating controller connected to the diesel generator without a potential catastrophic error. I decided it was more important to have a working unit after the power would be restored.

Know when to give up. Implementing untested ideas under time pressure might be worse in the long run than not doing anything.

Learn from your mistakes

After the heating season is over, I’ll definitely rewire the whole thing in a way that will make it easy to disconnect the heating unit from the power grid and plug it into a generator.

Every good engineer should learn from his own errors, omissions and plain stupidity. The lesson might have been expensive, but if you don’t improve your design, implementation or procedures based on what went wrong you just wasted a huge opportunity.

I’ll also buy a small camping gas stove … did I mention that we replaced a gas stove with a great new induction stove a few years back and never though about redundancy?

Don’t forget to update your disaster recovery design and procedures every time you introduce new technologies/products or replace an existing one.

Back in the real world: things are not looking good

The sleet caused a total collapse of a major 110 KV power line traversing the area where I live. If we're lucky we'll get power on Saturday or Sunday.

7 comments:

  1. Excellent post. I couldn´t agree more. How large is your Diesel tank BTW and how do you keep the Diesel from degenerating ? - just a thought before you run into the next mess ;).

    ReplyDelete
    Replies
    1. Unfortunately I'm not that advanced. It's a small (portable) generator and you keep the gas from degenerating by pouring it in when you need it ;)

      Delete
  2. My sympathies Ivan.
    We experienced same problems here 15 years ago:
    http://archives.radio-canada.ca/environnement/catastrophes_naturelles/dossiers/265/
    and lots of power-lines have been routed underground since then. Pylones have also been redesigned:
    http://www.quebecurbain.qc.ca/archives/jc100_0095b.jpg

    ReplyDelete
    Replies
    1. I sincerely doubt that our power companies will learn anything from this experience. We get black ice like this every 25 (or so) years and I haven't noticed any significant progress in their power line building technology.

      Delete
  3. Great post and lessons. I just had an outage in my small office in NY recently used my labs UPS to carry me for a bit. And I have been in your shoes in the past so I empathize what you went through and we get some harsh winters/ice storms etc too. In your case if for tactical reasons you could have just had some long extension cords, space heaters etc to deploy a temp distribution system from generator to house to heaters/lights without messing with inside wiring mess from contractor. Ugly but quick, easy, mobile and functional.

    Stay warm

    ReplyDelete
    Replies
    1. Yet another lesson: talk about your challenges ;)

      On a more serious front, you get way more bang for the buck if you manage to get the central heating up and running.

      Delete
  4. Lesson here is the same as always: if you want something done right, you have to do it yourself. They'll of course tell you that you must hire a licensed electrician, a licensed plumber and a licensed supervizor a licensed banker to finance the operation, etc, etc. But again, if you want things to work at the end of the day . . . there is simply no alternative but to do all the critical parts of the project yourself.
    No, i'm not a DIY maniac. I do hire people when i have to. But when I need a "life support system" i do it myself, even if it takes a long time to get educated and to implement. This approach has not failed me so far and yes, I do lose power from time to time, sometimes for a week or more.

    ReplyDelete

You don't have to log in to post a comment, but please do provide your real name/URL. Anonymous comments might get deleted.