The trouble with insurance, disaster recovery plans (DRP), or any reactive kind of policy, is that human nature steps in and looks at the odds. “It won’t happen to me!” or “It’s so rare, why should we worry about it?” Well, we here at BIT fortunately never succumbed to that kind of thinking and did our job in regards to preparing for the worst.
Mid-morning just over a week ago, we heard a commotion in the hallway and upon closer look discovered that there was water coming from the ceiling, down the halls and a waterfall in the stairwell. A pipe had burst a few floors above us and the whole building was being flushed with a seemingly unending torrent of water. As our neighbours were scrambling to put down cardboard boxes, buckets, old rolls of drafting paper and whatever other absorbent materials they could find, we went into emergency response mode and verified all the planning and protection we had in place was functioning properly. Though we were requested to evacuate the building as per the fire marshal, we were confident that no client services would be affected during this event. You see, we never put all our eggs in one basket and as a result, even with a shutdown of some internal servers, our client-facing infrastructure never noticed. Redundancy, data center hosting, online service and other key components to a good disaster recovery plan left us in a position where the only set-back we had was not being in the office. Notifications were sent to all our key account contacts within 2 hours of the evacuation informing them that contact via mobile phone would be the best method to get in touch with their dedicated tech staff and that we would update again when we were back in the office.