Don't Be Left In the Dark - Prepare Now for Power Blackouts
July 01, 2001
Appeared in Journal of Property Management
As the threat of power blackouts rolls across the nation, property managers may be lulled into a false sense of security by the presence of a backup generator or industrial-strength batteries designed to supply electricity during a sudden loss of power caused by the local utility or a natural disaster. You might be surprised to find out that just the mere presence of a UPS - uninterruptible power supply - system is not the solution to keeping the lights on and your data protected during a power blackout. In fact, thousands of companies across the nation are finding out the hard way that their backup power supplies are not fully reliable and cannot prevent serious downtime that will cost millions in lost revenue and lost “reputation.”
OUT: Timeline for a Power Outage
In order to avoid being caught in the dark, property managers need to plan for a potential power outage. The following actual case study scenario can be avoided through expert planning, engineering and processes.
13:59:01 Commercial utility failure. All power cut to facility.
14:00:08 Facility backup generation systems fail. Computer systems still supported by battery backup UPS.
14:04:27 Company's operations team attempt unsuccessfully to restart back-up generation systems.
14:06:45 Company's operations team notifies internal users. (Typically small operations teams can only get to 25 percent of users in time). 14:09:59 Computer system users make business decisions for orderly shut down or risk possibility of hard crash.
14:13:18 Operations team responding and trying to contact critical facility vendors for system support.
14:14:41 Operations team briefs senior management of risks and potential revenue implications.
14:15:30 Full system failure. Business operations cease! Computers shut down from thermal overload and lack of cooling within 8-10 minutes. Battery backup systems depleted.
15:06:00 Operations team waits for arrival of critical facilities equipment vendors—firms which run back-up generators. (In a major city outage hospitals take priority—could take several hours.)
16:09:00 Utility power restored or repairs made.
17:02:00 Critical facilities equipment vendors arrives; restart process begins—could take 4-8 hours.
23:56:01 Facility fully operational. (Restart process took 7-10 hours.)
Many backup systems are redundant in name only - they have not been tested since the threat of Y2K outages. At one building’s data center, batteries expected to last for 30 minutes failed after only two minutes, leaving the company in the dark and in total confusion. At another firm, facilities engineers were carrying around sets of inaccurate and outdated notes for responding to a crisis. Fortunately, the company avoided a potential disastrous outage by creating an electronic database containing a standardized, complete and accurate power recovery plan.
To complicate matters, most buildings rely on emergency generators and backup batteries for fire/life safety only. These systems are often insufficient to carry the loads required by data centers, server farms or other technology-dependant tenants. And, they require permits to be used as a power source beyond a life safety use. Companies will suffer stiff penalties and may be shut down by local authorities if they try to run their mainframes and Web servers with power sources meant for emergency lighting and fire alarm systems.
At a minimum, a property’s generator should be equipped with a separate transfer switch and switchboard so that power needed to support fire/life safety needs is controlled independent of power needed for any other uses. Building owners need to check the capacity of their generators to see if they can handle additional loads. Generators supplying computer loads need to be carefully sized and coordinated with UPS equipment to ensure they will not stall out when they are most needed. Also, in areas where the current energy crisis is expected to cause extensive rolling blackouts, facility managers and owners may need to file variances to get extensions beyond the 200 hours/year that most generators today are licensed to operate.
If the building’s generators are marginal in terms of capacity, property managers need to develop a load-shedding scheme to shut down non-essential loads such as workstation power and office lighting so computers can stay up and running. And generators should be tested at full capacity not just started up once a month. If not, the system may fire up just fine, but shut down when a full load is transferred during a blackout. Outside contractors can provide a “load bank” to adequately test the system by simulating a full load on the generator.
Quite often, property managers don’t have provisions in place to supply the amount of diesel fuel or natural gas needed to run their backup generators. Propane can be stored for natural gas-powered generators in the case of ruptured gas lines often caused by earthquakes or other natural disasters. Indeed, enough fuel should be on hand to handle your power needs for several days - the time it often takes for utility crews to reconnect severed gas lines. Building owners need to have solid contracts to supply diesel fuel or non-interruptible service from the gas company. The higher cost of these services must be weighed against the millions of dollars in lost revenue when a business has to shut down.
The Planning Process
Cost is a major consideration for property managers who are considering a site move. It’s not a simple matter to move a computer operation from one site to another without a disruption and loss of business. To move a UPS to a new location without disrupting existing operations takes careful planning. Computers should not be left without protection from utility disturbances during the transition to the new space.
Reliability hinges almost entirely on things you can control by means of expert planning, engineering and processes. The impact of uncontrollable circumstances can also be mitigated to near zero. This applies to existing facilities and operations, where people should be trained, cross-trained and regularly drilled. Emergency processes should be documented, improved and tested. Facilities should be retrofitted. Older backup systems should be upgraded. And it’s wise to contact your local utility and inquire about how advanced notice of power blackouts will be received.
Better yet, the site selection and planning process provides the very best opportunity to build reliability into a facility from the ground up. Doing that requires a little upsidedown thinking and a lot of up-front engineering. Here’s a quick test to gauge the uptime reliability of any one site:
Have you had one or more instances of power interruption in the last six months?
Is your facility located outside the critical “last mile” of a connection to high-speed fiber?
Can you be affected by a power company’s systematic rolling brownout?
Are you located in Tornado Alley? Earthquake Zone? Hurricane Country? Flood lands?
Has it been a year or more since you updated your maintenance processes or facilities manuals?
Does your indoor temperature fluctuate by more than five degrees?
Do you have water pipes in, overhead or within bursting range of your computer rooms?
Can an unauthorized individual come within 10 feet of your servers?
Are you susceptible to any single- point-of-failure in your facility or operations?
Are you less than 100 percent confidant that your facility will be up and running every morning?
The answers you come up with are a starting point for improving your operational reliability. It’s a good idea to conduct a regular schedule of downtime simulations to keep your people and processes ready to handle any crisis. Just relying on equipment to function properly no longer provides enough reliability when the life of your company or your tenants’ businesses are on the line.
Jerry Burkhardt, a partner at Syska Hennessy Group’s OnlinEnvironments, (www.onlinenvironments.com) specializes in the design, installation and maintenance of power systems for major corporations and government operations.