Power Requirements for Mission Critical Facilities
August 31, 2001

By Staff
Appeared in Consulting-Specifying Engineer

Case in Point: Designing a mission critical server farm for a global intranet

The power infrastructure of a 5000 SF server farm for a global intranet was engineered by Syska & Hennessy to provide a high level of service reliability.

Normally, the unit substation is fed from one medium voltage feeder. During power outage on this feeder, an automatic selector switch enables transfer to an alternate feeder.
The service switchgear contains the following components:
· 15KV, 3-phase, copper bus primary switch
· 600A, 15KV, 3- phase, copper bus, 25,000 AIC load breaking switch
· 1000KVA, 13,800/480-277V, 3 phase, liquid filled power transformer
· 2000A, 480/277V, 3 phase, 4 wire, main distribution switchgear

· Two ATS, each 1600A, 480V, 3-phase, double throw, electrically operated, mechanically held, transition switches with bypass isolation switch

Distribution switchgear
· A Main-Tie-Main arrangement with two separate boards connected with a tie-breaker
· Logic controllers allow automatic operation of tie-breaker and electrical interlock
· Each switchgear rated at 2000A maximum bus current, 480V, 3 phase, 4 wire
· Main breakers rated at 1600A, 480V, 3 phase

Standby generator
· Outdoor type diesel-powered engine: 1000KW/1250KVA, 3 phase generator
· Tanks supplied for 72 hours full load operation

Redundant UPS system
· Two static UPS modules, each 500KVA, capable of supplying entire critical load (based on calculated load density of 85 watts/sq. ft.)

Critical distribution panelboard and PDUs
· UPS systems provide power to critical distribution boards
· UPS output critical distribution boards provide power to PDUs located throughout raised floor area
· Each PDU has two power sources fed from different UPS systems
· Automatic transfer within 0.25 cycle

In addition, a house panelboard distributes power to lighting branch circuits, receptacles and small power for exhaust fans and office equipment, and two HVAC panelboards power heating, ventilating and air conditioning.

Clearly, this level of service reliability is not for everyone, yet it illustrates the design of a system based on an analysis of the nature of the owner's business and needs. In this case in point, the owner needed -- and got -- what they paid for.

For mission critical facilities such as call centers, corporate data processing centers, Internet servers, telco hotels and Web site hosting centers, 24x7 reliability has always been essential. With increasing demands being placed on the nation's power grids, the major possibility of power failures and brownouts is looming -- one needs only to look west to see the devastating economic effects. That is why owners of mission critical facilities today are demanding greater reliability than ever before.

While mission critical facilities' needs differ as widely as the nature of their business, they share certain needs for system components, design and interconnectivity. The following is an overview of the design issues related to electric service lines, the UPS system, critical distribution panelboard and power distribution units, and standby power generation.

Dual service essential
At a minimum, a mission critical site requires two electric service feeders. If possible, these should be fed from two different power generation substations to provide redundancy in the event that one of the power grids fails. The two lines should be tied to two sets of service switchgear - side "A" and side "B" -- connected with a main tie-breaker. This design enables each set of switchgear to operate independently under normal conditions, as well as to feed power to the other side, providing redundancy in the event of line loss or grid failure.

UPS system
An uninterrupted power supply (UPS) system is, first and foremost, a power conditioner which delivers clean power to sensitive electronic equipment. For most critical facilities, a static system is recommended, which uses DC batteries and rectifies the power through an inverter to produce clean AC power to run their electronic equipment.

During normal operation, a static UPS provides power to the mission critical load as a power conditioner while maintaining the battery charge. If there is a utility outage, its function is to provide temporary ("ride through") power during switchover to a standby generator system.

Depending on budget and space constraints, DC batteries can be specified to provide varied amounts of backup power, which enables time to do an orderly shutdown of IT systems in the event that the standby generator or generators also fail.

Depending on the degree of redundancy required by the facility, a UPS system maybe comprised of one module or multiple modules powered from each of the two sets of service switchgear. In a high-level facility with a pair of modules on each side, each module can serve as a backup for its partner, and if one side fails altogether, the pair on the other side provides backup.

Critical distribution panelboard and power distribution units
The power system can be designed so that the UPS system feeds either a critical distribution panelboard or power distribution unit (PDU), a larger, floor-mounted unit that may include a transformer to convert the power from the UPS from 480V to 120V. The panelboard or PDU in turn feeds the server racks and equipment on the data center floor. The power system can be designed to feed a PDU from either set of service switchgear through a static transfer switch, adding another level of reliability downstream -- closer to the critical load. A static transfer switch is capable of transferring the entire PDU load from UPS A to UPS B within a few milliseconds if either side fails without interruption to the electronics load.

Today, most servers come with dual cord power supplies -- one plug mold connected to the A side of the house, the other to the B side. If one cord fails, the other side keeps the server running. Even today, however, some equipment is supplied with only one cord. Fortunately, a couple of manufacturers offer rack-mounted static transfer switches, which plug into both plug molds, allowing the single-corded device to be connected to both UPS systems. This is highly recommended for mission critical facilities.

Typically the power system is designed to feed all the electronic loads from the PDU, with house panels reserved for lighting, electric receptacles and miscellaneous office equipment. Additionally, two critical HVAC panels are used -- one energized through switchgear A, the other through switchgear B -- to ensure redundant power for the computer room cooling system. A mission-critical facility must maintain its temperature and humidity to function properly.

Standby generator
The power system for a mission critical facility should be designed with one or more diesel generators -- capable of supplying the entire load of the facility with a minimum fuel supply of 24 to 36 hours. Multiple engines enable units to be taken off line for maintenance, as well as to pick up the load in the event one unit fails. Depending on the owner's business requirements, site and budget, multiple engines can be tied to a synchronizing or parallel bus to provide power to all mission critical loads and provide automatic priority load shedding to drop non-critical loads in the event of problems.

One or more automatic transfer switches (ATS) are part of the emergency generator system in most mission critical facilities. On normal operation, the ATS connects the entire load to the normal electric service switchgear through distribution feeders. In the event of a utility power failure, the ATS starts the generator and transfers the entire load to the generator. Upon restoration of normal power, the ATS transfers the load back to the service switchgear and the generator "cools down". During the period of transition, the UPS system feeds the mission critical electronics load without interruption.

How much is enough?
Mission critical facilities have varying degrees of criticality and, therefore, varying reliability needs. The nomenclature to describe it varies, too. Some owners, contractors and consultants use the N+1, N+2 … nomenclature; others use the system of "9s," in which the desired level of reliability is calculated by multiplying the number of 9s - say, 3-9s or 99.9 - by the number of hours in a year to determine the anticipated amount of annual downtime.

Although people may disagree about the proper nomenclature, most have one thing in common: they are pushing the envelope on reliability. Some owners are insisting on the need for 7-9s, 8-9s and 9-9s of reliability. To achieve this anticipated amount of downtime requires an enormous investment.

How much redundancy is really needed? To answer that question, one has to begin by setting aside preconceived ideas and assumptions and analyzing the nature of the owner's business in practical terms - what the business is and what the owner needs, including how they are going to run the IT side and process the data. Then translate this information an appropriate level of reliability, whatever nomenclature is used.

Say an owner wants to build three data centers across the country, and all are going to be online simultaneously. If any one should fail, the other two could easily handle the data processing. In this case, why would an owner need to build three facilities at 9-9s each when three facilities at 4-9s would ensure adequate reliability? But if a company were located in Hurricane Alley, with a single data center handling the entire business data processing, the owner might want to think about making the investment in a 6-9 to 9-9 facility. Other issues that must be addressed include the type of site - Greenfield, dedicated site versus multi-tenant high-rise, security, and on site staff capabilities.

Calculating critical load density
In addition to designing the system for reliability, equipment must be sized to accommodate an accurate reflection of the anticipated load densities.

To put things in perspective, a typical leased office space is delivered with 6 watts/sq. ft. (nominal). The critical power load of a mission critical facility can vary widely, depending on its function -- a corporate data center, Web hosting company and telco hotel, for example, are quite different in their functions and relative power needs. However, a realistic calculation of critical load will probably range from 40-80 watts/sq. ft.

Yet many owners today believe they need a power system with a minimum capacity of 100 watts/sq. ft., with future projections of up to 200 watts/sq. ft. Often their estimate is too high, which has costly ramifications. Many utility companies, whose power grids are rapidly becoming overtaxed, are now requiring owners to contribute to the cost of bringing in power to their site, with the promise of a rebate if - and only if - the customer reaches a certain level of consumption within a specified time period.

Second, many owners do not factor in the cost of cooling. In a mission critical facility, cooling equipment certainly will consist of computer room air conditioning, or "CRAC" units, but it will also require a system of condensers, dry coolers, cooling towers, chillers and pumps, depending on the site. In any case, air conditioning equipment will substantially raise the total power requirements.

At the end of the day, the owner, consultant and contractor alike need to ensure a mission critical facility is online within budget and on time. The owner's investment must deliver the proper level of reliability - but not to overpower it, both literally and figuratively. Proper planning is key.


Cyrus Izzo is a Principal/Vice President with Syska Hennessy Group, a consulting engineering firm headquartered in New York City, that provides technical solutions in such areas as mechanical/electrical design, facilities management, energy management, technology consulting/engineering, and turnkey design/build.