Nuisance-Trip Hardening for Critical Distribution Systems
April 01, 2005

By Staff
Appeared in EC&M

Sometimes attempts to improve reliability at the design stage can actually do more harm than good

Figure 1: Single-line diagram of the data center's multi-module parallel redundant UPS system Figure 1: Single-line diagram of the data center's multi-module parallel redundant UPS system

When the power went out at a major U.S. data center, the first question on everyone's mind was, How did this happen? The facility was protected by a number of uninterruptible power supply (UPS) systems, so an outage should have been out of the question. Figure 1 shows the single-line diagram of the data center's multi-module parallel redundant UPS system. This system uses draw-out airframe power circuit breakers in the paralleling switchgear cabinet (designated SBM) and maintenance by-pass cabinet (designated MBP). Breaker CBS is used for control and protection of the UPS system module output bus. Breakers FBP and CBP are used for control of systems automatic by-pass. Breaker MIB is used to isolate the UPS during maintenance operations while breaker MBP is supplying power to the critical load.

A quick check of the system revealed that breaker MIB had nuisance tripped. And because they were located in strategic positions downstream of breaker CBS, the static switch and all UPS systems lost power when this breaker tripped opened. Electrical maintenance personnel later manually restored power through maintenance bypass breaker MBP.

The data center owner launched an immediate investigation to find the root cause(s) of the event. Inspection of the trip circuit breaker flag showed that the breaker had recorded 5,130A for more than 200 seconds. However, power-monitoring equipment installed on both sides of the breaker or the UPS controller saw no changes in the 1,200A and 480V readings on the breaker at the time of the trip. An analysis of the current waveform's four cycles before the trip found no elevated levels of harmonics or impulse events. Everything appeared normal except the instantaneous loss of power when the breaker tripped.

Electrical maintenance personnel removed the breaker and sent it back to the manufacturer for extensive testing, but they were unable to find any defects in the device. Concurrently, thorough measurements of the site's environmental characteristics haven't led to any indications as to the probable cause or causes. To date, the owner has found no definitive reason for breaker MIB to have tripped.

  is unacceptable in data centers, but don’t create problems by overdesigning. Downtime is unacceptable in data centers, but don’t create problems by overdesigning.

So what happened?

Circuit protection overkill. The above example is the reason why electrical engineers design power distribution systems in today's critical facilities to 2N or better standards of redundancy (“N” being the amount of equipment necessary to support the facilities' critical loads). In this case, a 2N distribution system would clearly have mitigated the impact to the data center's business.

Though 2N is an overall solution, is this really the best option for you or your customer? In this situation, the nuisance trip at the data center demonstrates a more fundamental problem that could be eliminated at the system design stage. Although the system design should have been laid out with an eye toward reducing the number of components in the system and thereby increasing reliability, it specified an automatic trip unit in breaker MIB that actually reduced the system's reliability.

Over the years the use of standardized circuit breakers (CBs) in uninterruptible power supply (UPS) switchgear has become the “norm” for the industry. While this arrangement can make manufacturing more predictable, standardize testing, reduce diversity in inventory, and offer the perception of improved safety, the situation at the data center demonstrates that it has its disadvantages.

Automatic circuit protection in the UPS equipment at the data center was unnecessary because breaker CBS (when the UPS is operating normally) and breakers FBP and CBP (when in by-pass mode) would have provided all of the necessary protection for system components. The system would have been inherently more reliable if the distribution design used a non-automatic maintenance isolation switch (MIS) rather than the automatic breaker MIB.

Once you recognize that the UPS has become a more reliable system by removing the automatic trip function from breaker MIB, you can also apply this form of “hardening logic” to breakers FBP, CBP, and MBP. Now, either the upstream utility or 4,000A switchgear fuses will protect these devices. The removal of the automatic trip units from this equipment can further reduce the possibility of nuisance trips, which inherently increases the overall system reliability.

Multiple-level applications. This methodology is also applicable in situations where multiple levels of identical circuit breakers are present in any distribution system. For instance, a 2N UPS installation can be broken down into three distribution levels: source, intermediate, and critical load.

In this design scheme, all of the source and intermediate level breakers are identical. Those on the intermediate level are used to control distribution or isolation switchboards. Circuit breakers at the generator, service entrance, or source level switchgear provide circuit protection for this equipment.

Let's say this type of system has 28 intermediate level breakers. Specifying automatic circuit protection for all of these devices would unnecessarily reduce overall reliability of the system. But through detailed and careful analysis, it's possible to identify specific instances where you can use a non-automatic breaker or switch. As is the case with the single-UPS system, the elimination of as many automatic functions within the distribution system as possible will decrease its susceptibility to nuisance events and therefore improve overall system reliability.

Certainly, some cases will necessitate overcurrent protection. When providing NEC-required overcurrent protection of downstream busses or feeders, you must design fully protected electrical distribution systems — you can't make compromises with distribution system protection and safety. But you can exercise discretion with the type of equipment used to control or isolate components of the system. Whenever possible, use non-automatic devices for non-protective applications.