Setting the Stage for Liquid-Cooled Solutions
June 30, 2006

By Staff
Appeared in Energy & Power Management

Recent technology advancements by IT equipment manufacturers have resulted in heat-producing hardware that pushes the envelope of traditional air cooling methodologies. In response, various infrastructure vendors have introduced liquid-cooled solutions as viable alternative that can accommodate as much as 30 kilowatt (kW)-per-rack or more. Liquid-cooled hardware is defined as data center equipment in which the primary heat transfer medium is a liquid that exists internally within the electronics. These technologies and some to be introduced have different efficiencies and advantages.

First, with the exception of some uncommon specialized hardware systems (and surprisingly, laptops), for practical purposes, no liquid-cooled hardware is used in commercial data centers today. Some supercomputers and old legacy mainframes used liquids piped directly into hardware to cool heat-generating electronics. Some specialized military electronic packages use evaporative spray-cooling technologies, and some computer game manufacturers utilize true liquid-cooling designs.

The liquid-cooling solutions hitting today’s market still require air-to-liquid heat exchangers that provide air cooling at the individual rack where the electronic equipment is housed. The IT-hardware chassis still includes integral fans that move cool air through the inside of the hardware chassis to move the heat from the electronics to the air within the cabinet (or rack). Technically, this is still air-cooled hardware.

Defining the cooling scenarios typical of today’s datacom facilities and some baseline assumptions and limitations associated with this analysis makes apple-to-apple comparisons easier, especially if the goal is to focus on what (if any) energy savings can be attained using the various cooling options available. Obviously, different geographic locations will have different outside environmental conditions that can affect the potential efficiencies of heat rejection to atmosphere. Air-cooled chiller, dry-cooler, and cooling tower efficiencies are affected by the outdoor ambient conditions. Some locations afford the potential (at least during certain seasonal periods) for employing air or water-side economizers. For comparison, these parameters will be normalized to allow a better comparison of the different scenarios that exist within the data center proper.

The comparison includes five distinct scenarios involving current technologies and three additional scenarios that may become available in the future. The scenarios are:

Parallel heat flow/rejection paths (e.g. some heat to pumped liquid, some to room) where a mixture of the above scenarios are employed are not considered within the context of this comparison.

Each scenario begins at the heat source (i.e., the IT hardware) and ends with the heat being rejected to the outside ambient environment (i.e., to atmosphere). Each breaks down the heat transfer path to incremental steps to include the medium (water, air, refrigerant, etc), the prime mover (fan, pump, compressor, etc), and the heat-transfer component (heat exchanger, fluid-cooler, chiller, cooling tower, etc.). Then, the scenario details total system energy efficiency for each scenario. Also for the sake of comparison, each scenario assumes the hypothetical facility employs recognized best practices such as minimized bypass air, balanced air and/or water distribution, and equipment properly maintained and operated per the manufacturer’s recommendations.

The energy calculations are based on assumptions and/or “givens” for a representative facility of 1 MW equipment input and cooling equipment selected according to current best practices to handle that total quantity of heat rejection:

Scenario 1 – Conventional Computer Room Air Conditioners (CRACs)

IT hardware chassis fans move the generated heat from the hardware chassis through the cabinet to the ambient room environment. CRAC fans pull the air across the cooling coil (heat exchanger), which absorbs the heat into a central chilled water system. The chilled water is pumped back to a chiller plant, which transfers the heat via a compressorized refrigerant cycle to either the outside environment directly (air-cooled chiller) or to a condenser-water system (water-cooled chiller). In the case of a water-cooled chiller, a pump moves the condenser water to a cooling tower (water-to-air heat exchanger), which rejects the heat to the outside environment. (air-cooled chillers are not considered here.)

Scenario 2 - Rear Door-Mounted Cabinet Cooler

IT hardware chassis fans move the generated heat from the hardware chassis through a cabinet rear door-mounted cooling coil (air-to-liquid heat exchanger). Note that this scenario uses the IT hardware chassis fans to move the heat through the rear door-mounted cooler. Ideally, the air leaving the cabinet cooling coil is at ambient room conditions. The heat absorbed by the liquid in the cooling coil is pumped back to the central chiller plant where the remainder of the heat transfer methodology matches that described above in scenario 1.

Scenario 3 - Compressor-less Liquid-Cooled Cabinet

IT hardware chassis fans move the generated heat from the hardware chassis. A cabinet mounted fan moves the heated air across a cabinet-mounted cooling coil that utilizes a refrigerant (air-to-liquid heat exchanger). The refrigerant leaving the coil can be either liquid or gas. The refrigerant is then cooled by a cooling distribution unit (that can support more than a single cabinet). The refrigerant leaves the cooling distribution unit as a liquid where it is pumped back to the cabinets. The cooling distribution unit incorporates a refrigerant-to-water heat exchanger. The water is typically chilled water and then pumped back to a central cooling plant and again proceeds as described in Scenario 1 above.

Scenario 4 - Water-Cooled Cabinet

IT hardware chassis fans move the generated heat from the hardware chassis to the air contained within a sealed cabinet. The sealed cabinet includes an integral fan-coil typically located in the bottom of the cabinet. The air within the cabinet is continuously recycled through the cabinet and the heat rejected from the IT hardware is absorbed by the integral fan-coil. Note that this scenario uses the IT hardware chassis fans to move the heat to the cabinet air and the cabinet fan-coil moves the cabinet air across the cooling coil. Ideally there is no air leaving the cabinet to the room ambient environment. The heat absorbed by the liquid in the cabinet fan-coil is pumped back to the central chiller plant where the remainder of the heat transfer methodology matches that described above in Scenario 1.

Scenario 5 - Conventional Air Handling Units

This scenario is essentially the same as that described in Scenario 1 except that fewer, larger air handling units replace the smaller CRACs.

New Technologies

The remaining three scenarios depict true water-cooled IT hardware that essentially does not currently exist in the commercial datacom repertoire. It is quite possible that as IT equipment heat densities continue to climb, IT manufacturers may soon introduce new product lines that incorporate actual liquid mediums in direct contact with the electronics to remove the heat from within the chassis to a point outside the equipment where the facilities utilities can transport the heat to the outside atmosphere. Currently, ASHRAE’s Technical Committee TC9.9, Mission Critical Facilities, Technology Spaces, and Electronic Equipment, is preparing a 4th book in the “ASHRAE Datacom Series” that will address the facilities’ interface to existing liquid-cooled cabinets and to some extent, possible liquid-cooled IT equipment of the future. This book, tentatively titled Liquid Cooling Design Considerations for Data and Communications Equipment Centers, is slated for publication as early as the summer of 2006.

Scenario 6 - Water-Cooled IT Hardware with Chillers

This scenario assumes introduction of IT hardware that employs a water-cooled heat sink (possible a cold-plate) that “utility” water is piped directly to. Note that there would be no chassis fans or cabinet fans and the IT hardware would be liquid-cooled by standard central plant chilled water.

Scenario 7 - Liquid-Cooled Hardware with Chillers

This scenario is similar to Scenario 6 except that the IT hardware would incorporate a liquid medium pump within the chassis and an associated pump to move the heat to a separate cooling distribution unit. The cooling distribution unit could support one or more liquid-cooled IT components and is essentially a liquid-to-water heat exchanger. Chilled water from the central cooling plant would cool the cooling distribution unit and the remainder of the heat transfer process again matches that described in Scenario 1.

Scenario 8 - Liquid-Cooled Hardware without Chillers

This final scenario is similar to Scenario 7 but eliminates the need for the central cooling plant and assumes the cooling distribution unit is cooled by water (or water-glycol mix), which is pumped outside to a fluid-cooler where fans reject the heat directly to the outside atmosphere.

General Conclusions

Some general conclusions can be drawn from this analysis, as follows:

Some specific conclusions can be drawn from this analysis:

Power demand in to the equipment will always balance with the energy out in the form of heat. Newer technologies relate to the costs of transporting that heat from the IT equipment to the ambient surrounding the facility. Obviously, some techniques for transferring this heat are more energy efficient than others, and these need to be considered as part of the total cost of ownership of the enterprise and associated physical facility. Other issues relating to equipment reliability, maintainability, and especially the issues relating to the IT equipment configurations as they relate to the core function of the facility, must be considered completely before any decision is made in selecting the most appropriate cooling system for a data center facility.

About the Author: Vali Sorell has over 20 years of experience as an HVAC design engineer, with at least half of those years involved with design and commissioning of HVAC systems for critical facilities. As Syska’s National CFD Expert, he performs CFD analyses to compare and evaluate various system design options for high density equipment cooling configurations. In addition, he researches how to quantify the environmental conditions inside as well as outside cabinets as a function of cabinet configuration and cabinet load. Prior to joining Syska, he served as an in-house consultant to a co-location firm that develops and operates data centers in various cities nationwide.

Terry L. Rodgers, CPE and senior associate at Syska Hennessy Group has had over 25 years experience as a mechanical engineer. Previous to Syska, he was employed within the Nuclear Aerospace, Government, and Financial sectors of critical facilities. Currently at Syska Hennessy Group, Terry is a client leader and is slated to manage their Charlotte City office beginning in July of 2006. Terry is also a member of ASHRAE TC9.9 “Mission Critical Facilities, Technology Spaces, & Electronic Equipment”, participated on the ANSI Homeland Security Standards Panel working group for Enterprise Power Security, and is a member of Syska’s Critical Facilities Technical Leadership Committee.