Setting the Stage for Liquid-Cooled Solutions
June 30, 2006
Appeared in Energy & Power Management
Recent technology advancements by IT equipment manufacturers have resulted in heat-producing hardware that pushes the envelope of traditional air cooling methodologies. In response, various infrastructure vendors have introduced liquid-cooled solutions as viable alternative that can accommodate as much as 30 kilowatt (kW)-per-rack or more. Liquid-cooled hardware is defined as data center equipment in which the primary heat transfer medium is a liquid that exists internally within the electronics. These technologies and some to be introduced have different efficiencies and advantages.
First, with the exception of some uncommon specialized hardware systems (and surprisingly, laptops), for practical purposes, no liquid-cooled hardware is used in commercial data centers today. Some supercomputers and old legacy mainframes used liquids piped directly into hardware to cool heat-generating electronics. Some specialized military electronic packages use evaporative spray-cooling technologies, and some computer game manufacturers utilize true liquid-cooling designs.
The liquid-cooling solutions hitting today’s market still require air-to-liquid heat exchangers that provide air cooling at the individual rack where the electronic equipment is housed. The IT-hardware chassis still includes integral fans that move cool air through the inside of the hardware chassis to move the heat from the electronics to the air within the cabinet (or rack). Technically, this is still air-cooled hardware.
Defining the cooling scenarios typical of today’s datacom facilities and some baseline assumptions and limitations associated with this analysis makes apple-to-apple comparisons easier, especially if the goal is to focus on what (if any) energy savings can be attained using the various cooling options available. Obviously, different geographic locations will have different outside environmental conditions that can affect the potential efficiencies of heat rejection to atmosphere. Air-cooled chiller, dry-cooler, and cooling tower efficiencies are affected by the outdoor ambient conditions. Some locations afford the potential (at least during certain seasonal periods) for employing air or water-side economizers. For comparison, these parameters will be normalized to allow a better comparison of the different scenarios that exist within the data center proper.
The comparison includes five distinct scenarios involving current technologies and three additional scenarios that may become available in the future. The scenarios are:
- Conventional computer room air conditioners (CRACs)
- Rear door-mounted cabinet cooler
- Compressor-less liquid-cooled cabinet
- Water-cooled cabinet
- Conventional air-handling units
- Water-cooled IT hardware with chillers (future)
- Liquid-cooled hardware with chillers (future)
- Liquid-cooled hardware without chillers (future)
Parallel heat flow/rejection paths (e.g. some heat to pumped liquid, some to room) where a mixture of the above scenarios are employed are not considered within the context of this comparison.
Each scenario begins at the heat source (i.e., the IT hardware) and ends with the heat being rejected to the outside ambient environment (i.e., to atmosphere). Each breaks down the heat transfer path to incremental steps to include the medium (water, air, refrigerant, etc), the prime mover (fan, pump, compressor, etc), and the heat-transfer component (heat exchanger, fluid-cooler, chiller, cooling tower, etc.). Then, the scenario details total system energy efficiency for each scenario. Also for the sake of comparison, each scenario assumes the hypothetical facility employs recognized best practices such as minimized bypass air, balanced air and/or water distribution, and equipment properly maintained and operated per the manufacturer’s recommendations.
The energy calculations are based on assumptions and/or “givens” for a representative facility of 1 MW equipment input and cooling equipment selected according to current best practices to handle that total quantity of heat rejection:
- 1 megawatt (MW) of IT equipment input corresponds to approximately 285 tons of cooling.
- For air handling unit (AHU) systems assume a 20°F temperature delta with 25% extra air to account for leakage, stratification, etc (approx. 196,000 cfm total).
- For liquid-cooled cabinet systems assume 20°F temperature delta with no leakage (approx. 157,000 cfm total).
- For pumping systems with water as the medium, the specific gravity is assumed to be 1. For pumping systems with refrigerant as the medium, the specific gravity varies between 1.3 to 3.4, depending on which refrigerant is used.
- For AHU systems, fan efficiency is assumed to be 85% for 40,000 CFM fans at 0.3-in. external static. For CRAC unit fans, the fan efficiency is assumed to be 70% at 16,000 CFM; the pressure drop is assumed to be 0.3 in. For cabinet fans, the fan efficiency is assumed to be 40%; the pressure drop is assumed to be 0.3 in. Cooling tower fans are assumed to be 75% efficient at 0.3-in.static pressure.
- Chilled water piping systems are assumed to be designed at 12°F delta T. Condenser water systems are assumed to be sized for 3 gallons per minute/ton.
- Large pumps, those serving condenser or chilled water systems, are assumed to be 75% efficient. Small pumps, such as serving cooling distribution units and/or individual cabinets circuits, are assumed to be 50% efficient.
- Typical pump heads are 75–125 feet for chilled water systems and 40 feet for condenser water systems. For piping between heat exchangers and/or CDUs, assume 50 feet of head.
- Compressors for water-cooled chillers are assumed to be selected between 0.45–0.55 kW/ton.
- Air-cooled chillers and compressorized CRAC unit systems are not considered. Each has analogous systems that are considered below that have superior performance. Since the goal of this exercise is to identify which system can be more efficient, considering the systems that are clearly less efficient is not necessary.
- Energy usage is normalized to 1 MW of input at the IT equipment
Scenario 1 – Conventional Computer Room Air Conditioners (CRACs)
IT hardware chassis fans move the generated heat from the hardware chassis through the cabinet to the ambient room environment. CRAC fans pull the air across the cooling coil (heat exchanger), which absorbs the heat into a central chilled water system. The chilled water is pumped back to a chiller plant, which transfers the heat via a compressorized refrigerant cycle to either the outside environment directly (air-cooled chiller) or to a condenser-water system (water-cooled chiller). In the case of a water-cooled chiller, a pump moves the condenser water to a cooling tower (water-to-air heat exchanger), which rejects the heat to the outside environment. (air-cooled chillers are not considered here.)
Scenario 2 - Rear Door-Mounted Cabinet Cooler
IT hardware chassis fans move the generated heat from the hardware chassis through a cabinet rear door-mounted cooling coil (air-to-liquid heat exchanger). Note that this scenario uses the IT hardware chassis fans to move the heat through the rear door-mounted cooler. Ideally, the air leaving the cabinet cooling coil is at ambient room conditions. The heat absorbed by the liquid in the cooling coil is pumped back to the central chiller plant where the remainder of the heat transfer methodology matches that described above in scenario 1.
Scenario 3 - Compressor-less Liquid-Cooled Cabinet
IT hardware chassis fans move the generated heat from the hardware chassis. A cabinet mounted fan moves the heated air across a cabinet-mounted cooling coil that utilizes a refrigerant (air-to-liquid heat exchanger). The refrigerant leaving the coil can be either liquid or gas. The refrigerant is then cooled by a cooling distribution unit (that can support more than a single cabinet). The refrigerant leaves the cooling distribution unit as a liquid where it is pumped back to the cabinets. The cooling distribution unit incorporates a refrigerant-to-water heat exchanger. The water is typically chilled water and then pumped back to a central cooling plant and again proceeds as described in Scenario 1 above.
Scenario 4 - Water-Cooled Cabinet
IT hardware chassis fans move the generated heat from the hardware chassis to the air contained within a sealed cabinet. The sealed cabinet includes an integral fan-coil typically located in the bottom of the cabinet. The air within the cabinet is continuously recycled through the cabinet and the heat rejected from the IT hardware is absorbed by the integral fan-coil. Note that this scenario uses the IT hardware chassis fans to move the heat to the cabinet air and the cabinet fan-coil moves the cabinet air across the cooling coil. Ideally there is no air leaving the cabinet to the room ambient environment. The heat absorbed by the liquid in the cabinet fan-coil is pumped back to the central chiller plant where the remainder of the heat transfer methodology matches that described above in Scenario 1.
Scenario 5 - Conventional Air Handling Units
This scenario is essentially the same as that described in Scenario 1 except that fewer, larger air handling units replace the smaller CRACs.
The remaining three scenarios depict true water-cooled IT hardware that essentially does not currently exist in the commercial datacom repertoire. It is quite possible that as IT equipment heat densities continue to climb, IT manufacturers may soon introduce new product lines that incorporate actual liquid mediums in direct contact with the electronics to remove the heat from within the chassis to a point outside the equipment where the facilities utilities can transport the heat to the outside atmosphere. Currently, ASHRAE’s Technical Committee TC9.9, Mission Critical Facilities, Technology Spaces, and Electronic Equipment, is preparing a 4th book in the “ASHRAE Datacom Series” that will address the facilities’ interface to existing liquid-cooled cabinets and to some extent, possible liquid-cooled IT equipment of the future. This book, tentatively titled Liquid Cooling Design Considerations for Data and Communications Equipment Centers, is slated for publication as early as the summer of 2006.
Scenario 6 - Water-Cooled IT Hardware with Chillers
This scenario assumes introduction of IT hardware that employs a water-cooled heat sink (possible a cold-plate) that “utility” water is piped directly to. Note that there would be no chassis fans or cabinet fans and the IT hardware would be liquid-cooled by standard central plant chilled water.
Scenario 7 - Liquid-Cooled Hardware with Chillers
This scenario is similar to Scenario 6 except that the IT hardware would incorporate a liquid medium pump within the chassis and an associated pump to move the heat to a separate cooling distribution unit. The cooling distribution unit could support one or more liquid-cooled IT components and is essentially a liquid-to-water heat exchanger. Chilled water from the central cooling plant would cool the cooling distribution unit and the remainder of the heat transfer process again matches that described in Scenario 1.
Scenario 8 - Liquid-Cooled Hardware without Chillers
This final scenario is similar to Scenario 7 but eliminates the need for the central cooling plant and assumes the cooling distribution unit is cooled by water (or water-glycol mix), which is pumped outside to a fluid-cooler where fans reject the heat directly to the outside atmosphere.
Some general conclusions can be drawn from this analysis, as follows:
- In general, the fewer “steps” employed to move the heat from the heat generating source (IT equipment) to the outside atmosphere, the less energy required, and therefore the better the overall energy efficiency. One notable exception is the recognized inherent loss of efficiency of air-cooled chillers compared to water-cooled chillers in conjunction with cooling towers.
- Fewer, larger components within any particular “step” are typically more efficient than a greater quantity of smaller components providing the same function.
- The specific gravity of the liquid transport medium also plays an important role in the amount of energy consumed within each step. According to the fan equation: Brake horsepower = GPM x head (in feet) x specific gravity / (6350 x fan efficiency) It is clear that as specific gravity increases, the input power required to move the fluid increases proportionately. Water (specific gravity = 1) is a good transport medium. Refrigerants (specific gravity range of 1.3 to 3.4) is not as efficient a medium. Dielectric liquids (such as Fluorinert, specific gravity of 1.9) is not as good as water but is better than some refrigerants.
Some specific conclusions can be drawn from this analysis:
- AHU systems are inherently more efficient that CRAC systems because of the higher efficiency of the fans.
- Compressor-less liquid-cooled cabinets are the least efficient of the scenarios considered due in part to the larger number of heat transfer steps, and in part to the use of a refrigerant transport medium.
- Water-cooled cabinets are less efficient than conventional CRAC unit or AHU systems.
- A rear-door cooler can be as efficient (or even slightly more efficient) compared to an AHU system.
- Liquid and water-cooled equipment (of which none are yet commercially available) are not expected to be significantly more energy efficient than the other relatively efficient and available systems (i.e. AHU or rear-door coolers). However, due to other significant advantages, mainly that much higher load densities are achievable, these liquid and water-cooled equipment may still be found to be very desirable.
- Liquid-cooled equipment without a chiller plant (also not yet commercially available but technically very feasible) can offer significant energy savings compared to all other options. In addition, because of the system’s simplicity, it would be expected that this type of system would be easier to maintain and significantly more reliable than the other scenarios.
Power demand in to the equipment will always balance with the energy out in the form of heat. Newer technologies relate to the costs of transporting that heat from the IT equipment to the ambient surrounding the facility. Obviously, some techniques for transferring this heat are more energy efficient than others, and these need to be considered as part of the total cost of ownership of the enterprise and associated physical facility. Other issues relating to equipment reliability, maintainability, and especially the issues relating to the IT equipment configurations as they relate to the core function of the facility, must be considered completely before any decision is made in selecting the most appropriate cooling system for a data center facility.
About the Author: Vali Sorell has over 20 years of experience as an HVAC design engineer, with at least half of those years involved with design and commissioning of HVAC systems for critical facilities. As Syska’s National CFD Expert, he performs CFD analyses to compare and evaluate various system design options for high density equipment cooling configurations. In addition, he researches how to quantify the environmental conditions inside as well as outside cabinets as a function of cabinet configuration and cabinet load. Prior to joining Syska, he served as an in-house consultant to a co-location firm that develops and operates data centers in various cities nationwide.
Terry L. Rodgers, CPE and senior associate at Syska Hennessy Group has had over 25 years experience as a mechanical engineer. Previous to Syska, he was employed within the Nuclear Aerospace, Government, and Financial sectors of critical facilities. Currently at Syska Hennessy Group, Terry is a client leader and is slated to manage their Charlotte City office beginning in July of 2006. Terry is also a member of ASHRAE TC9.9 “Mission Critical Facilities, Technology Spaces, & Electronic Equipment”, participated on the ANSI Homeland Security Standards Panel working group for Enterprise Power Security, and is a member of Syska’s Critical Facilities Technical Leadership Committee.