Free Cooling: the Server Side of the Story
by Johan De Gelas on February 11, 2014 7:00 AM EST- Posted in
- Cloud Computing
- IT Computing
- Intel
- Xeon
- Ivy Bridge EP
- Supermicro
Fan Speed
How do these higher temperatures affect the fans?
It is clear that the fan speed algorithm takes more than just the CPU temperature and inlet temperature into account. There's definitely an ability to detect when a low power CPU with low tCase is used. As a result the fans are spinning faster with the 2650L than with the 2697 v2. That also means that the server has more headroom for the Xeon E5-2697 v2 than we first assumed based on the CPU temperature results. At higher inlet temperatures, the fans can still go a bit faster if necessary on the 2697, as the maximum fan RPM is 7000.
Power Measurements
The big question of course is how all this affects the power bill. It's no use saving on cooling if your server simply consumes a lot more power due to increased fan speeds (and potentially down time when replacing fans more frequently).
The difference in power consumed is not large between the three inlet temperatures. To make our measurements clear, we standarized on the measurements at 20°C as the baseline and created the following table:
| Xeon E5-2697 v2 | Xeon E5-2650L | |||||
|---|---|---|---|---|---|---|
| CPU load | Inlet 20°C | Inlet 35°C | Inlet 40°C | Inlet 20°C | Inlet 35°C | Inlet 40°C |
| 0-10 | 100% | 105% | 106% | 100% | 106% | 112% |
| 10-20 | 100% | 98% | 103% | 100% | 104% | 108% |
| 20-30 | 100% | 105% | 110% | 100% | 103% | 107% |
| 30-40 | 100% | 102% | 105% | 100% | 102% | 105% |
| 40-50 | 100% | 109% | 108% | 100% | 97% | 109% |
| 50-60 | 100% | 105% | 108% | 100% | 108% | 111% |
| 60-70 | 100% | 106% | 107% | 100% | 104% | 110% |
| 70-80 | 100% | 106% | 104% | 100% | 105% | 109% |
| 80-90 | N/A | N/A | N/A | 100% | 109% | 108% |
| Average | 105% | 107% | 103% | 109% | ||
As the fans work quite a bit harder to keep the 2650L below the low Tcase, they need a lot more power. We notice a 9% increase in power when the inlet temperature doubles. The increase is smaller with the Xeon E5, only 7%.
The most interesting conclusion is that raising the inlet temperature from 20 to 35°C results in almost no increase in power consumption (3-5%) on the server side, while the savings on cooling and ventilation can be substantial, around 40% or more.
Facebook
Twitter
RSS
48 Comments
View All Comments
extide - Tuesday, February 11, 2014 - link
Yeah there is a lot of movement in this these days, but the hard part of doing this is at the low voltages used in servers <=24v, you need a massive amount of current to feed several racks of servers, so you need massive power bars and of course you can lose a lot of efficiency on that side as well.drexnx - Tuesday, February 11, 2014 - link
afaik, the Delta DC stuff is all 48v, so a lot of the old telecom CO stuff is already tailor-made for use there.but yes, you get to see some pretty amazing buswork as a result!
Ikefu - Tuesday, February 11, 2014 - link
Microsoft is building a massive data center in my home state just outside Cheyenne, WY. I wonder why more companies haven't done this yet? Its very dry and days above 90F are few and far between in the summer. Seems like an easy cooling solution versus all the data centers in places like Dallas.rrinker - Tuesday, February 11, 2014 - link
Building in the cooler climes is great - but you also need the networking infrastructure to support said big data center. Heck for free cooling, build the data centers in the far frozen reaches of Northern Canada, or in Antarctica. Only, how will you get the data to the data center?Ikefu - Tuesday, February 11, 2014 - link
Its actually right along the I-80 corridor that connects Chicago and San Francisco. Several major backbones run along that route and its why many mega data centers in Iowa are also built along I-80. Microsoft and the NCAR Yellowstone super computer are there so the large pipe is definitely accessible.darking - Tuesday, February 11, 2014 - link
We've used free cooling in our small datacenter since 2007. Its very effective from september to april here in Denmark.beginner99 - Tuesday, February 11, 2014 - link
That map from Europe is certainly plain wrong. Especially in Spain btu also Greece and italy easily have some day above 35. It also happens couple of days per year were I live, a lot more north than any of those.ShieTar - Thursday, February 13, 2014 - link
Do you really get 35°C, in the shade, outside, for more than 260 hours a year? I'm sure it happens for a few hours a day in the two hottest months, but the map does cap out at 8500 out of 8760 hours.juhatus - Tuesday, February 11, 2014 - link
What about wear&tear at running the equipment at hotter temperatures? I remember seeing the chart where higher temperature = shorter life span. I would imagine the OEM's have engineered a bit over this and warranties aside, it should be basic physics?zodiacfml - Wednesday, February 12, 2014 - link
You just need constant temperature and equipment that works at that temperature. Wear and tear happens significantly at temperature changes.