Debunking the NYT feature on the wastefulness of data-centers

This weekend's NYT carried an alarming feature article on the gross wastefulness of the data-centers that host the world's racks of server hardware. James Glanz's feature, The Cloud Factory, painted a picture of grotesque waste and depraved indifference to the monetary and environmental costs of the "cloud," and suggested that the "dirty secret" was that there were better ways of doing things that the industry was indifferent to.

In a long rebuttal, Diego Doval, a computer scientist who previously served as CTO for Ning, Inc, takes apart the claims made in the Times piece, showing that they were unsubstantiated, out-of-date, unscientific, misleading, and pretty much wrong from top to bottom.

First off, an “average,” as any statistician will tell you, is a fairly meaningless number if you don’t include other values of the population (starting with the standard deviation). Not to mention that this kind of “explosive” claim should be backed up with a description of how the study was made. The only thing mentioned about the methodology is that they “sampled about 20,000 servers in about 70 large data centers spanning the commercial gamut: drug companies, military contractors, banks, media companies and government agencies.” Here’s the thing: Google alone has more than a million servers. Facebook, too, probably. Amazon, as well. They all do wildly different things with their servers, so extrapolating from “drug companies, military contractors, banks, media companies, and government agencies” to Google, or Facebook, or Amazon, is just not possible on the basis of just 20,000 servers on 70 data centers.

Not possible, that’s right. It would have been impossible (and people that know me know that I don’t use this word lightly) for McKinsey & Co. to do even a remotely accurate analysis of data center usage for the industry to create any kind of meaningful “average”. Why? Not only because gathering this data and analyzing it would have required many of the top minds in data center scaling (and they are not working at McKinsey), not only because Google, Facebook, Amazon, Apple, would have not given McKinsey this information, not only because the information, even if it was given to McKinsey, would have been in wildly different scales and contexts, which is an important point.

Even if you get past all of these seemingly insurmountable problems through an act of sheer magic, you end up with another problem altogether: server power is not just about “performing computations”. If you want to simplify a bit, there’s at least four main axis you could consider for scaling: computation proper (e.g. adding 2+2), storage (e.g. saving “4″ to disk, or reading it from disk), networking (e.g. sending the “4″ from one computer to the next) and memory usage (e.g. storing the “4″ in RAM). This is an over-simplification because today you could, for example, split up “storage” into “flash-based” and “magnetic” storage since they are so different in their characteristics and power consumption, just like we separate RAM from persistent storage, but we’ll leave it at four. Anyway, these four parameters lead to different load profiles for different systems.

a lot of lead bullets: a response to the new york times article on data center efficiency (via Making Light)


  1. It’s still okay to hate the “cloud” as a buzzword and not think that the cloud is the answer to every IT problem, right?

  2. The electricity pouring into the computers was overheating Ethernet sockets

    ….really?  I’d expect processors and hard drives to grind to a halt long before the heat of the ethernet sockets was a concern.

    I’d love to see this “wasteful” problem compared with an equivalent amount of home computers, in terms of the actual use made of the systems balanced against the power and cooling they use.  Lots of datacenters have “green” cooling and power acquisition measures that go far beyond anything cost effective for home users.

  3. I don’t believe that Doval ‘debunked’ anything; that seems to be a bit of a mischaracterization.  His critique of the Time’s use of ‘average’ or mean (in lieu of mean, median, mode and standard deviation?) seems like a real stretch.  I mean, come on, the Times looked at 20k servers at 70 data centers for a newspaper article with rather modest [and wholly accurate] claims that there are inefficiencies and waste inherent in this industry/sector.

    Doval’s demands that the Times examine tens (hundreds?) of thousands of additional servers of wider variety comes across as a knit-picky overreach, in my opinion.  His discussion of the different types of processes and associated energy usage in which the servers are engaged is rather irrelevant to the core claim of the article: There’s unnecessary waste in dem there sky servers.  

    I would also note that the Times article is not presenting itself as a scientific/academic paper with fine grained detail, nor was it intended to be comprehensive.  I find Doval’s treatment of the piece to be rather defensive, actually. 

    1. The tone and verbiage of the Times article were somewhere between ‘dreadful’ and ‘overt yellow journalism’. As for the demand that more servers be surveyed; I think that there is actually an important point there: the problem, when coming up with datacenter efficiency numbers, is that the people most willing to talk, or easy to measure from other data, are the ones who are worst at efficiency. 

      The less expertise you have access to, as with smaller operations, (or the greater the value of reliability compared to electricity costs, as with banking or other the-computers-must-not-go-down types of industries) the lousier your efficiency is likely to be; but the more likely it is that you might be willing to talk, at least in round numbers, about what you are up to or how you’d like to be spending less on it.

      If you are a large player, Google, Amazon, Facebook, those sorts, server efficiency is one of your major competitive advantages; but, for that reason, you are unlikely to say anything terribly useful, aside from the occasional carefully worded boast. That’s the real problem with the Times sample: It isn’t that it’s small, it’s that it isn’t obviously representative.The ‘inefficiency secrets revealed! tone also struck me as somewhat odd because, if you look at the sales pitches from either datacenter equipment vendors, or the occasional carefully-handled glimpse of one of the very large operations, it’s all ‘efficiency, efficiency, efficiency’. Intel says you just have to see the performance/watt on their new xeons, Vmware says you can consolidate your server workloads by 70%, APC has a new rack with better hot aisle/cold aisle separation…It certainly strikes me that one could ask some rather pointed questions about how much energy we use so that our cat videos can be delivered more swiftly, and other banal or downright sinister applications of information technology can keep grinding along; but the incentives of those doing the grinding(especially if they have to do it on ad-revenue margins, or on a massive scale, or both) are all in the direction of efficiency. We certainly burn hideous amounts of energy on frivolous bullshit; but the race is on to deliver ever more frivolous bullshit per watt…

  4. But … but … but it’s McKinsey!

    Having dealt with a few people from McKinsey, I’d modify the old saying to be “Those that can, do.  Those that can’t work for McKinsey.”  Which might still be giving them too much credit.

    1. Like dioptase, I have had access to several confidential McKinsey presentations and “reports” over the years. In each case an intern could have produced analysis of the same or better quality. But I will give them one thing: they know how to make a PPT look great!

      1. I thought they outsourced making the presentation to India years ago.  At least I remember an article about his and lauding it as “freeing time for the actual work”. 

        1. Well then, I retract that bit about good PPTs. Hang on… then that means McKinsey doesn’t do anything worthwhile… Who would have guessed?

  5. That being said, is there a comprehensive. statistically significant result that we can draw from on this subject?

  6. I do not understand why the Times felt they had to contract with McKinsey — for a cost probably greater than that of a couple junior reporters’ salaries — to call up some companies, ask for some data on background, and then analyze that data.

    1. Yes, it’s a mystery to me, too. It’s not as though McKinsey is well known for its analysis of data centres. 

    2.  Odds are it’s part of a marketing campaign/strategy to create the appearance of a problem to which they will have a solution.  I tend to think it’s all one big hunk of astroturf.

      1. McKinsey always has a solution. Moreso when there is no problem. I like the astroturf idea. I hadn’t thought of it that way. 

    3. McKinsey is a very odd choice, over such obvious better options as Gartner or a big IT consultancy like HP or IBM.
      This series of articles in the Times has been really, really disappointing.  Completely argued from anecdote, full of naive ideas like the idea that servers should run near 100% at all times (with no mention of the obvious fact that the major challenge is building infrastructure capable of handling PEAKS), and anything less represents “waste,” and foolhardy insinuations like the idea that data centers typically run on dirty diesel generators (vs. having the generators there AS BACKUPS just like they have generators at hospitals and factories).  

      It really feels like the Times came up with a thesis – “the internet is a wasteful, dirty threat to the environment” and subsequently sought to piece together anecdotes to support the thesis.  

      1. Even more annoying than the articles, however, is the naive comments posted to them.  Hundreds of posts like “I had no idea!  I am going to delete my extra emails and stop using Facebook during peak hours to help stop global warming!”

  7. Totally agree with Lance that this is not a compelling criticism of the NYT article.

    I’m professionally employed as a statistician (more specifically a Research Scientist focusing on data). I would never say that an average is a “fairly meaningless number” in the absence of more data. In fact, it’s often a very useful statistic. 

    There are echoes of climate change skepticism in this rebuttal, in that Doval appears to demand artificially high levels of evidence in order for data centers to be called “wasteful.” It’s a bit silly – data centers are certainly wasteful by the second law of thermodynamics, but are they more wasteful than they should be? I’m not sure from reading either piece, but I’m much more convinced by the NYT article than the rebuttal.

  8. I didn’t read the rebuttal, but then again I didn’t have to. The NYT article is pretty obvious with its bunk. Oversimplifying the technology, anecdotal evidence, glossing over major technology gaps, and refusing to acknowledge that things are steadily improving due to new technologies.

    Companies want to pay less in energy costs, and that reflects the way the industry designs enterprise severs. Blades now run 70 degree Celsius outlet temperatures and rack mount are running on 45 degree inlet temperatures as a standard procedure for both HP and Dell. Reducing power consumption and more efficient cooling are #2 to actual processing power (or storage capacity, another thing the article conveniently skips over).

    Silicone is now designed for higher temperature operation and lower power operations except for very targeted computing tasks in specialized data centers, and HD technologies are pushing their material’s limits to live in a warmer environment. Hell, even data centers are changing to a recycling nature to improve efficiency, eBay is using the new HP technologies to use (relatively) low power self-contained data centers that are more like trailers than warehouses. This isn’t something people are doing for fun, the market demands costs cuts and the technology is being delivered. A few years ago data centers needed a 35 degree inlet constantly and had to ramp fans to meet heavy loads. Oh yes, and that’s another component – both data centers A/C and the server’s fans require power. If you rely too much on either your power costs go through the roof.


    Turns out the rebuttal is pretty good and covers all the same issue that struck me with the article. Even the averages is a good point from the perspective of what data there was to make that statement from.

  9. I was going to post something on the NYT comment section, but the article was too silly.  The total wattage of Google divided by daily users came to about 4 watts.  Granted that could be 1 watt, but we’re obsessing over 2 percent of the nation’s power load.

    1. That 2% figure was what caught my eye and is probably the more important. It seems large to me rather than small. I’d find it very interesting to know what proportion of electricity usage is computing (from front to back, from datacentre to laptop charger), and perhaps more importantly how fast its growing.

      I’d expect the big players who run their own datacentres to be pretty efficient. I’d also expect datacentres offering racks and hosting for rent to be fairly inefficient. As long as each rack doesn’t go over it’s allocation, the individual customers probably don’t care too much that their kit is a bit old and a bit underused for the power it’s consuming.

      I’m also not sure there was much comment in the research about the A/C. I’d guess that most datacentres just dump all that waste heat with no attempt to make use of it. And that a lot of them are located in places with a relatively hot climate.

      1. A cloud provider like EC2 typically supports companies that want to scale quickly if their market expands, during an initial phase of growth with low investment.  The investment in hosting is low, but so is the investment in optimization, so the whole software/hardware stack gets terrible users per server unless it grows to the point where optimization is worthwhile.

        The latter phase is where users move to their own datacenters; it’s more like the typical in-house model except running the same cloud infrastructure (eg Rightscale).  The software can be tuned then as well, to get much higher usage per server.  

        The growth-before-investment model can be wasteful due to the postponement of many efficiency measures, but it has built-in incentives to improve, and it’s better than what came before it, both technically and economically.  

    2. 2% is the preferred number for all these kinds of articles:

      – The airline industry emits 2% of global greenhouse gases. 
      – The ICT industry is responsible for 2% of global CO2. 
      – US emissions declined by 2% last year. 
      – The US dairy industry makes up 2% of emissions. 
      – The US has 2% of all the oil reserves globally. 
      – China’s energy use was down by 2% in 2011. 
      – Using broadband technologies could cut energy use by 2%

  10. If your rate of increased use is faster than your rate of efficiency gains, then you’re still consuming more and more electricity. 

    So the efficiency gains by some companies are good and smart (also for their bottom line), but doesn’t solve the fundamental problem of increased energy use.  To do that, tech companies should invest in, or commit to, purchasing renewable energy – stimulate that market to grow fast enough to meet the data center demands.  Some big companies are already doing this, and since they’re big customers (and growing in terms of energy use), they’ve got real clout.

  11. I found the part about “10% of the time doing calculations” in the NYT article pretty bogus. Netflix’s servers, for example, would very likely be IO bound. The CPU load isn’t necessarily an indication of the system utilization.

    As an example, I ran on my Mac. I got 57Mbit/s download (comcast) but the CPU load barely reached 18% on one core.

    1. There are also things like the memcached (and analogous systems) servers of the world that generally don’t lean too hard on their CPUs; but which can save the world from an amazing number of thrashing HDDs under the right circumstances.

      It won’t even show up on your ‘% of time doing calculations’ numbers; but if you replace a rack full of 15k RPM screamers with SSDs or lots of RAM, your power bill will probably thank you.

  12. Seems like an opportunity for learning some lessons from cogeneration power plants. You have lots of spare heat? Can’t think of an easy way to get rid of it? Try doing something useful with it; odds are high somebody else might want some heat. What about those hot-tubs? Or maybe a large scale sous-vide cooking operation? How about keeping the National Plasticine Stock warm so it is easily mouldable?

  13. If this was an article slamming print and the wastefulness of pulp publishing, I think BoingBoing would love it. The main point is, none of this stuff is “free,” ecologically. If we burn out the planet driving servers or presses it won’t matter to the planet. 

    1.  Here, here. Let’s not lose sight of the fact that data centers will double their cumulative C02 emissions in ten years, and most of these centers are powered by coal. The internet and the cloud is dirty, regardless of what we think of the Times article. People need to come to grips with the fact that tech’s image of purity doesn’t correlate with the reality of tech’s toxic ways. Doesn’t mean we have to be anti-tech, it just means we need to do better.

Comments are closed.