Goodhart's Law: Once you measure something, it changes

Goodhart's Law is one of those neat formulations that codifies something I've been trying to put my finger on for years: "once a social or economic indicator or other surrogate measure is made a target for the purpose of conducting social or economic policy, then it will lose the information content that would qualify it to play such a role."

That is, once you start measuring GDP as a way of gauging social welfare, people will start to figure out ways to make GDP go up without improving social welfare (say, by swapping dirty financial derivatives). Once Google starts measuring inbound links as a way of evaluating the importance of web-pages, people will figure out how to increase the inbound links to unimportant pages (splogging, blogspam). And once you measure fat or calorie content as a proxy for the healthfulness of food, manufacturers will figure out how to decrease fat and calories without making the food more healthful (reducing fat by adding sugar, reducing calories by adding poisonous artificial sweeteners).

The law was first stated in a 1975 paper by Goodhart and gained popularity in the context of the attempt by the United Kingdom government of Margaret Thatcher to conduct monetary policy on the basis of targets for broad and narrow money, but the idea is considerably older. It is implicit in the economic idea of rational expectations. While it originated in the context of market responses the Law has profound implications for the selection of high-level targets in organisations.
Goodhart's law (Thanks, Steve!)


  1. Please consider that just because it has the word law in it, doesn’t mean it is universally true. Good indicators don’t lose all value just because they can be gamed.

    1. True, but indicators that can be “gamed” (nice wording, by the way)may attract more attention and thus bias the resources and efforts. My fave example would be gmat scores and how they pervert educational focus and have led to an entire industry around the gmat test. The test does not guarantee that the person responding to it understands math

  2. That would explain why “Measure twice, cut once” doesn’t work for me.

    Actually, interesting article. Human nature, ain’t it grand?

  3. This theory reminds me of a Douglas Adams Quote…

    “There is a theory which states that if ever anyone discovers exactly what the Universe is for and why it is here, it will instantly disappear and be replaced by something even more bizarre and inexplicable.”

  4. Goodhart’s law was essentially stated in Heisenberg’s uncertainty principle 50 years earlier than Goodhart. Look it up. Apology accepted.

  5. People trade dirty derivatives in order to fool social welfare surveys? And all this time I thought it was a money making scheme…

  6. Sadly, there is no mention of the solution: StealthStatistics !

    The future:
    Government announces everyone is very rich and content – Denies revealing what they measured;
    Opposition says everyone is very – Denies revealing what;
    Interwebs says – Denies revealing;

  7. At least for economics, this implies that GDP in fact measures anything (useful) at all. It may be ture that when a government focuses on the GDP, it will start to game numbers to make the GDP look better. But at least part of the reason that it’s so easy to game the GDP is that it really isn’t a very good measure of economic health in the first place. GDP is an attempt to boil a concept, domesitic production, into one number as if such a resulting number were even close to an accurate portrayal of the nearly inifinete microeconomic factors that actually constitute the state of the economy.

  8. This doesn’t track for me. It assumes (for example) that the point of selling snack food is to poison your customers, rather than to keep your margin up and make money. They aren’t switching to artificial sweeteners and fats (Olestra?) because their customer base has caught on to their previous method of toxin delivery and they need to find a new way to kill them, they’re doing it because their customers are demanding it.

    Maybe it’s just the way Cory describes it I have trouble with, but since that’s the crux of the story, I’ll say I have trouble with the story.

  9. “poisonous artificial sweeteners”? C’mon, let’s get some science in here. No matter how many forwarded e-mails you get from anti-artificial sweetener activists and the suckers that fall for them, there’s still no evidence that nutrasweet & friends are bad for you.

    You know what IS bad? Diabetes. Kills millions. But it’s hip to hate the sweeteners that save lives because they’re not ‘natural’.

  10. How true this is. In the UK health service, the gov’t announced a target that arriving patients “would be seen by a nurse or doctor within thirty minutes”.
    So now they employ a nurse known as a “greeter”. You arrive. She greets you, asks after the problem, and shows you to a seat in the waiting room. You may still wait for hours … but you have “been seen by a nurse or doctor within thirty minutes”. Target met, but now meaningless.

  11. This law is especially true in the corporate world, where managers’ bonuses are tied to ‘measurable’ performance ratings. Everyone, EVERYONE at my old job worked to their metrics with near perfect blindness to the big picture.

  12. Yeah, I strongly doubt that Cory’s interpretation of Goodhart’s Law is particularly close to what Goodhart intended. The one citation in the very short Wikipedia article he links to contains an obscure journal article that seems to focus entirely on British monetary policy.

    Cory may be on to something, but it ain’t Goodhart’s Law.

    (Also, the article is nine years old. And Goodhart’s Law is even older. Maybe a slow blogging day?)

  13. So what you are saying is the Fed by dropping the M3 measurement five years ago before it really started to show the inflation was a cool quantum way to keep it floating in a Schrödinger’s Cat like fashion. Genius, now only by everyone noticing that they lost their house and starving do we find out the box has a dead cat in it. Closing eyes and disbelieve, disbelieve, disbelieve!

  14. I love the fact that I open my morning by checking my Twitter feed, see a tweet by Cory, linking to Craphound, go to, read the article, see a comment by Steve Song, and then check boingboing to find an entry based on that.

    It’s like I’ve entered some heightened state of internet-causal-knowledge.

  15. Another way of looking at this is that people will game the system to get what they want. You could argue that the point of high school is to learn something, but most people treat it as a vehicle to get into college. So, the real “outcome” of high school is a GPA or SAT score, not an education.

  16. Interesting concept, which I find possibly suspect, if only for the assertion that base assumptions (or hypotheses)to explain an observation are automatically given credence as fact, and with a truly causal link that is not established;

    for example, in the example given:

    “And once you measure fat or calorie content as a proxy for the healthfulness of food, manufacturers will figure out how to decrease fat and calories without making the food more healthful (reducing fat by adding sugar, reducing calories by adding poisonous artificial sweeteners).”

    A- it is first asserted as Fact that fat and caloric content is a causal factor in unhealthy food otems

    B- manufacturers respond by reducing fat, but by adding sugar, yet the food item is no more healthful.

    Yet, in reality, A is not established as a fully valid explanation (ie. by some theories, it’s not given that fat per se is bad for you, it’s the presence of glucose as the primary energy source in one’s diet (glycogen response) that results in storage of the fat content in food, which can lead to obesity and associated health problems). Therefore, B can be seen as a result of the human *response to* A, without being mechanistically related to A whatsoever. Put another way- people may eat fat, say in avocadoes, and remain fully healthy, but if they eat the avocado habitually in combination with a pint of coca cola, may likely gain weight. Attempting to solve the problem by responding to the avocado, rather than the coca cola, merely exacerbates the problem, which of course still remains- as the coca cola was never recognized as the chief causal element in the observed result.

  17. It’s not really the act of measuring that changes it, so much as the reporting of the measurement techniques after the fact. If you measured it but never told anyone about it, then it would change nothing.

    My city has two bridges leading into downtown. If one is busy, the radio stations tell everyone to use the other one. Then the other one gets busy. But if the radio stations kept their mouths shut then I could take the old bridge like I always do and there wouldn’t be any problems.

  18. I take issue with the term “poisonous artificial sweeteners.” Anything is poisonous or toxic if you consume too much of it, including oxygen, including water. Artificial sweeteners have been subject to highly stringent safety and toxicological tests, and have been approved for use in foods by the appropriate safety bodies (EFSA if you’re in Europe and the FDA in the US). They’re perfectly safe in the doses consumed in soft drinks and other uses, with a large margin of error. I’m sick of seeing bad press about anything “artificial” in foods.

  19. Isn’t this law just an obvious extrapolation of Heisenberg’s uncertainity principle ? Or what in social sciences is called “Enlightenment effect”.


  20. I guess Goodhart hadn’t heard of Heisenberg’s Principle. “Once you measure something, it changes” is a pretty basic aspect of quantum theory from the nineteen-thirties.

  21. “Wherever there’s a system, there’s a racket to beat it.” – Jerome Ravetz , Scientific Knowledge and its Social Problems, Oxford 1971

  22. “poisonous artificial sweeteners”

    [Citation Needed]

    Is there any evidence that they’re poisonous, or is it just that they

    a) are sweet, and everything that tastes good must be bad for you.

    b) are chemicals made via industrial processes, so they’re full of *toxins.* Remember folks, everything that western society makes is bad for you. If it’s not organic and natural, it’s a *toxin.*

    c) allow you to get something without having to pay for it, which is immoral. All pleasure must be accompanied by an equal dose of suffering.

    1. Ironically resulting in chemically pure substances (one ingredient) being called ‘toxic’ when ‘natural extracts’ with the same *active* ingredient are full of all kinds of crap (sometimes literally)

      ie. Aspirin vs. willow bark tea

    1. I guess the ones concerned will try to maxismise the GNH at least cost to them. If the GNH measurement consists of a yearly ‘Are you happy?’ census, the easiest way might be to give everyone a small present on the day of the census. Sure lifts spirits.
      Actually the GNH in Bhutan seems to be subjectively defined, making it a StealthStatistic and therefore impervious to Goodhart’s Law (at the cost of being BS)

  23. “And once you measure fat or calorie content as a proxy for the healthfulness of food, manufacturers will figure out how to decrease fat and calories without making the food more healthful (reducing fat by adding sugar, reducing calories by adding poisonous artificial sweeteners).”

    This is true, however, I don’t think making food unhealthy is the goal of the food manufacturers. Their goal is to make the food tasty, addictively so if possible, cheap to produce, etc.

    If they could make a food that was cheap to produce, crazily tasty so that you eat a whole bunch of it, but also healthy, there is no doubt they would actually do it. It just so happens that doing this is incredibly difficult. It’s like how when you make software, you can make it quickly, cheaply, or with quality, and you only get to pick two. You can’t have all three. Food is the same way. You can have it be healthy, tasty, or cheap. Pick two. Nobody has yet figured out how to get all three.

    The thing is, if you’re in the food business, and you want to profit. Well, it has to be cheap. If it’s expensive, profit is down. It has to be tasty. You can’t trick people into thinking poop tastes like fois gras. Healthy, however, you can replace with marketing.

    The only real solution I see is to invest in food science, and to figure out how to break the rule, and make tasty, healthy, and cheap food that breaks all the rules. Imagine eating a whole bag of potato chips that are crazy yummy, and it’s good for you!

    1. You’re quite right in saying that food can’t be tasty healthy and cheap all at one. Things that inherently make food taste good, like salt, fat and sugar, are the things that are bad for you (if you eat a lot of them), so it makes creating low calorie food that tastes good is very difficult. You’ve probably noticed how low fat doesn’t necessarily mean low sugar, it’s because there’s got to be something that replaces the qualities in the food which the fat provides, be it mouthfeel, flavour, whatever. (But that’s a whole books worth of discussion)

      1. the thing there is that as long as we where hunter/gatherers, those food elements where harder to come by. But as we “gamed” the system by going farmers, and then scaling that to industrial levels, what was once rare, and so we became genetically conditioned to eat as much as we could when found, now became abundant. End result the body is eating as ordered, but rather then there being 1/1000 meals, its maybe 1/10 or even 1/1…

  24. A Swedish political commentator wrote it in even fewer words slash general terms some fifteen years ago when he was trying to nail a truth about technological, social and economical frontline developments. That made a similar impact on me, hence I remember it well, roughly it would translate late this:

    Once it becomes possible to apply understandable exact statistics on important factors of frontline progress they are no longer really significant (frontline factors).

  25. As an educator I find that this is a huge problem in education. We have become so focused on exams, assignments, grades, and degrees as proxy measures of learning that actual learning gets neglected. Students routinely game the system in order to get better grades than they really deserve — e.g. through cheating, trying to talk their instructor into raising their grade, or even having their parents sue the school if their precious snowflake gets a bad grade. Some people even go so far as to purchase degrees they don’t really deserve from so-called “diploma mills”. Educators also game the system in various ways — e.g. grade inflation, teaching to the test, grading on a curve, etc.

    I have actually been told by my superiors that I give too many “A”s in freshman-level courses. How did they make this determination? Did they test my students to see how well they had learned the material I taught them? No. They just compared the grades I gave to the distribution of grades they would expect to see in a typical freshman-level course, and noticed that I awarded an above-average number of “A”s. That’s how ridiculous things have gotten in the world of education: The numbers that are used to evaluate how well students are learning have become more important than the actual students or the material they are supposed to be learning.

    The simple truth of the matter is that I do award lots of “A”s. In fact, I’d be happy if every single one of my students earned an “A”. And I let them all know on the very first day of class that each of them can earn an “A” if they’re willing to work for it and not cheat. That’s because I care about learning, not about grades. I wish I didn’t even have to assign grades at all. I believe in learning for learning’s sake, not merely for the sake of earning some token of accomplishment that you can show to potential future employers as evidence of your worthiness to be hired. And I feel that students learn better when they don’t have to feel stressed out about grades. Sure, if a student cheats, he or she deserves a failing grade. And I have little sympathy for students who don’t care about the class, don’t show up for lecture half the time, don’t do the assigned work, and come up with some lame excuse to try to justify their poor performance. But as long as a student really wants to learn, and is willing to put in the effort, I’m going to try my best to help that student learn, and am going to cut him or her some slack on grading. Obviously, if a student just doesn’t get the material, then I can’t really give him or her an “A”; but as long as the student can demonstrate that he or she understands the material that I have taught, I am happy to award an “A”, even if his or her performance on exams and assignments is less than perfect. I feel that my job as an instructor is to help my students learn, not to evaluate their overall intellect, their writing skills, or their potential worth to future employers.

    I really wish that we would do away with grading in education altogether. Instead, instructors should simply evaluate a student’s learning as being either: satisfactory, unsatisfactory (the course must be repeated), incomplete (the student must do more work in order for his or her performance to be considered satisfactory), or failure by virtue of academic misconduct (which would result in disciplinary action). After all of their coursework is complete, students should then be required to take a battery of qualifying exams, covering all of the material they have studied, in order to get their degree. But these exams would all be evaluated on a pass/fail basis, and students should be allowed to retake them as many times as it takes in order to pass (and should also be allowed to take short refresher courses in preparation for these exams). Regardless of how this is done in practice, something has to be done to get the focus of education back on learning rather than on grades.

  26. Aspartame causes cancer in rats.

    If it causes cancer in rats, that a good enough reason for me to avoid it. And you know what? Fuck artificial sweeteners on the whole. If I want something sweet, I’ll eat real sugar, honey, maple syrup, agave, wine reductions, fruit, grains etc. Artificial sweeteners are a purely capitalistic invention. A solution to a problem than never existed, one that was was invented to make money off of people’s bad habits and lack of education. The problem isn’t with sugar, the problem is that people misuse and over use it. Teach them to eat properly and there wouldn’t be a problem to begin with. But that’s not lucrative enough.

    1. I’m not a toxicologist, so I can’t really comment on that study, but I can make a couple of points.

      1) Aspartame has been approved for use in foods for more than 20 years, and both before and after it’s approval, there have been numerous studies conducted into it’s safety (I think the total is in the several hundreds). Every single one of these studies has been considered by the regulatory bodies (see my first post) when deciding whether or not it is safe for use in foods. The fact that aspartame has been approves for use in foods means that experts have looked at the evidence for and against it’s safety, and decided that it is safe. One or two studies amongst several hundred that conclude that it is carcinogenic doesn’t mean it is.

      2) It’s incredibly difficult to compare results found in rats to expected results in humans. Aspartame has been consumed by hundreds of millions of people over the last 20 years, which represents billions of man-years of safe exposure.

      You might want to look at the following study: “Aspartame: Review of Safety” in Regulatory Toxicology and Pharmacology 35, S1–S93 (2002)

      1. With all the crap we eat, drink, breathe, put in contact with our skin, and mucus membranes, I imagine it would be difficult to isolate the effects of something like aspartame in humans. But in rats, who are studied in a completely controlled environment, it is quite easy. As I said, if it leads to cancer in rats, thats a bad enough sign for me. Honey doesn’t give rats cancer. You can keep your aspartame. Snake oil, even if it’s non toxic, is still snake oil. And you should be ashamed of yourselves for peddling it.

  27. tasting the batter changes it a little.

    tasting the iced cake changes it enough to get you into trouble esp if it’s not your birthday.

  28. Short version: ‘people will try to game the system’.

    It’s nice and all to try to make it a Heisenbergian sounding statement, but it’s not physics, it’s ethics.

  29. policies cannot make incentives go away. if people perceive that some benefit comes from an activity, they will engage in that activity, or engage in a suitable replacement. sugar tastes good. sex and drugs feel good. money and success are desirable.

    very few policies have ever been truly successful. most of them miss their targets, causing proponents to call for more policies. when policies hit their targets, it’s a sign that people have found ways around them.

  30. The “reducing calories by adding poisonous artificial sweeteners” comment is really unfortunate in an otherwise intelligent post.

    There’s a number of reasons why people might want to cut back or eliminate their sugar consumption (diabetes, to start with). Artificial sweeteners attempt to reproduce the same response as sugar from the taste buds… seems to me like a perfectly valid goal for food science.

    Also, measuring your sugar intake by counting the calories in high sugar content products (like candy, soda, etc.) is perfectly fine… since there’s no way to cut back on the calories while using the same amount of sugar. Same with fats, carbs, etc. Nothing like cheating on the GDP.

  31. Look at the national education legislation, No Child Left Behind. It’s obvious what is happening, but conservative ideologues can’t see past their common-sense solutions. The solution for them is adding more measurement systems that are supposed to correct the natural adjustments humans make to adapt to them. The outcome is likely to be circular.

  32. The “reducing calories by adding poisonous artificial sweeteners” comment is really unfortunate in an otherwise intelligent post.

    There’s a number of reasons why people might want to cut back or eliminate their sugar consumption (diabetes, to start with). Artificial sweeteners attempt to reproduce the same response as sugar from the taste buds… seems to me like a perfectly valid goal for food science.

    Also, measuring your sugar intake by counting the calories in high sugar content products (like candy, soda, etc.) is perfectly fine… since there’s no way to cut back on the calories while using the same amount of sugar. Same with fats, carbs, etc. Nothing like cheating on the GDP.

  33. It’s simply another way of looking at the Bad Benchmarks are Bollocks problem.

    The core of the problem is that it is very difficult to devise benchmark tests that actually provide a meaningful measure of anything other than really trivial; sure 0-60mph time of a sports car is easy to establish, but does it tell you anything actually useful about the real world usefulness? Similarly the GHZ rating of a cpu is close to meaningless, and you rapidly go off the rails when you get to anything as complex as tests relating to the education of children.

    Since it is difficult, most people don’t actually try. They settle for a stupid benchmark. They pretend it has meaning. They make teachers teach children based purely on how well they will perform under this pointless benchmark. Software and computer systems designs are distorted to improve their score. Wars are fought based on stupid simplifications.

    A car’s value gets reduced to a single, pointless number. A child’s value gets reduced to a single pointless number. A school’s quality is defined by its ranking in a pointless table.
    Stupid. And I have a pointless single number that proves it :-)

      1. thats also the problem of observing remote communities or wild animals, one risk disturbing their behavior and so spoiling the observations.

    1. @romusnr and @jeremyryanpeterson:

      It could be, if what you were trying to measure were a quantum mechanical observable. The problem, then, may be that we don’t know what observable we’re trying to measure. Education? Societal well-being? Total economic production? These are essentially undefined yet undeniably important terms. We don’t know what they mean, so we can’t define a measuring apparatus for them. Instead, we find something else we *think* is close to what we really want, and measure it instead.

      And if we ever get to a point where we can measure well-being, or happiness, or utility directly in a numerical sense, I have to ask: what would the complementary variable be that becomes uncertain? Remember, you can measure position or momentum as accurately as you like; the uncertainty limit is only if you try to measure both of them.

  34. Seems like the problem isn’t the measurement, per se, it’s the tying of measurements to incentive.

    I’m seeing this in the US No Child Left Behind Act legislation, as well. My son’s school was tagged as failing. He’s in elementary school. For passing, they measure only reading and mathematics (in this context, really arithmetic). Easy solution: stop teaching social studies and use that time for math and reading.

    But I don’t see how we can just give up measurement, pace those who really hate this law. The abuses this law was created to fix are real: school systems were just letting students fail — and especially students in racial and linguistic minorities.

    So we aren’t going to be able to just give up measurement (would you really want the medical system to give up trying to tell if drugs really work?).

    I’d add another “law” to this — legislative bodies are almost unimaginably bad at incentive engineering.

  35. 1/Quantum physics teaches us that you can’t measure anything without changing the state of the “object”. Basically you can’t even look at a quantic object to check if it’s red because just looking at it will change its color. (Max Plank just died a little more but you get the idea)
    2/In the past few weeks I had a lot of discussions about business practices that don’t seem to make any sense. For instance you would expect that when a plane has been delayed, the airline will try to schedule it as soon as possible even if it means that you need to delay a bit the other planes.
    But, as we just saw with the volcanocalypse, this is not how airline operate. Once you are late you fall at the bottom of the priority list because, well, you are already in the pissed off category and they don’t have a “very pissed off” quality counter so they really don’t care much how pissed you’ll get.
    Hotlines work in the same way. Once an operator becomes available, common sense dictates that he should pick the customer who has been waiting for the longest time. But it’s not the case. He will pick a customer who is still in the “normal” waiting time average.
    Why do they do that ? because instead of having 2 customers thinking they are getting bad service they have only one. But, boy, this one is really screwed up.
    3/ Thanks to Wall Street we hear a lot about Management Bonuses and how they can lead to horrors. We have no troubles understanding how “old school” bonuses based, for instance, on stock price can lead to styles of management that can sometimes be extremely one-sided and very short-term focused but it seems that any bonus scheme can, eventually, end up in a disaster. Yes. Even “customer satisfaction programs” can turn evil. I’ve seen it.

    Thanks to BoingBoing, I was able to put a name on the phenomenon at work in the above examples : the Goodheart’s law.
    I see this law as basically the social science version of quantum physics : as soon as a statistical object is used as a control measure, it loses its value and becomes an incentive to twist reality.

    That’s why “quality of service” contracts who are, in essence, good ideas become a tool that can allow companies to claim “high satisfaction ratios” while knowingly raping a significant portion of their customers.
    It’s not so much that good performance indicators are hard to build (they are). It’s probably just that there is no limit to human inventivity when it comes to “meeting the target at all cost”.

    And that led me to a pretty sad thought.
    I used to think and that, yes, the fight against poverty was sure taking forever but that, one day or another, things would improve because, well, you can’t just abandon those people can you ?

    But now I’m afraid that once you fall into any “really screwed up” sub-population such as those who have been getting elevator music for 45 minutes on the goddam hotline or those living in urban ghettos with 70% unemployment rate, you have actually become a dead statistics with a red label that reads “Doomed”
    And there is no limit to your doomed status because, you know, if your plane takes off now it would delay all the other planes and even if it’s only by a couple minutes that would really fuck up our Quality of Service meter and we can’t let that happen, can we ? Oh. And your luggage is lost too.

    1. hmm, rats stimulating themselves to death thanks to a switch that gives them a zap to the brain?

  36. I haven’t heard this articulated before, but it’s nice to have a name for it, because it’s an important point. Although people exploiting loopholes adds a lot to this, I think to some extent it might be a fundamental result of using metrics.

    Essentially, something like economic health is too complex to define precisely. If we want to evaluate decisions, we have to choose some surrogate measurement, say GDP, that captures some part of what we mean.

    If this works well, higher GDPs will generally mean healthier economies, and in most cases raising one will raise the other. The problem is that relentlessly maximizing it guarantees that you will find any cases where they don’t match up, and which point it will not help and likely harm other parts of health.

    This would happen even if nobody was trying to game the system, simply trusting that the measurement was a perfect encapsulation of the basic idea. I expect this sort of mistaken over-reliance on simple numbers probably happens as often as deliberate distortions; it’s not uncommon for schools, businesses, or countries to sabotage themselves.

    As a note, none of this has much to do with the Heisenburg uncertainty principle, which is not really about observations changing reality and has more to do with mathematical properties of waves.

  37. It is indeed a nice formulation of a consistent pattern.

    True, snack food makers aren’t out to poison people (for example). They are out to compete successfully and make a profit. But emergent from that is…

    Almost invariably when you want to manage something, you measure something else as a surrogate for the unquantifiable thing you are after. And almost invariably, there is a way for someone to get ahead by enhancing the surrogate without enhancing the true objective. Therefore, almost invariably, measuring and deciding on the basis of the surrogate rapidly debases the surrogate.

  38. Once you measure something, it changes. Finally, I have a valid explanation for my girlfriend. That’s why my measurements of stuff and her visual perception of the same stuff doesn’t match up.

Comments are closed.