Thought-provoking essay on cause and correlation in modern science


39 Responses to “Thought-provoking essay on cause and correlation in modern science”

  1. Mujokan says:

    Already submitterated just for the record. But you guys’ volume has been great lately, helping keep me occupied.

  2. Wait… so we’re not allowed to be wrong and learn from our mistakes? I mean, I get it, we don’t know as much as we think we know. But that’s no excuse to throw up our hands. The example given for torcetrapib is important, because perhaps without going through the trials and ultimately discovering its failure, we wouldn’t have been able to add that knowledge to our understanding of the whole interrelated cholesterol mess.

    So to my mind, anyway, this is an indication of the success of science, not its failure.

    • eyebeam says:

      I’m not sure what article you think you read.

      Do you think having one’s back permanently maimed, because some scientist decided to label normal anatomical structures as “abnormal,” doesn’t qualify as “failure?”

      • I said it wasn’t a failure of science. We learn from our failures in trials. No doubt, it’s a tragedy, but if we didn’t do anything unless we were absolutely guaranteed that nothing would go wrong, we wouldn’t do anything at all, nor would we learn anything.

      • Joshua Ochs says:

        Doctors had patients with back pain, and new technology let them see there were bulges and “odd” things happening with the associated spinal structures. There was no reason to run an (expensive) MRI on folks with no back pain to see whether those odd structures showed up in them. Correlation became causation – but incorrectly. Then one day they decided to check it out, and lo and behold, those abnormal structures – while perhaps part of some other issue, or perhaps not – had nothing to do with the back pain. The view on what causes back pain and how to treat it changed as a result.

        The only failure here was accepting correlation for causation, which is the whole point of the article – that we cannot be quick to do that, however “obvious” and tempting it might be.

        • Marianna says:

          See the thing about the “there was no reason to run MRIs on healthy people” argument means that this was a failure of medical care, not science, because the scientific method most definitely prescribes MRIs on healthy people in that situation.

  3. eyebeam says:

    I’ve known several people who suffered back pain, and had surgery to remedy it. In none of the cases did the pain improve; in all of them their debilitation increased after surgery, some of them receiving multiple “experimental” procedures in an attempt to relieve the damage caused by previous operations.

  4. Ianto_Jones says:

    I think I will still continue to conclude that the herniated lumbar discs that are impinging directly against my sciatic nerve are much more likely to be the cause of my low back pain than depression (I’m not depressed) or smoking (I don’t smoke).

    • eyebeam says:

      What this article is saying, is that doctors were too quick to look at an instrument scan, and assume that what they were looking at was the cause of the pain, rather than a symptom, then use radical surgery to “fix” the “problem,” often causing more, permanent, damage.

      An alternative explanation is that the lumbar discs may not be the problem in and of themselves; it may be that your muscles are pulling on your back in a way that deforms the discs, and pinches your nerves. The solution in that case wouldn’t be to remove the disc, it would be to retrain your muscles to stop doing that, through postural adjustments and exercise.

      • Greg Davis says:

        “The solution in that case wouldn’t be to remove the disc, it would be to retrain your muscles to stop doing that, through postural adjustments and exercise.”

        The problem is that this results in a physical therapist being paid, instead of a doctor and a pharmaceutical company. I’m sure you can see why this prescription is disfavored (or, more precisely, why the medical industry is largely set up to at least not encourage and perhaps actively discourage doctors from learning this lesson).

  5. gus mueller says:

    i’m really glad to read this. a month ago i had such bad “old man lower back” that every time i got up from the computer, i had to clutch myself for a few seconds.  now, though, it’s totally back to normal. benign neglect was, it turned out, the best treatment.  had i taken homeopathic fairy dust, i’d be singing its praises instead.

    • eyebeam says:

      I had that problem. Know what fixed it? I got a new chair.

      • I chose the lower back pain example because I have that issue myself. Luckily, I also had an amazing, evidence-based doctor who explained that researchers had done all these studies comparing the effectiveness of various treatments … drugs, surgery, chiropractic, etc. But they had to keep stopping these studies because the control of moderate exercise seemed to work better than any of those things. 

        So now I exercise regularly. When I keep it up at at least 3 times a week, my back doesn’t hurt. 

  6. A doctor misusing new equipment leading to a poor diagnosis is a far cry from what a scientist does. A scientist would know that any good hypothesis has to be taken in the context of the systems it interacts with or is affected by. Otherwise, it ain’t a valid hypothesis.

    • eyebeam says:

      This isn’t about individual doctors misusing new equipment (though certainly that is a problem), it’s about students going to medical school, and learning these causal doctrines as accepted truth, that they then play out on their patients.

      • Joshua Ochs says:

        And that’s why any decent doctor keeps up-to-date on the field – medicine is simply one of the fastest-moving fields there is. My dad illustrated this vividly over Christmas by pointing out that of all the drugs in the (thick, heavy) PDR when he was in medical school, only a handful are still used; similar things can be said of a lot of treatments. While the basics of anatomy and patient care are very much the same, most of his drug and treatment facts are obsolete, so doctors have to keep up-to-date. At the same time, they have to be careful not to be swept away in the latest treatment or diagnosis until it’s been properly tested and established. A tricky balancing act, to be sure.

        • eyebeam says:

          Few doctors can do that, not with the workloads they’re expected to carry. I’ve discussed new research and treatments with doctors who had never heard of such things. The system itself is designed to discourage it.

  7. Todd Knarr says:

    It’s a fairly common mistake: when you run a test only when there’s a problem apparent, you find a correlation where none really exists. The classic example is asking “Does drinking water make you more likely to be a criminal?”. The answer is obviously “No.”. But if you take a sample of jail inmates, find out how much water they drank and crunch the numbers, you’ll find there’s a very high correlation between any consumption of water and criminal behavior. That isn’t because drinking water causes criminal behavior, it’s because *everybody* drinks water including criminals.

    If you want to ask “Does X cause Y?”, you can’t select your sample based on Y and look at the occurrence of X. You have to select samples of X and not-X, and then look at whether the occurrence of Y differs significantly between them.

    • eyebeam says:

      What do you do when you have samples of [A, B, C, D, E, F, G, H, I, J, K, L, M, N, O, P, Q, R, S, T, U, V, W, X, Y, Z, Z1, A22, HM-23], and [not-A, B, C, D, E, F, G, H, I, J, K, L, M, N, O, P, Q, R, S, T, U, V, not-W, X, Y, Z, Z1, A22, HM-23]?

      • eyebeam says:

        … and that’s the simple case.

      • AnthonyC says:

        Option one: Look for simpler subsets of the data/simpler versions of the problem, if they exist, and then test to see if the conclusions carry over. This is easier but will give more wrong results.

        Option two: collect a whole heck of a lot of data and accept that your analysis has to be 29-dimensional. This is much more expensive.

    • usfoodpolicy says:

      Naw, if everybody drinks water, criminals and non-criminals alike, then there is not a “very high correlation between any consumption of water and criminal behavior.”  The correlation in this example is zero.

  8. Luis Teixeira says:

    Please note that medicine and science are not the same thing. Many medical procedures are based on case studies or trial and error. With a real scientific approach the control (analysing a group of healthy individuals with the same technique) would have been done.

    • DeargDoom says:

      I am neither a scientist nor a doctor but I find it astonishing that the results of the MRI scans of patients suffering back pain were not compared against a control group for well over a decade.

    • madopal says:

      Indeed.  It’s shocking how control groups are forgotten so often in medicine.

  9. usfoodpolicy says:

    I think the WIRED passage does not use the word “correlated” correctly.

    This sentence says that people with degenerated discs were MORE likely to have back pain, compared to people without degenerated discs: “Back pain was strongly correlated with seriously degenerated discs, which were in turn thought to cause inflammation of the local nerves.”

    By contrast, this sentence says that people with degenerated discs were LESS likely to have back problems, compared to people without degenerated discs: “The presence of such abnormalities is just as likely to be correlated with the absence of back problems….”

    The author probably meant the first sentence to say “people with back pain commonly have degenerated discs” and the second sentence to say “people without back pain also commonly have degenerated discs.”

  10. jimmoffet says:

    Why did it take 20 years of using an MRI machine to diagnose back pain before a set of control images was created? Did they use one of those life-size plastic skeletons for a control?

    How could you use an imaging technology for 20 years without looking at a few symptomless people?

    I’m pretty sure that if you asked a bunch 12th graders how to set up an experiment, at least a few of them would be able to tell you that it involves a control group and that large sample sizes are better than small ones.

    It seems like this is a failure of basic scientific competence rather than a reason to question the limits of the scientific method.

  11. cmpalmer says:

    But you might have a situation where prisoners only had water to drink and people outside drank soda, beer, wine, tea, etc. Then you would have a positive correlation of “People who drink more water than other drinks are likely to be criminals.”

  12. Michael Mandell says:

    Sounds like doctors who decided that back abnormalities caused back pain did not do a controlled test of this hypothesis. That means they were not doing science.

  13. Crispiann says:

    Phew, I’m glad that at least global warming is easy to figure out.

  14. TaymonBeal says:

    Bayesian inference is your friend. I love the back pain example; it’s a classic example of what happens when you fail to take conditional probability into account.

  15. Virgil22 says:

    This is actually a pretty bad example of the root problem, namely the confusion of cause-effect with correlation.  In the back pain example, there’s not even a correlation to begin with!

    A better examples is from Helmut Sies writing in Lancet in the 1970s… birth rates in Germany have declined over the past 5 decades, and the population of wild storks also declined.  Correlation, yes.  Cause and effect, probably not!

  16. The causal shortcuts spoken of in the article are exactly why I’m skeptical of GMOs sold as solutions to poverty and starvation – “feeding the poor”. Apart from the standard capitalist fallacy that the solution should be created afresh rather than change from within (through more equitable food/money/power/energy distribution) we build a solution to sell. But the only evidence we have is that it grows better without this or that. 

    Absolutely *no* long term impact studies on what happens when your GMO crop cross pollinates with my organic crop accidentally and we all end up eating Monsanto’s crop. But we have no idea that tweaking this or that within the crop wont affect our nutritional uptake, or any number of other potential dietary problems…

    Anyway, my 2c

Leave a Reply