How experimental design can create conflicting results


11 Responses to “How experimental design can create conflicting results”

  1. Nylund says:

    No matter what you do, you’re always just hunting for the missing counterfactual and until we find a way to access an infinite number of parallel universes where we can find the one identical in all ways except for the one detail of interest, we’ll never truly ever find that missing counterfactual.

    Whatever you use as a substitute for that missing counterfactual will affect your conclusions.

  2. mgatch says:

    This can be as simple as different doses. I recall reviewing manuscripts from two different labs, one which insisted that a particular had a strong effect, the other that it had no effect at all. My lab had just finished a characterization of the full dose effect of this particular compound and yep–the dose lab two tested was at the bottom end of the dose effect function, and the dose lab 1 tested was at the top end. So, both labs were correct while actually missing the big picture.
    The problem can be compounded when a set of studies is analyzed by a meta-analysis–the effect sizes from all the studies are combined and an overall effect size is calculated. If very different methods were used in different studies–the effects were different due the experimental manipulation, not just differences in sampling. I’ve found enough cases of apparently weak and/or non-sensical effects to be due to variables no one’s looking at to provide plenty of stories for the next time I teach a research design class. Now there was the time with the calcium channel blockers . . . .

  3. Lobster says:

    Wait, so… what you’re saying is that… vaccines DON’T cause autism… but anti-biotics DO!? 

  4. smut clyde says:

    If the autistic sample includes kids with weird dietary fixations, like refusing to eat anything except jello and lettuce, then I would be surprised if their gut microbiome *didn’t* differ in some way.

  5. smut clyde says:

    Are there children who don’t have dietary fixations?

    Point taken.
    I bring more than the usual amount of skepticism to any research on gut-flora differences in autism because

    (a) there is the question of causation. The bacterial population is affected by the dietary issues and odd bowel habits often found in autistic children.

    (b) there are too many quacks and scammers promising to cure autism by changing the gut flora with their special diets.

    (c) the claims keep changing (as documented in the Wonderland post). I know of one researcher, Finegold, who began by publishing a paper (in “Medical Hypotheses”, never a good sign) promoting the idea that Clostridium bacteria in the gut were responsible for autism.
    Two years later he had changed his mind in the face of the evidence (and good on him for that); he was telling an AoA conference that Firmicutes (the bacterial phylum containing Clostridium) “was particularly high in the control group”, whereas a group of autistic subjects had more than their fair share of a different bacterial phylum, Bacteroidetes.

    That did not pan out either, it seems, and in the subsequent year he followed the shotgun approach, running DNA sequencing on innumerable samples of poop and looking up the results in the libraries to find anything different between the autistic and control groups. Pity the poor grad students handling the testtubes!

    In a 2011 Med.Hyp. paper he announced that Desulfovibrio was the lucky bacteria. Finegold deserves some sort of medal for persistence, if nothing else. And if the difference isn’t confirmed, well there are plenty of other species so he can keep trying.

    Everyone wants to discover the next Helicobacter pylori and win a Nobel prize.

Leave a Reply