How experimental design can create conflicting results

Is coffee bad for you or good for you? Does acupuncture actually work, or does it produce a placebo effect? Do kids with autism have different microbes living in their intestines, or are their gut flora largely the same as neurotypical children? These are all good examples of topics that have produced wildly conflicting results from one study to another. (Side-note: This is why knowing what a single study says about something doesn't actually tell you much. And, frankly, when you have a lot of conflicting results on anything, it's really easy for somebody to pick the five that support a given hypothesis and not tell you about the 10 that don't.)

But why do conflicting results happen? One big factor is experimental design. Turns out, there's more than one way to study the same thing. How you set up an experiment can have a big effect on the outcome. And if lots of people are using different experimental designs, it becomes difficult to accurately compare their results. At the Wonderland blog, Emily Anthes has an excellent piece about this problem, using the aforementioned research on gut flora in kids with autism as an example.

For instance, in studies of autism and microbes, investigators must decide what kind of control group they want to use. Some scientists have chosen to compare the guts of autistic kids to those of their neurotypical siblings while others have used unrelated children as controls. This choice of control group can influence the strength of the effect that researchers find–or whether they find one at all.

Scientists also know that antibiotics can have profound and long-lasting effects on our microbiomes, so they agree on the need to exclude children from these studies who have taken antibiotics recently. But what’s recently? Within the last week? Month? Three months? Each investigator has to make his or her own call when designing a study.

Then there’s the matter of how researchers collect their bacterial samples. Are they studying fecal samples? Or taking samples from inside the intestines themselves? The bacterial communities may differ in samples taken from different places.

Read the full story at The Wonderland blog

Image: Apples & Oranges - They Don't Compare, a Creative Commons Attribution (2.0) image from thebusybrain's photostream


  1. No matter what you do, you’re always just hunting for the missing counterfactual and until we find a way to access an infinite number of parallel universes where we can find the one identical in all ways except for the one detail of interest, we’ll never truly ever find that missing counterfactual.

    Whatever you use as a substitute for that missing counterfactual will affect your conclusions.

  2. This can be as simple as different doses. I recall reviewing manuscripts from two different labs, one which insisted that a particular had a strong effect, the other that it had no effect at all. My lab had just finished a characterization of the full dose effect of this particular compound and yep–the dose lab two tested was at the bottom end of the dose effect function, and the dose lab 1 tested was at the top end. So, both labs were correct while actually missing the big picture.
    The problem can be compounded when a set of studies is analyzed by a meta-analysis–the effect sizes from all the studies are combined and an overall effect size is calculated. If very different methods were used in different studies–the effects were different due the experimental manipulation, not just differences in sampling. I’ve found enough cases of apparently weak and/or non-sensical effects to be due to variables no one’s looking at to provide plenty of stories for the next time I teach a research design class. Now there was the time with the calcium channel blockers . . . .

  3. If the autistic sample includes kids with weird dietary fixations, like refusing to eat anything except jello and lettuce, then I would be surprised if their gut microbiome *didn’t* differ in some way.

    1. Not much room for “if” there. Even milder autism, even sub-clinical, is associated with dietary fixations in childhood.

  4. Are there children who don’t have dietary fixations?

    Point taken.
    I bring more than the usual amount of skepticism to any research on gut-flora differences in autism because

    (a) there is the question of causation. The bacterial population is affected by the dietary issues and odd bowel habits often found in autistic children.

    (b) there are too many quacks and scammers promising to cure autism by changing the gut flora with their special diets.

    (c) the claims keep changing (as documented in the Wonderland post). I know of one researcher, Finegold, who began by publishing a paper (in “Medical Hypotheses”, never a good sign) promoting the idea that Clostridium bacteria in the gut were responsible for autism.
    Two years later he had changed his mind in the face of the evidence (and good on him for that); he was telling an AoA conference that Firmicutes (the bacterial phylum containing Clostridium) “was particularly high in the control group”, whereas a group of autistic subjects had more than their fair share of a different bacterial phylum, Bacteroidetes.

    That did not pan out either, it seems, and in the subsequent year he followed the shotgun approach, running DNA sequencing on innumerable samples of poop and looking up the results in the libraries to find anything different between the autistic and control groups. Pity the poor grad students handling the testtubes!

    In a 2011 Med.Hyp. paper he announced that Desulfovibrio was the lucky bacteria. Finegold deserves some sort of medal for persistence, if nothing else. And if the difference isn’t confirmed, well there are plenty of other species so he can keep trying.

    Everyone wants to discover the next Helicobacter pylori and win a Nobel prize.

Comments are closed.