"Erroneous analyses of interactions in neuroscience: a problem of significance," a paper in Nature Neuroscience by Sander Nieuwenhuis and co, points out an important and fatal statistical error common to many peer-reviewed neurology papers (as well as papers in related disciplines). Of the papers surveyed, the error occurred in more than half the papers where it could occur. Ben Goldacre explains the error:
Let’s say you’re working on some nerve cells, measuring the frequency with which they fire. When you drop a chemical on them, they seem to fire more slowly. You’ve got some normal mice, and some mutant mice. You want to see if their cells are differently affected by the chemical. So you measure the firing rate before and after applying the chemical, first in the mutant mice, then in the normal mice.
When you drop the chemical on the mutant mice nerve cells, their firing rate drops, by 30%, say. With the number of mice you have (in your imaginary experiment) this difference is statistically significant, which means it is unlikely to be due to chance. That’s a useful finding which you can maybe publish. When you drop the chemical on the normal mice nerve cells, there is a bit of a drop in firing rate, but not as much – let’s say the drop is 15% – and this smaller drop doesn’t reach statistical significance.
But here is the catch. You can say that there is a statistically significant effect for your chemical reducing the firing rate in the mutant cells. And you can say there is no such statistically significant effect in the normal cells. But you cannot say that mutant cells and mormal cells respond to the chemical differently. To say that, you would have to do a third statistical test, specifically comparing the “difference in differences”, the difference between the chemical-induced change in firing rate for the normal cells against the chemical-induced change in the mutant cells.
Princeton University psych prof Susan Fiske published an open letter denouncing the practice of using social media to call out statistical errors in psychology research, describing the people who do this as “terrorists” and arguing that this was toxic because of the structure of social science scholarship, having an outsized effect on careers.
Blue writes, “Peter Watts has be stricken with debilitating pain, loss of range of motion and motor control. Watts’ doctors remain baffled despite a battery of tests, and Watts has reached out to his fans to ask for their theories and ideas as to what might be causing his illness.”
Today, I’ve launched a very special Kickstarter with two friends, Timothy Daly and Lawrence Azerrad. A year in the making (and many more years on our minds), the Voyager Golden Record: 40th Anniversary Edition is the first vinyl release of the stunning golden phonograph record launched by NASA in 1977 aboard the Voyager spacecraft, one […]
If you own a dog, you’ve most likely heard of BarkBox – the monthly subscription box for dogs. What started as a simple idea to try out the subscription model on pet owners has since developed a cult following of dog lovers. If you haven’t given it a try yet, this one month free deal is the […]
With the iPhone headphone jack having gone by the wayside, we’re excited about the addition of the FRANKLIN Bluetooth Headphones in our store. These headphones are foldable so they’re easy to carry around, but most importantly, they pack impressive sound. Our biggest struggle with Bluetooth headphones is the worry of them dying at the worst moment. This pair lasts an impressive 8-10 […]
Evan Kimbrell, founder of the digital agency Sprintkick, recently released a series of online courses that feature some of the best advice we’ve come across. These courses are well worth your time, and will save you from making many typical mistakes down the line if you ever want to start your own business.With this Business […]