A smashing editorial in Nature catalogs the many ways in which scientists end up tricking themselves into seeing evidence that isn't there, resulting in publishing false positive. Many of these are familiar to people who follow behavioral economics (and readers of Predictably Irrational). But, significantly, the article advocates a series of evidence-supported techniques (some very simple, others a little more mostly/tricky) to counter them.
The problems with reproducibility in scientific results are now understood to be grave, and the scientific establishment is on the lookout for ways to improve the quality of results. Insisting on some or all of these methods as a condition of publication would significantly advance the field.
One debiasing procedure has a solid history in physics but is little known in other fields: blind data analysis (see page 187). The idea is that researchers who do not know how close they are to desired results will be less likely to find what they are unconsciously looking for13.
One way to do this is to write a program that creates alternative data sets by, for example, adding random noise or a hidden offset, moving participants to different experimental groups or hiding demographic categories. Researchers handle the fake data set as usual — cleaning the data, handling outliers, running analyses — while the computer faithfully applies all of their actions to the real data. They might even write up the results. But at no point do the researchers know whether their results are scientific treasures or detritus. Only at the end do they lift the blind and see their true results — after which, any further fiddling with the analysis would be obvious cheating.
How scientists fool themselves – and how they can stop
In a new interview with the Journal of the American Medical Association above, Anthony Fauci, National Institute of Allergy and Infectious Diseases (NAID), said he expects the US will have 100 million doses of a COVID-19 vaccine before the end of the year. “Then, by the beginning of 2021, we hope to have a couple […]
Just for kicks, Paul Rule, 66, participated in a study launched by the Cambridge Natural History Society that enlisted citizen scientists and nature-lovers to help deepen knowledge of the flora and fauna in Cambridge, England. Rule recorded nearly 600 different animal species in his “ordinary” city garden, including an elephant moth like the one seen […]
Astronaut David Scott re-created, in 1971 during the Apollo 15 mission, Galileo’s “falling bodies” experiment by dropping a hammer and feather on the moon at the same time. Simply, both fell at the same rate because there was no air resistance. screengrab via Wonders of Physics/YouTube (Digg)
The world is holding its collective breath. As states begin cautiously reopening, no one is sure exactly what to expect. But one thing is clear: most Americans are worried about their bank accounts. By the end of March, the average American household was spending 40 percent less on their credit cards than they were one […]
Over 25 years, eBay has carved out its space as the commerce hub of choice online. With 182 million users worldwide, that works out to about 35 percent of all US mobile users who shop those eBay storefronts. But did you know there are usually around 1.3 billion — with a B — active for-sale […]
Software apps are a dime a dozen. Well, if you’re going by their actual monetary cost, maybe not really. But considering how useless some poorly conceived, poorly executed apps are at doing the job you actually downloaded them to accomplish, it isn’t a stretch to think that many apps aren’t even worth a free download. […]