How bad research gets published (and promoted)

In 2010, a group of scientists claimed to have found bacteria that could build its DNA using arsenic, instead of the phosphorous used by the rest of Earth's life forms. Within days, the research behind "arsenic life" was under serious scrutiny and we now know that it was totally wrong. But the work was peer-reviewed. It was sponsored by NASA. How do so many experts make such a big mistake? Dan Vergano at USA Today has an excellent article looking at just that — and it includes the peer review comments that helped the arsenic life paper get published. Though normally secret, Vergano got a hold of them through a Freedom of Information Act request.



  1. Of course, history of science is very clear that mainstream science is not always right.  BoingBoing seems observably uninterested in reporting on cases where mainstream science is actually losing the battle to defend the textbooks.  For science journalists, this is surely the safe route that they can take to avoid risk, but the cost is simply shifted to the public, by transforming the role of the press from a critical check upon scientists, into a self-supportive, non-critical echo chamber.

    It also creates an additional barrier within the media which disruptive innovators who are actually achieving success must then surmount.  Perhaps the most prominent case in point is Gerald Pollack of the University of Washington, whose work is routinely validated within mainstream laboratories, but is generally not reported on by science journalists.

    One gets the impression that science journalists increasingly view their job strictly as debunkers, cut from the mold of Carl Sagan; they appear to not realize that they also possess a burden to society to act as a conduit for apparently good ideas at the fringes of science, as history of science is also clear that establishment scientists are not — in truth — as eager to throw away their own legacy in order to further the cause of science.  They are humans, just like us.There is some danger to casting a light upon the one type of story, but not the other, for it lends a false appearance to the public that our science textbooks are perfect, and that controversies in science do not rage on to the current day, over many decades.  And in a world where our textbooks are imagined to be perfect, students will assume that their own task is to memorize them, rather than question them and make them better.BoingBoing doesn’t have to repeat the same mistake that other news aggregators and science educators are making en masse.  The education reform movement is mostly constructivist in nature, anyways.  When Carl Sagan claims that, “Extraordinary claims demand extraordinary evidence,” the constructivist responds by asking, “Extraordinary to what worldview?”  Where science journalists typically derail is in their willingness to agree with established scientists that we are capable of constructing just one scientific worldview, based upon the data.  And they largely presume this without actually checking that it’s actually true, by actually interviewing the people who are trying to build those competing models.

    1. Brilliant comment. I fully agree !

      More and more informed members of the public are noticing exactly the same thing.

      Journalism has the moral obligation to be critical. But nearly all science journalists are in complete harmony with the topic they are reporting on. They act as if they were the PR department of science itself.

  2. Something that the linked article mentioned but didn’t really stress is that there are really two types of peer review in science. There is the formal peer review by a couple of scientists before the paper is published, and there is the informal review by the entire scientific community *after* something is published. Both are crucial to the workings of science — scientists don’t take everything published as truth, especially when it goes against what is generally known. Yes, sometimes this leads to resistance to real groundbreaking results, but it also serves to as a self-correcting mechanism for the far more common cases like arsenic life, cold fusion, and the like.

  3. “NASA released [the reviews] to USA TODAY in January after a Freedom of Information Act request.”

    Why not first ask the authors of the study? I am not totally sure about this, but usually, the authors (and the journal) should have a say in this, not NASA as an institution.

    Anyway, Maggie: reviews are anonymous, not secret. No secrets involved beyond the (chosen) anonymity of the reviewers. Their identity can only be disclosed by themselves, or by the journal/publisher with their consent.

    Reviews are send by email, unencrypted, to the submitting author. No secrets, just very uncommon to be discussed in public. It’s really bad style. And, POI,  I do wonder if USA Today had the right to send the reviews to other scientists, actually, without the consent of the publisher.

  4. It’s not like there’s some board at NASA (or any major research institution or university) that reviews papers for merit and/or accuracy before they’re sent out.  There is usually a review for release, but that’s for IP and ITAR.  The people doing this review are not scientists, and probably not familiar with your field.  They’re looking to see if you put a circuit diagram in.

    If you’re a scientist at NASA (or anywhere else I’ve ever worked), you’ll show your paper around to your colleagues, hope that some of them go to the effort to really read it, catch your mistakes, and offer corrections.  If it’s a collaboration, then there will be a collaboration review process.  There’s no institutional review anywhere I’ve ever been.

    Then you shoot it off to Nature, Science, PRL, or wherever, and hope you get good reviewers (i.e., in your field) who aren’t rivals.  That’s a big ask.  Lots of petty stuff goes on here, though it’s been my experience that most people are too kind in this process.  No one is “approving” you sending it to Nature, or not.  That’s entirely up to you.

    If it catches someone’s attention, or if you’re the type of self-promoter (how you get ahead in this biz) that’s going to work the PR department to get this out, it still likely won’t be vetted by anyone else.  They’ll take your word for it.  You’re likely one of the institutions (and maybe world’s) experts in this field – who are they going to get to review it?  It’s already been peer-reviewed by Nature, for crissakes.  How’s some PR flack at NASA or Stanford or Utah going to do better than that?

    That’s how bad research (and good) gets published.  And corrected.  As in this case.  It’s the peer-review process.  The worst one in the world, except for all the others.  The problems with it are well known, and well documented.  Except no one has figured out a better way to do it.

    And it’s not as though, for the most part, anyone is trying to publish bad research.  This stuff is hard.  Scientists are subject to the same failings as any other group of people.  Hubris, self-centeredness, work-induced tunnel vision, blind spots, pride.

Comments are closed.