Scientist proposes replacing peer review with something like Reddit

At Genomes Unzipped, Joe Pickrell, a graduate student in genetics at the University of Chicago, makes a case for replacing the traditional peer review system for scientific publications with something more akin to social networking. Or, really, a professional version of Reddit, plus some deep search features. Not having dealt with the peer-review system myself, I'm curious what the scientists out there think. Good idea? Bad idea? Good idea for some fields, bad idea for others? (Via Daniel MacArthur)

• [Anonymous]

  possibly good. peer-review has many problems but it does allow for some really great positive criticism and manuscript improvement before publication. Some journals and societies are experimenting with open comments of submitted manuscripts to increase the potential number of reviewers. The problem with a reddit type system is the lack of good feedback and revision and it would be subject to the popularity of the author or approach.

• [Anonymous]

  sounds like http://www.scholasticahq.com is doing.

• [HotPepperMan]

  The peer review system is there for a specific reason. Putting an article out for ‘review’ in a Redditt style would, in part, be nonsense. Compare performing a Google search when you are looking for something specific and, due to the frequency or popularity basis that forms part of the algorithm of the result set returned, the specific information you require will be ‘drowned out’ by background noise. As a simple example, searching for ‘Mumbai’ currently will not return tourist information but will be pushed further down the list by current events (one of my pet peeves with Google).

  Peer review is exactly that – a review of your ideas and proposals by people who are (or theoretically should be) eminently qualified to perform a balanced assessment of the work you are presenting. There are countless examples of research performed that would have mired in obscurity were it not for peer review. However, as some have pointed out, there are some areas of peer review that need a serious restructure. Science and medicine is perhaps more competitive than many other areas of research and life where ‘grudges’ and ‘jealousy’ (aka bitchiness) have been known to quash perfectly valid research. e.g. where someone’s research completely overturns an established theory upon which an established career and reputation has previously been based.

• [Robert Goldman]

  The level of effort involved in peer-reviewing journal articles will not be easily replaced by something Reddit-like. Also, there’s some (minimal) payback to academics for participating in the peer-review process to help make it work (look for lines about “reviewer for” in CVs). How would this replace that? Finally, there’s the anonymous component which can be critical in catching problems in materials by the prominent. It seems like anonymous reviewers are “just anonymous enough” — they can’t just be internet trolls, because the editorial committee know who they are, but they can be protected against retaliation if they catch real errors.

  A reddit that tries to help people find interesting journal articles might be helpful, but not as a replacement for peer review.

• [awjtawjt]

  Any system that bases judgment of validity on anything other than reproducibility is prone to gaming. Currently no system in place requires public display of the raw data, and only requires a summary of the methods used to arrive at the conclusion. Sure, public registry of data could also be prone to gaming, but it would be public, and assessment of reproducibility would eventually rule.

• [McLuhanesque]

  A big problem in the traditional academic peer-review process is that much of what gets accepted, certainly by the tier-1 and generally by the tier-2 journals, must conform to the prevailing fundamental theories and tenets of the subject area. Any suitably controversial approaches, methodologies, or findings are generally suppressed by the system of entrenched received wisdom in which the “expert” reviewers who have themselves been published by this incestuous system are vested in their own approaches, methodologies, and findings. John Ioannidas has written about this (peer-reviewed, I might add) in Ioannidis, J. P. A. (2005). Contradicted and initially stronger effects in highly cited clinical research. Journal of the American Medical Association, 294(2), 218-228, and in Ioannidis, J. P. A. (2005). Why most published research findings are false. Public Library of Science – Medicine, 2(8).

  Having been academically published myself, I finally decided to give up on much of this game (that mostly supports the entrenched gaming of the tenure system) because I became sick and tired of explaining to the reviewers why their received knowledge in my field was quite wrong… err… not supported by the recent evidence, and inconsistent with re-examination of highly cited, but incomplete, older studies. It became more effort to satisfy reviewers and editors than it was to do the research in the first place.

  Whether a socially mediated, more open review system would be more efficacious in “producing” useful knowledge is up for grabs. Personally, I think the Slashdot moderation system is not a bad model. However, because the publication system is so intimately tied to the academic systems of tenure and promotion (that themselves are hugely problematic, IMO), and so vested in very large sums of money, it’s unlikely that we’ll see a substantive change anytime soon.

• [Funklord]

  This is a bad idea in that it would result in a mostly unreviewed and unread literature. Where it has been tried it has frequently failed. PLoS has put great effort into spurring post-publication peer review, yet only around 10% of their papers receive any comments (and very few of those comments are substantive). The Third Reviewer is another recent failed attempt (http://www.thethirdreviewer.com) where the site has sat idle for many months.

  Most studies on peer review include common remarks from researchers that hey refuse to read any articles that haven’t been peer reviewed, which creates a major issue here–who is going to be the first one to read any article? How much time should each scientist commit to reading unreviewed (and likely awful) papers and posting the first review? Given the enormous amount of papers currently submitted and filtered by professional editors, moving this responsibility onto the shoulders of working scientists seems counterproductive. Don’t we want scientists spending more time at the bench? Some times it make sense to pay experts to do the work you don’t want to do or don’t have time to do…

• [KanedaJones]

  yes the peer review system has been ‘gamed’ for the benefit of corporate interests.

  BUT

  but adding a social network to it, no matter how filtered or watched, is going to be ‘gamed’ to the same extent or, more likely, further.

  maybe a more authoritarian investigation of who gets chosen for what in the current system would be a better place to start.

• [Anonymous]

  This would not solve the primary problem with peer review, which is the fact that it is not double blind. The presence of a famous and respected name in the author field has enormous effect on the analysis done by a reviewer. Until this is corrected science reviewing will be sub-optimal, no matter what form it takes.

• [Don]

  There’s a typo in the URL.

• [Sam]

  Good idea for anthropology, bad idea for gynecology.

• [Anonymous]

  It’s an interesting idea, but is certainly has as many shortcoming as peer review. There are three common complaints about peer review:
  It takes too long
  It is overly subjective
  It can keep research streams alive that are arcane at best and methodologically and conceptually flawed, by circulating them within a closed group.

  This “ACreddit” idea would prevent all three of these issues, but it has major issues of it’s own. I’ll give an example from my field, which is public health. In PH, chronicity and primary healthcare are fairly dominant research topics. The trouble with a “voting” scheme is that other areas of research , like health equity, international health, social determinants of health etc. would be at best ignored and at worst actively downvoted. There is important work that is controversial, and there is important work that is unpopular, and a voting scheme would force research to the conceptual mean, rather than encourage exploration.

  • [Anonymous]

    #3, welcome to what all of us in controversial areas of research have to put up with. I think that we should look more closely at the merits and potential impacts. Any system where the pack can hunt the sharks is a good thing

• [Anonymous]

  Bad idea all around. It’s not terribly difficult to get an article published if you want/need to get it out there. However, it is difficult, and it should be, to get it into a “top-tier” journal, and it should be.

  Replacing it with a reddit like system would simply lead to even more pronounced publication trends coupled with a real fear of gaming the system.

  Now, creating a repository of journal articles that is searchable and centralized, you know like web of science, is a good idea.
  Which is why it’s already been done.

• [AA]

  Excellent Idea for sure…And if i may add, while an article is in “peer review” mode, the authors details should remain hidden.

  • [Jonathan Badger]

    There are journals that hide the authors’ names from reviewers. It isn’t very useful in practice because typically papers cite many previous papers by the group writing it that can used to infer the authorship — and you really can’t hide those citations because diligent peer reviewers often need to read previous work to make a fair review.

• [Snig]

  Also, anyone doing research on magnets would get trolled mercilessly.

• [Don]

  I’m a fan of peer review. I like the fact that nonsense gets winnowed out, however imperfectly, before scientific work is published in a journal. The alternative idea is to have peer review happen after publication, in public space. There wouldn’t be as high a bar against bad work, but I imagine the consequences of bad work would be higher (public, rather than private, embarrassment). I would hope comment-writing would be restricted to people with actual scientific credentials, to keep the signal-noise ratio high.

• [KnoxHarrington]

  Peer review is definitely broken, so I’d be willing to give anything a try.

  But one of my biggest complaints with reviewers is that they don’t make the effort and time to understand what a submission is about. I’ve gotten a lot of reviewer comments to the effect of “You failed to cite X’s work!” or “You didn’t address issue Y” even though X and Y aren’t remotely relevant to the topic and are instead just hobby horses of the particular reviewers. I’m not sure I would put a lot of faith in a Reddit-like system to address this. They aren’t exactly known for people restraining themselves from snap judgements.

  One other concern: would the ranking or reputation of a paper be too influenced by the popularity or self-promotion of the author? One small advantage of the peer review system is that my papers aren’t (that) disadvantaged compared to the ones authored by the guy who’s at every conference glad-handing.

  Could both system exist side-by-side? Traditional peer review for the more prestigious journals, and the Reddit-like system for pre-prints, smaller articles, updates to previous work, and the sorts of things that would have gone to more minor conferences and work shops?

• [Julian]

  This may work well for the peers that review peer reviewed journal articles (ie. experts in the field of the article), but it doesn’t provide enough fact-checking to prevent studies that are improperly conducted, based on faulty logic, or completely fabricated from being up-voted by people who don’t know enough about the particular topic to catch them but are impressed by them. The peer reviewed journal system has many issues, but I suspect this will not improve things.

• [Anonymous]

  It already exists, in a way, with the Faculty of 1000

  f1000.com/

  Some 5000 people, who are experts in their fields, review the most promising articles (that have already passed the peer review system, granted) in their field.

• [Zbohannan]

  Not sure about some of the assertions in the article.

  The author seems to discount that although reviewers can make silly demands at their whim, they also often contribute useful ideas for further experiments that strengthen the science.

  Would an astrophysicist be able to “Like” an article about “arsenic-eating” bacteria? The hive mind is not the same as an authoritative expert, so the entire idea is based upon the belief that humans are 100% logical based solely upon information given. We know how well that works. I’m not sure the establishment of scientific fact is analogous to awesomely cute pictures of kitties.

• [Anonymous]

  As someone who has been through the wringer on this I am ambivalent for several reasons.
  One: The review process is supposed to be blind. A social network system would undermine this because persons familiar with your work could comment (for good and bad).
  Two: Journal editors farm out submission reviews to reviewers with two key characteristics:
  1) They have a some credentials that make them an authority on the topic.
  2) They bother to do the work (read, respond) in a timely manner.

  For this system to work and retain credibility both of those criteria still have to be met or you just end up with speed reading trolls dominating the review process. And as anyone who has submitted a paper for review can tell you, you the current system always have includes that one scholarly troll you can’t satisfy. So no gain there.
  Three: The editor still decides which comments have merit, so what will this change really accomplish? The real decisions really get made in the editor’s office.

• [Cogent91]

  Peer Review in smaller scientific circles is something closer to self-sustaining paradigms than it is earnest efforts at expanding beyond the limits of our current knowledge.

  Even when changing facts show hypotheses certainly true, convincing these groups they’ve been wrong can at times demonstrate rationality isn’t quite as solid a stalwart of the scientific community as many presume. There’s more emotions and politics governing the Sciences than most would guess.

• [Andrew]

  A closely related project is gaining steam amongst academic philosophers: http://www.sympoze.com/

  • [warmlogic]

    Sympoze looks like it’s much further along than this project that a friend of a friend is working on (I know very little about the details), but it’s a similar idea: http://www.scholasticahq.com/

    Also, I feel like something like reddit already exists for some field of mathematics, but I can’t remember what it is called.

• [nate_freewheel]

  What about a site build on the Elgg platform. There could be two front-ends to the site; one for scientists, one for novices or professionals in other realms of research, engineering, etc. On the novice side, scientists could gain insight into their research by having it approached from hundreds of unconventional but sensible perspectives. On the scientists’ end, it would be a updated and open version of traditional peer review. Both channels would be filterable.

• [Toff]

  Even just as a reader of science journals for laypeople, and having taken only a couple undergraduate science courses, my reaction to the proposal of social networking review as an overall replacement for traditional peer review is still one of complete horror. There may be merits to elements of the proposal, however.

• [Jonathan Badger]

  As I serve as an academic editor besides being a working scientist, I deal with reviewers both from the author of papers being reviewed as well as from the other side as an editor. While certainly there are bad peer reviewers (the system isn’t perfect), they aren’t just random people — they are people picked by the editor because they have written papers similar to the one being reviewed. Despite the faults, I don’t see how opening the review process to anyone with a bone to pick is an advantage. There are more crackpots than people qualified to judge any paper.

  • [Briiony]

    Agreed. The other major problem that I foresee is getting scientists to actually participate in numbers high enough to make this work. The other reason editors hand pick reviewers is to make someone responsible for committing time to review the work.

• [Anonymous]

  Anonymity is the biggest appeal of peer review. Granted the anonymous reviewers are selected by editors who may have their own interests at heart.

• [Anonymous]

  Wow that strikes me as a really bad idea. That would make the scientific community even more susceptible to hivemind and groupthink than it already is. We want to encourage new ways of thinking; Science Reddit would downvote controversial ideas into oblivion.

  For an illustration of how bad this is: Reddit is little more than 4chan with a better vocabulary. Would you really want to leave the job of determining which pieces of science are valid up to 4chan?

• [tmdpny]

  I think thee are some obvious issues with the peer review system that need tinkering, and we can all probably pull dozens of examples of how the system has ‘failed’ – ie, papers with misleading data, misunderstanding of data, blatant misuse of facts.

  But the peer review system overall presents a system where reviewers are pre-qualified and verifiable. The system itself can be audited.

  This is something a more social networking environment can not deliver consistently. There are too many trolls, too many agendas that can easily mis-represent themselves in this type of environment.

  In pharmaceutical research there is an ongoing and serious crisis when it comes to papers being published and the peer review process in general. There are entire departments whose sole job is to ensure peer reviewed papers are published in time with gaining FDA’s favor. This has lead to ghost writing and obvious gaming of the review process.

  There are no easy answers in fixing this issue, but we can at least lean on the editors of publications and the best practices they employ to be transparent and open about the process. It seems at first social networking would deliver this same process (transparency), but under a microscope it is still unable to deliver credible transparency.

• [toekneesan]

  I work in university press publishing and we publish peer-reviewed humanities scholarship. I’ve long wondered the same thing: Could online communities somehow replace the peer review process, or perhaps make it a bit more efficient? I think they can, but I suspect that something like Metafilter would be a better model than Reddit. In other words, it would need to focus on community and not popularity. I’ve also been playing with Google+ recently and that really seems to have enormous potential to aid scholars in their work. If they could better integrate Google Book and Google Scholar, I suspect the academic community would instantly embrace it and it would quickly become a major tool in the scholar’s tool box.

  There was recently a study on the use of social networks in the creation of scholarship conducted by the Centre for Information Behaviour and the Evaluation of Research , or Ciber, which is an interdisciplinary research center based in University College London’s department of information studies, and it seems to indicate that thus far, scholars have used social networks to organize their research at the beginning of the process, and then again to disseminate the work at the end of the process, but thus far it hasn’t been widely adopted for doing the actual work and evaluation. There are some experiments we can look at, including this one Nature attempted in 2006, and this one hosted by MediaCommons for the Johns Hopkins journal Shakespeare Quarterly. I think there’s great potential for open peer review, but I’m not sure we should be looking at Reddit as a platform, unless the scholarship concerns the lifecycle of the meme.

• [toekneesan]

  I work in university press publishing and we publish peer-reviewed humanities scholarship. I’ve long wondered the same thing: Could online communities somehow replace the peer review process, or perhaps make it a bit more efficient? I think they can, but I suspect that something like Metafilter would be a better model than Reddit. In other words, it would need to focus on community and not popularity. I’ve also been playing with Google+ recently and that really seems to have enormous potential to aid scholars in their work. If they could better integrate Google Book and Google Scholar, I suspect the academic community would instantly embrace it and it would quickly become a major tool in the scholar’s tool box.

  There was recently a study on the use of social networks in the creation of scholarship conducted by the Centre for Information Behaviour and the Evaluation of Research , or Ciber, which is an interdisciplinary research center based in University College London’s department of information studies, and it seems to indicate that thus far, scholars have used social networks to organize their research at the beginning of the process, and then again to disseminate the work at the end of the process, but thus far it hasn’t been widely adopted for doing the actual work and evaluation. There are some experiments we can look at, including this one Nature attempted in 2006, and this one hosted by MediaCommons for the Johns Hopkins journal Shakespeare Quarterly. I think there’s great potential for open peer review, but I’m not sure we should be looking at Reddit as a platform, unless the scholarship concerns the lifecycle of the meme.

• [NikFromNYC]

  The blogroll at the top rated science site on the Net is a perfect example of public peer review, though currently it functions as post publication review. About a dozen serious studies by the Climategate crowd have been shown to have been based an “artifacts of bad mathematics” (Dr. Richard Muller of the new Berkeley Temperature Project coined this phrase) as reported on Watts Up With That.

  • [Chevan]

    No, that’s a perfect example of why this wouldn’t work. WUWT is the LAST place anyone should be going for credible, factual information on climate change.

• [Mister44]

  Well… not a scientist, but I am a big fan. I do have one friend who is one of those ‘jack-of-all-trades’ geniuses. He has started several companies, owns a dozen plus patents, as self taught engineer, and has eclectic hobbies like astronomy and paleontology. He is in the process of grinding 44″ mirror to find extrasolar planets.

  While he started grinding the mirror, which is 5 min of setting up, and an hour of waiting, he decided to buy a SEM and look for soft tissue in dinosaur bones, ala Dr. Schweitzer.

  Now what he didn’t have was a fancy degree. So when he wanted to publish his finding – solid science with lots of evidence and collaboration with other scientists – he had a hell of a time getting his foot in the door. It is sort of an “ol’ boys’ club” in that respect. It wasn’t until he got his name attached to a natural history museum that he was able to get his findings published in one of the smaller journals.

  So – I see of another system that could act as an outlet for amateur scientists to be able to show their findings.

  • [snatchamoto]

    Although your friend may have had trouble publishing, I seriously doubt that it was because of a lack of credentials. I don’t have credentials, and I’ve found that if my findings are non-trivial, add something to the field, and are written up properly (a step that many scientists don’t realize is important) they get published.

    Credentials may get your stuff through with less scrutiny, but lack of them is not going to stop you from being published.

    Also, if he/she only submitted to Nature and Science and got rejection letters, they’re doing it wrong.