Most social science results have never been replicated

Replication — where researchers re-do experiments to see if they get the same result — is a really important part of the scientific process. And it's hardly ever done in social science.

Notable Replies

  1. You'd think that psychologists, of all the people in the world, would know that they might be fooling themselves, so replication, replication, replication. What the heck are they afraid of?

    Next on the chopping block: economics! Where are the paired experiments?

  2. Just pointing out that, in addition to this, most social science is assumed to be universal despite not being corroborated with a cross cultural study. I'd say, before you try to claim universality, you should at least have people from a variety of social classes, representatives from a couple of different non-european cultures, and representatives of some tribal culture. Psychologists are the worst for underestimating the degree to which culture shapes people.

    It actually gets worse. Clinical psychologists might as well be witch doctors. The disorders in the DSM4 are generally assumed to be something innate to people, but they are in fact almost completely culturally constructed, and the spread of western-trained mental health professionals around the world is bringing those disorders to people who previously didn't have them:

  3. No one in science actually replicates experiments, in any field. Good experiments are costly. They're costly in terms of design, in terms of setup, in terms of time of highly skilled people, etc. Replicating someone else's experiment saves you a little bit of that design time, but even the best journal article won't give all the details needed to properly replicate an experiment, and there's still going to be a lot of time just coming up with how to replicate a given experiment.

    And in the end, there's very little to gain from the replication. If you get the same answer, then your result won't be significant enough to be published. If you get a contradictory result, then you're likely in for a prolonged battle with reviewers who are (naturally) skeptical of a paper that directly contradicts existing literature.

    As a result of all of this, the value gained from replication is low, and so it's nearly impossible to find funding for such experiments. The only people who would fund such work would likely be corporate entities with a vested interest in such a contradictory result (eg, oil companies funding alternative hypothesis climate change research). However, that runs the risk of tainting the research, and there are no such vested interests in social sciences, that I can think of.

    In my mind, there's actually a reasonable solution to this quandary: graduate students. As per our traditional academic system, earning a Master's or PhD in the sciences involves conducting and writing on unique research in the field. However, because many scientific disciplines are fairly mature, the research in those fields tend to be esoteric to the point of being useless. Case in point, student theses have some of the lowest impact of any published works. In essence, the only valuable product that comes out of a student's years of research work is the researcher them self. The grant money that gets funneled into that student is largely wasted.

    We could, however, solve both problems in one fell swoop. Create a new path to a degree by allowing students to replicate an appropriate experiment (or experiments) as determined by their thesis committee. The students gain firsthand knowledge of how experiments in their field are designed and carried out, and they make a meaningful contribution to their field by validating works that could be fundamental.

  4. Well, speaking from within the social sciences, I'd say that most social science is presumed to be universal by people who don't know much about social sciences. In my field when I was in graduate school there was intense conflict about every damn little thing - to the point of insanity. And cultural context was one of the many nuances that make it damn near impossible to draw conclusions about anything when talking about humans.

    That doesn't mean there isn't scientific value in building our understanding of how people interact and how we operate socially. It just means it's really hard to do. In comparison to physics or even biology we are just now sorting out the periodic tables and discovering leukocytes, and there are still a lot of people searching for phlogiston or advocating a good healthy bleeding to set the patients straight.

    The problem is when so many people are making vast policy and political prescriptions based on nascent sciences. Economics as a discipline is like a two year old - full of potential, slowly discovering its own limits and the world, cute at times, with a shocking capacity for violence and unable to control its bodily functions. No way does the toddler get to drive the car - and yet we make huge, life and society altering decisions based on their latest or most prevalent 'discoveries'.

  5. But they have figured out that repeating the same senseless ideological viewpoint for decades results in a steady paycheck from wealthy patrons. Because the only thing standing between us and utopia is the progressive income tax and the EPA.

Continue the discussion bbs.boingboing.net

39 more replies

Participants