NYT on the controversial ESP paper

The New York Times sums up the controversy around respected psychologist Daryl J. Bem's new paper claiming evidence for precognition. The paper (linked to below) was accepted for publication by the respected Journal of Personality and Social Psychology. Lots of other respected scientists are somewhat-respectfully pissed off about it. But you already knew that, didn't you. From the New York Times:

The paper describes nine unusual lab experiments performed over the past decade by its author, Daryl J. Bem, an emeritus professor at Cornell, testing the ability of college students to accurately sense random events, like whether a computer program will flash a photograph on the left or right side of its screen. The studies include more than 1,000 subjects. Some scientists say the report deserves to be published, in the name of open inquiry; others insist that its acceptance only accentuates fundamental flaws in the evaluation and peer review of research in the social sciences.

“It’s craziness, pure craziness. I can’t believe a major journal is allowing this work in,” Ray Hyman, an emeritus professor of psychology at the University Oregon and longtime critic of ESP research, said. “I think it’s just an embarrassment for the entire field.”

The editor of the journal, Charles Judd, a psychologist at the University of Colorado, said the paper went through the journal’s regular review process. “Four reviewers made comments on the manuscript,” he said, “and these are very trusted people.”

All four decided that the paper met the journal’s editorial standards, Dr. Judd added, even though “there was no mechanism by which we could understand the results.”

"Journal’s Paper on ESP Expected to Prompt Outrage" (NYT)

"Feeling the Future: Experimental Evidence for Anomalous Retroactive Influences on Cognition and Affect" (PDF)

• [genre slur]

  Morphic Resonance or BUST

  “”It startled him even more when just after he was awarded the Galactic Institute’s Prize for Extreme Cleverness he got lynched by a rampaging mob of respectable physicists who had finally realized that the one thing they really couldn’t stand was a smartass.” – Douglas Adams, HHGG” — bwahahahahaha!

• [monopole]

  The Times article links to a pretty good statistical refutation of the results:
  http://dl.dropbox.com/u/1018886/Bem6.pdf

  Just from a cursory skim of the paper I saw a bunch of dubious things which would make me reject it. If the approach was applied to photons with a claim that it proved a breakdown in causality on the scale of 1 second it would be savaged in any physics journal.

  Given the number of signal images (i.e. erotic) in the signal I’d expect shot noise to give 4-5 “bad” signals, for 100 trials I’d put the shot noise value at 2.3% pretty close to the variance reported. So, off the top of my head, I’d just classify it as a statistical fluke.

  Given the gravity of the results (a macroscopic breakdown of causality) I’d purge the test of pseudo-random number generators and use a provably random source (noise off of a tunnel diode for example) and use galvanic skin response, iris dilation, even a plesmograph in an attempt to quantify and time the supposed arousal. I’d also use different positive/negative “signals” (kittens/roadkill).

• [Richard]

  I remember that Rupert Sheldrake has been running similar but more benign experiments on his web site here for a few years:

  http://www.sheldrake.org/Onlineexp/portal/

  If I remember correctly, his results, which are mostly not published at his web site are not quite the same as this paper is reporting. So it remains to be seen if the different testing modalities are related enough to be comparable.

• [Soliloquy]

  So what do you use to remote view?

  I drink. And I find classic rock helps.

  Any music in particular?

  Boston. Boston usually works.

• [Anonymous]

  An argument as old as Psychology. If you haven’t read Deborah Blum’s excellent book Ghost Hunters, about the founding of the parapsychology movement (by the founders of psychology, including Henry James), it is well worth the read. You’ll see that for 130 years, people have been having this same discussion, as respected scientists have looked into these fields and then treated as loonys by their peers. It’s fascinating.

• [Halloween Jack]

  Dr Ray Stantz: Personally, I liked the university. They gave us money and facilities, we didn’t have to produce anything! You’ve never been out of college! You don’t know what it’s like out there! I’ve *worked* in the private sector. They expect *results*.

• [Miedvied]

  @ninny

  But I was not criticizing the presence of the emotion: just the extent to which you use it to determine your beliefs about reality. That is to say, I wasn’t attacking your love of your kids on the basis that it may be psychotic: I’m attacking your assertion that your kids are the smartest and most good-looking in the neighborhood. Regardless of whether your belief is sincere (or whether that sincerity is a form of dementia), it is not in and of itself any sort of valid evidence for your kids being the smartest and best-looking.

• [RangerGordon]

  The paper itself is interesting and thought-provoking. Full disclosure: I do not reject the possibility of psi phenomena outright — which, to some people, may make me a “believer.” And, because of my own personal biases, I really want to believe that Retroactive Priming actually works.

  Still, I’m approaching these results cautiously. One thing in particular puzzles me (possibly because I’m not an expert in psychology or statistics): In most of the experiments, women test subjects seemed to outnumber men by a large margin. Since sample-group sizes ranged only from 100 to 150, the analyses of specifically male participants’ data were based on very small subsamples indeed.

  And, considering that participants in many experiments were further subdivided based upon whether they were “stimulus-seeking” or not, wouldn’t we expect the results to fall somewhat outside statistically expected values?

  For instance, say we’ve got 100 participants, 30 of them male, and 15 of those responding as stimulus-seekers. How meaningful are conclusions based on a subsample of 15 stimulus-seeking males?

  Beyond that, one thing that always, always turns me off is the use of Lewis Carroll quotes.

  Now, I love Alice in Wonderland and Through the Looking-Glass.

  But these days so many authors talk about believing six impossible things before breakfast, and how you have to run as fast as you can just to stay in the same place, and how things are curiouser and curiouser. Carroll quotes have become cliche, and I’d just as soon read some random passage from Ulysses or even Fight Club. You know, just to keep things fresh.

  Still, is it “false balance” to say that rejecting the whole thing based on one’s conviction that psi results cannot be explained is as biased as fervently embracing it based on one’s desire to believe?

  I don’t think so. The study is not one of physics, but psychology, and deserves to be considered on its merits.

• [Anonymous]

  I had Ghost Busters flashback when I saw those cards. When Ray keeps shocking Egon and not the hot chick during the test. Classic…

• [Davin]

  Given that these are 9 studies over 10 years, there’s going to be some compounding of error based on which studies were cherry-picked. The authors do a great job of listing their stats, but those stats show a rather tiny effect size. Why’d the journal let this in?

  • [Anonymous]

    Why’d the journal let this in?

    I believe it’s because it met their standards. If the article agreed with your opinions, you wouldn’t question the methodology.

    The church suppressed the findings of Nikolai Kopernik… but not because they cared whether he was right, rather because they cared if people thought the church was wrong.

    • [Davin]

      There’s some real problems with the article — the writing style is fine, but for such controversial claims, some rather important information is missing.

      I say this as a Psychology researcher. Psychology, as a field, needs to bolster both its understanding of statistics and its critical eye in the review process.

    • [karl_jones]

      The church suppressed the findings of Nikolai Kopernik…

      Kopernik suppressed himself for many years: worried about the Church, yes, but not actively oppressed by the Church.

• [Richard]

  And now the NYT follows up with a ‘discussion’ on this topic:

  When Science Goes Psychic
  http://www.nytimes.com/roomfordebate/2011/01/06/the-esp-study-when-science-goes-psychic

• [Anonymous]

  The Inquisition is obviously still alive and well and gone viral–religious affiliation or credentials no longer required.

  None are so blind as those who refuse to see. Precognition and many other psychic abilities have already been proved ad nauseum over the past 100 years through the use of standard scientific method-based experiments, including double-blind tests, at leading universities such as Standford, Princeton, and Duke.

  Do your homework. Better yet, do your own experiments.

• [ninnyfriedcheez]

  So, I’ll expect to be written off as a nut, which is why I don’t ever talk about this stuff openly: I am a normal person who has experienced several very specific incidents of precognition through dreams. I hate the controversy around this subject, but there’s a part of me that knows in a hundred years, maybe longer, this will be proved out or at least accepted as fact. I often feel the scientific method is not useful because it cannot prove events that are truly random. And this is clearly something that very few people have experienced. It comes down to this: if you have experienced it, you don’t question it (nor do you talk about it!), if you have not experienced anything like this, you are absolutely positive it doesn’t exist.

  • [Owen]

    You might have experienced something that you believe validates psychic phenomena. But you might be wrong. Humans are fallible. We all believe things that are not true. We all see patterns where none exist.

    That is why the scientific method is important – because we need to filter out what we wish to be true in order to see what is true.

  • [karl_jones]

    It comes down to this: if you have experienced it, you don’t question it (nor do you talk about it!), if you have not experienced anything like this, you are absolutely positive it doesn’t exist.

    An interesting observation; it rings true; thank you for sharing.

    I’m somehow, obliquely, reminded of something that a professional tattoo artist once told me about people with tattoos:

    The difference between people with tattoos and people without tattoos is that people with tattoos don’t care whether you have a tattoo or not.

• [Anonymous]

  Good read:

  Extraordinary Knowing: Science, Skepticism, and the Inexplicable Powers of the Human Mind

  by Elizabeth Lloyd Mayer. She is reading/commenting on the first part of her book here:
  http://www.youtube.com/watch?v=AClVSWvNsWw

• [Anonymous]

  All this pyschic stuff is untrue, so untrue we should never allow it to be tested by scientific methods, it should never even be spoken of. Science is about faith, and the public doubting of the popularity contest that is peer review is blasphemous to that faith.

• [Prufrock451]

  “What are you trying to prove here, anyway?”

  “I’m studying the effect of negative reinforcement on the peer review process.”

  “The effect? I’ll tell you what the effect is! It’s pissing me off!”

• [ninnyfriedcheez]

  @Miedvied and @Owen – Yes, and I may not *actually* love my children. It might just be my fallible memory or a temporary psychosis. But of course, I predicted your responses and I didn’t even have to use my magic psychic powers.

• [querent]

  Right or wrong, no amount of mockery should ever dissuade a scientist.

• [Miedvied]

  Scientists are angrily railing against the ESP paper because it was poor science.

  To give a brief run-down:

  The experimentors used an alpha level of .05. This means that the results had a 5% chance of occuring by random chance. If you’ve ever heard the expression “exceptional claims require exceptional evidence,” you’d understand that a .05 is far too generous.

  They used one-tailed tests. One-tailed tests are only applicable where a phenomenon is only interesting in one direction. For precognition, that is not the case (whether a study subject gets the right answer 100% of the time or the wrong answer 100% of the time, the result is the same: a significant deviation from random chance in response to a future-stimulus). A two-tailed test would have been appropriate; using a one-tailed test was suspicious, but doubled the chance of false positives (from 5% to 10%, effectively, if a bit oversimplified).

  They created extra and unneccasary sample groups. For instance, they tested males and females separately, even though there is no reason to suspect a priori that the two would differ in precognition. Creating twice the number of tests, however, doubles the chance of false positives again (you can correct for multiple trials by choosing a lower p-value, but these experimenters didn’t). Are we up to 20% yet? Yes, yes we are.

  They then ran the tests on a number of variables, some of which were rather suspicious. For instance, men showed no response to erotic stimuli – so they redid the test with stronger erotica. This time the men came back showing a response. There was no reason to throw out the initial results, and in any experimental field with less controversy would have been immediately identified as hunting for false positives when your initial results didnt pan out.

  Moreover, the effect size was tiny (53/47% bias in the precogs, versus the 50/50 you’d expect – I won’t go into the details regarding what was being measured). A small effect size coupled to a suspiciously high likelihood of false positives and a generous p-value given the phenomena under investigation all adds up to *utter and total nonsense*.

  No, scientists aren’t ripping this paper apart because they’re too lazy to go out there and do “the scientific method.” They’re ripping this paper apart because it’s a joke.

  • [Ambiguity]

    The experimentors used an alpha level of .05. This means that the results had a 5% chance of occuring by random chance. If you’ve ever heard the expression “exceptional claims require exceptional evidence,” you’d understand that a .05 is far too generous.

    I disagree. I don’t think there should be two standards for inclusion into a journal (one for science the editors like, one for science they don’t). If p < 0.05 is an acceptable level for research within the domain, then it is acceptable.

    It would be absurd to believe in precognition based upon a 5% likelihood, but it's not absurd to accept such a paper for publication.

    Let's say for the nonce that the effects are real but small. The only way that you could empirically verify them is to accept such work. When (if, given the current climate) the experiments are replicated a meta-analysis can be performed to increase the statistical significance. This could never happen if the paper was rejected outright, and that's bad science.

    If, on the other hand, the effects aren’t real, replication/meta-analyses will show this too.

    So, everybody wins.

    • [Miedvied]

      I did not say that inclusion to the journal should be based on p-value. I was criticizing the value of the paper, and the intentions of the authors, in claiming to have found something significant (in the colloqiual sense) at a p-value that regularly pops up in random sampling error.

      As to replication: the publication author claims that he’s been collecting hundreds of subjects over the course of ~9 years to gather the amount of data needed to “prove this conclusively.” It’s a quirk of statistics that with a big enough sample group, you could always get whatever p-value you want for pretty much anything you want — it’s significant that at a decent sample size, his experiments showed a number of positive results nearing the generous cut-off. Regardless of which, this doesn’t justify a p-value so generous that *he was gauranteed to get positive results*.

      I guess I made it more implicit than explicit in my initial post. Let me elaborate: p @ .05 gives 5% false positives. One-tailed means we don’t need to deviate nearly as far from the mean; we’ve effectively bounced up to a 10% false positive rate, if we don’t modify the p-value downwards (which we don’t). We split the trial into males and females, creating twice as many trials without a valid bio/psych reason for doing so: doubled the likelihood of false positives again. Up to 20%.

      I forget now the exact number of categories: I believe that each gender was tested in ~4 areas. Any of them would grant us a “positive” worth publishing, right? Additive probability indicates there’s an 80% chance of at least 1 false positive. There were in fact 2 positives in the entire study. More or less what you’d expect given the false positive rate.

      This is beyond the fact that there’s a difference between exploratory and confirmatory studies. We come up with data in informal explorations all the time; it then requires more rigorous testing to confirm. You can’t blend one into the other, because then you bias your confirmatory studies with what may be simple coincidences to begin with. It’s poor practice. This study did an ad-hoc “erotica” alternative with the males after their first trial failed, got a result they wanted, and *kept it and reported it as confirmatory*, where it was clearly exploratory.

      There’s just poor practice all around.

      And, no, p< 0.05 is not a random binary acceptable/unacceptable. Statistics is more complex than that, and it’s those sort of vast misunderstandings that make people think statistics is synonomous with lying. Most stats ppl these days are migrating towards Bayesian analysis for a reason.

      Signed: a biostatistician grad student with significant weariness and wariness of psychometrics.

      • [Anonymous]

        Thanks for posting the post I wanted to post. If they’re making that many concessions to get positive results, they should have an effect size of a whole lot more. I’d say BB readers are more interested in finding evidence for ESP than most, but this isn’t it. Even as a marketing academic, if we used those kinds of lax standards, there’s no way we’d get accepted.

        One-tailed tests says it all really.

        Perhaps a more interesting test would be to use a sample of self-professed psychics, or those who claim to have had psychic experiences, and then to see what kinds of effects they can produce. Even from a complete outsider’s view, it seems strange that they confuse predicting the future with having an influence on the results of the future- these need to be separated out. And where’s the double blinding?

        Even from the social sciences, this research seems flakey.

• [Anonymous]

  Let me know when the research is published on ‘Lot Six’.

• [Ito Kagehisa]

  For people like myself, who do not assign supernatural power to the ritual of peer review, this is going to be really funny.

  Perhaps Dr. Bem should have had a press conference before publishing, eh?

• [Beryllium]

  Dear Outraged Scientists (and media cohorts):

  Don’t get angry. Get testing. It’s science, bitches. Apply the method, don’t armchair hypothesize.

• [AirPillo]

  Well publishing research isn’t intended to put the journal’s reputation behind what is published and say it’s all true. The point is making sure the contents adhere to the scientific method so others can discuss, scrutinize, and repeat the experiment.

  There may well be flaws in the research and the findings, but that’s exactly why you publish it, so that the research, flaws and all, is given exposure and can be analyzed and confirmed or debunked.

• [deckard68]

  There is always a comment along the lines of Ray Hyman’s “I think it’s just an embarrassment for the entire field” when research into edgy subjects is published. Hyman and his ilk need to recognize that embarrassment is likely a pretty good sign that one has failed their responsibility to be impartial.

• [Anonymous]

  “It startled him even more when just after he was awarded the Galactic Institute’s Prize for Extreme Cleverness he got lynched by a rampaging mob of respectable physicists who had finally realized that the one thing they really couldn’t stand was a smartass.” – Douglas Adams, HHGG

• [GreenJello]

  Is this really news? People get PO’ed everyday, for a variety of reasons. What’s really interesting, IMHO is the content of the paper, not all the controversy….

• [tyger11]

  Unless these outraged scientists have found a flaw with the experiments because they have REVIEWED the paper themselves, then they’re just doing what they’re complaining about. If someone who has actually reviewed the paper has a problem, by all means, let’s hear it.

  • [coaxial]

    It’s simple. It’s poor statistics, by reviewers that either don’t understand statistics themselves, or simply rubber stamped it without checking the math. My bet is both.

    Which is kind of the larger point. The review committee (and the field in general) doesn’t understand stats.

• [Anonymous]

  I know how this is going to turn out.

• [Steven Stwalley]

  Perhaps instead of getting pissed off that research these angry scientists don’t believe could possibly be true is being taken seriously, the angry scientists should try something known as “the scientific method” on the research. Who knows, maybe the results may not actually jibe with their preconceived notions… an outrageous idea, I know.

• [Rob]

  Ray Hyman is on the executive board of the Skeptical Enquirer. He’s really biased. I saw a speech he gave in Oregon about parapsychology research… in it he wasn’t compelling, and at the end during the Q&A session couldn’t stand up to the questions.

  He’s hardly a unbiased critic. The advice he used to give people was pay no attention to parapsychology research, because it takes advanced degrees in science to understand why their claims aren’t true… at this talk I was at he stepped back from that and gave his listeners talking points to use when confronted by a ‘believer’.

  Point being.. they have already decided what they think, and they run a campaign to promote their view.

  Parapsychology research is quite interesting, if you take a look… Ray and his skeptic cronies advise against it though.

  • [Brainspore]

    Parapsychology research is quite interesting, if you take a look…

    I actually find the topic very interesting but not for the same reasons that ESP believers do. For me the real show is watching the lengths that otherwise intelligent people will go through to convince themselves that some people have magic powers.

    • [Ambiguity]

      I actually find the topic very interesting but not for the same reasons that ESP believers do. For me the real show is watching the lengths that otherwise intelligent people will go through to convince themselves that some people have magic powers.

      In all fairness, it’s also interesting to see the lengths that otherwise intelligent people go to dismiss the paper, outside of scientific protocols.

      I’m a big fan of Sir Karl Popper when it comes to thinking about the scientific method, but one of the biggest criticisms of his work is that “science doesn’t really happen that way” (not that it shouldn’t happen the way he describes, but that it doesn’t). Events like this show these criticisms to be not entirely without basis.

• [farkinga]

  Bem has a p > 0.05 chance of outright lying here. He’s known to some in the social psych world as a “prankster” (or “fraudster” if you’re not similarly “humorous”).

  @Miedvied: psychometrics are good science.

• [Jer]

  A take on why the research should/should not have been published in the journal is here: http://theness.com/neurologicablog/?p=2701

  And for those hoping these experiments really do prove that psi powers are real – no one has been able to replicate the results yet. That doesn’t mean that the results were rigged, it just means that perhaps Bem and co. had an anomalous result.

  (According to the link above the effect was tiny anyway – 53% vs. the expected 50%. Not a psychology researcher, but I do research for a living and that’s the kind of minor-but-significant result that makes me suspect unconscious bias on the part of the researcher and cries out for attempts at replication before anyone even tries to find an explanation for it.)

  • [Ito Kagehisa]

    Although I found your own comment here to be intelligent and well reasoned, the column you linked to made me laugh out loud.

    • [Brainspore]

      I guess my sense of humor is off today. Which part of Dr. Novella’s article made you laugh?

      • [Ito Kagehisa]

        Ithe early part of the essay, Dr. Novella was objecting to the virgin purity of peer-reviewed journals being sullied by pseudo-science. Because, you know, once it gets into a journal, it poisons the future. Once you’ve read a hundred years’ worth of scientific journals that bit seems pretty hilariously ironic.

        To give Dr. Novella his due, though, much of what he wrote is well reasoned, and his opinion of the value of Dr. Bem’s data-gathering is impossible to disagree with, and all of it is well written. I suppose I should have made that clear before I commented on my amusement at his overly worshipful view of mainstream scientific literature. Dr. Novella correctly wants to see Bem’s experiments disseminated, reproduced and discussed, rather than dismissed. He points out there is more here than just a possibility of psi, there is also the possibility of learning something about testing human responses and analysing the results.

        I just think his idea of relegating Bem’s research to a special literature ghetto – so nobody will think it is scientifically approved – is funny!

• [Anonymous]

  I have read the paper. In my opinion it describes some very simple and cheap experiments, that can be easily replicated all over the world. Let’s do it, follow the protocol, collect a sample size of 10 000 or more, and see what happens. The samples reported in the paper do not “prove” psi by themselves. They only set a preliminary, tentative effect size range that we may expect to get in those experiments.
  I would be very excited if the effects were confirmed. It would be possible to go on testing time paradoxes! Suppose that we rig the random generators against the subjects, for example by putting the erotic pictures under the curtain that has *not* been chosen with a probability higher than 1/2. Will then the psi disappear?

  • [Miedvied]

    A couple of labs have already conducted tentative replication trials. They all failed. It is looking increasingly like Bem’s positives were, in fact, random false positives.

    Though frankly, given that the phenomenon is (1) extremely unlikely, (2) so far disproven in every experimental setting (2a) except those with visibly broken methodology, such as Bem’s paper… why in the hell would we waste the resources on repeating this *yet again*? This has already been disproven more or less as thoroughly as something can be disproven within the limits of scientific epistemology, at least in its current incarnation (perhaps some other variant of psi may indeed be real – but certainly none of those touted as real by current believers, as the crop has been well exhausted experimentally).

    At some point, it pretty much comes down to “people who want to believe” vs. “people that can evaluate evidence.”

    • [Ito Kagehisa]

      At some point, it pretty much comes down to “people who want to believe” vs. “people that can evaluate evidence.”

      I can’t tell you how many times I heard people say that research into high-temperature superconductors was a dead end, and that basic physics proved that any such research was “wasting resources”. (Now, of course, it was brilliant and physicists have no problem with it.)

      I’ll counter your duality with a trinity – “people who believe without evidence”, “people who disbelieve without evidence”, and “people with open minds”. But I don’t really think that’s any more correct than your statement. People’s belief systems are complicated and interrelated, and (as you have noted) epistemology is a slippery subject since all data is filtered through our fallible human physiology.

      Personally, I would like to see reproducible evidence of clairvoyance, because I have observed phenomena that I cannot explain any other way. Without reproducible evidence, however, I have to acknowledge that my observations could be incorrect, and that some other explanation may be available that I’m not smart enough to figure out. So, I will happily receive new data; but for now it seems wise to act as if anything that seems to be clairvoyance is in reality random chance, fraud, or delusion.

• [MrJM]

  Get out there and disprove the research scientists.

  And while you’re at it, disprove the flat-earth theory and Lamarckism.

  • [singingdragon]

    Hey, don’t be too hard on Lamarck. He actually had almost everything right in his formulation of evolution. His only mistake was in thinking acquired traits were passed on; the rest of his theory is practically identical to Darwin’s, and Darwin didn’t have any inkling of the actual mechanism of inheritance, either.

    For that matter, modern studies of epigenetics are starting to make it look like Lamarck was actually partially right that acquired traits can be passed on by some biological mechanism. It doesn’t explain most inherited traits, but it happens.

    Short version: actually, it’s worthwhile to test things we’re absolutely sure are true, because sometimes we’re at least a little bit wrong.

    • [Beelzebuddy]

      Short version: actually, it’s worthwhile to test things we’re absolutely sure are true, because sometimes we’re at least a little bit wrong.

      To add to your post, we’re always a little bit wrong. About everything. Every scientific theory is at best a really good approximation of what’s actually happening. The best we can do, given two competing theories which we know are a little bit wrong, is to find a test which tells us which is less wrong, then go with that. That’s the lesson I’d like to stress – all theories are wrong, but some are more wrong than others.

      The mistake, which Roscoe’s postscript beautifully illustrates, is when we conflate “a little bit wrong” with “total fucking nonsense that we really wish were true.” It’s hard to stay scientific when you have a vested interest in a particular result, be that psychic mind bullets, creationism, or anti-vaccine woo.

    • [Ito Kagehisa]

      SingingDragon, your last sentence is heresy. True Science must not be questioned; the possibility of psi phenomena must never be researched! This paper must be supressed – it could lead the proletariat astray!

      Singingdragon is a witch! Burn her! She turned me into a newt!

• [Anonymous]

  What is happening in the world? I feel like I’m On the Road.

• [Anonymous]

  Tyger11: I just read the preliminary version available as pdf. There are methodological flaws including (as far as I can tell, not all methods are described in suitable detail imo):
  - eliminating ‘outlier’ data points or points that didn’t agree with the experimenter (for instance, data from 3 subjects was thrown out of one experiment because they got at least 25% of picture ratings ‘wrong’, meaning that they disagreed with the experimenter as to whether it was a ‘positive’ or ‘negative’ picture)
  - retroactive subgroup analysis
  - altering protocols partway through (e.g., men didn’t show a statistically significant result in guessing erotic pictures, so they went and got new sets of pictures and tried again; when they got a marginally positive result, they concluded not that it was due to chance on that trial but that they finally had a suitably ‘strong’ set of erotic pictures for men)

  That’s on a pretty rapid read, and I wasn’t planning to post a comment, so I didn’t keep notes. But I have no doubt someone (Ben Goldacre?) will compile a list fairly soon.

  Disclaimer: I’m not a psychologist, I’m a brain surgeon; I’m not familiar with the psi literature or their usual research methods. But this paper would fail standard critical analysis at our journal club.

• [semiotix]

  A software program randomly posted a picture behind one curtain or the other — but only after the participant made a choice. Still, the participants beat chance, by 53 percent to 50 percent, at least when the photos being posted were erotic ones. They did not do better than chance on negative or neutral photos.

  So in other words, college kids will warp the fabric of reality itself to see porn.

  Trivial, obvious, and already well-established. Bad study!

• [ryxxui]

  Statistics are a funny thing. Just as an example- take the experiment where people had to guess whether the image would appear on the left or right screen. The probability that every single person would guess every image correctly is exactly the same as the probability that every single person would guess every image incorrectly. That’s just how randomness works. I’m sure that if the experiment was repeated multiple times with the same population, you’d get wildly different results every time. I’m sorry to anyone who actually has bought into either this paper or the idea of ESP in general, but you have to agree that the results of this paper are not nearly statistically significant enough given the unlikeliness of the claims they are making. Extraordinary claims require extraordinary proof, and these results are not going to be repeatable.

• [mausium]

  “Dear Outraged Scientists (and media cohorts):

  Don’t get angry. Get testing. It’s science, bitches. Apply the method, don’t armchair hypothesize.”

  Who is going to request grant money to verify these tests? They’re not worth following up on. Also, “it’s science, bitches”? Should “science” respond to every quack that’s managed to get in a position of power?

  “My understanding is that many in the science and engineering ranks of Princeton’s faculty long considered PEAR an embarrassment, but they did come up with some rather interesting findings.”

  PEAR believed that PEAR’s findings were interesting. Princeton’s science and engineering departments did not. Once the donations stopped flowing in, PEAR rotted.

• [Micah]

  Sounds kind of like the work of the Princeton Engineering Anomalies Research lab, which tested the ability of thinking to influence machines (rather than the ability to pre-conceive what the machines would do). My understanding is that many in the science and engineering ranks of Princeton’s faculty long considered PEAR an embarrassment, but they did come up with some rather interesting findings.

• [sirkowski]

  I can turn lead into gold. Prove me wrong scientists!

  I’m Santa Claus too.

  • [Ugly Canuck]

    Hey that’s cool…I myself can do something similar. What I can do, I can turn gold into all kinds of other things.

    Give me your gold, and come back in a few days. I’d prove it to you. You’d see.

• [Miedvied]

  @ Farkinga:

  I don’t have a grudge against psychometrics as a science, so much as the statistical illiteracy that pervades it. As dodgey as the paper in question was, its practices were not significantly at odds with the standard statistical analyses in the field. More than a few folks in my program saw that paper and the instant response was: “And that sums up everything that’s wrong with psych studies.”

  • [Davin]

    I must agree here — as a Psychology researcher, the pervasive lack of proper statistical education in our field is pretty disheartening. And leads to me being depressed for my field. Because we really so heavily on measuring constructs rather than measuring “things”, it’s important that our stats knowledge is airtight. That’s generally not the case. We’re trained to throw things into SPSS and to clap our hands when the p-value is significant. When the sign of “good” stats reporting is just having effect sizes, well, the field’s in trouble.

• [Daemon]

  I’d be willing to bet that nobody would complain about his methodology if his results were in line with what is currently accepted as The Truth within the scientific community. Half of them are mostly upset because they don’t want their work to appear in the same journal as a paper with those findings. You just can’t have those results and be accepted… He’s basically become that weird kid in highschool that nobody wants to be seen talking to.

• [Beelzebuddy]

  Looking at the paper, that’s some mighty insignificant significant effects they’re positing there. I’m not familiar enough with psych experiment methodology to critically comment on their experimental setup, but I am enough of a cynical bastard to find the way they stacked their analysis a little bit suspect. Significant effects can happen wholly by chance, often with hilarious results. One of the biggest mistakes modern science makes is thinking p values are the end-all be-all of experimentation.

  A little independent verification (via a colleague in another psych department, say) would have needed doing before it passed my muster, but again, I’m not psych. For all I know, that’s what this publication is intended to provoke, the way physics has arxiv and biology has conference poster sessions.

• [LukeWhite]

  He should have probably seen this coming.

• [Deidzoeb]

  ‘All four decided that the paper met the journal’s editorial standards, Dr. Judd added, even though “there was no mechanism by which we could understand the results.” ‘

  Wow. In other words, the journal’s editorial standards include publishing papers for which they have “no mechanism” to understand the results.

  • [Ambiguity]

    Wow. In other words, the journal’s editorial standards include publishing papers for which they have “no mechanism” to understand the results.

    I don’t see that as a bad thing,especially when it comes to experimental (as opposed the theoretical) science. Empiricism is, after all, one of the cornerstones of science.

• [treacle]

  I wish I’d known about the “study after test” improvement potential when I was in university…

• [Miedvied]

  @ninnyfriedcheez:

  Have you ever had a bad reaction to meds, that produce a temporary psychosis? I have.

  What you said *applies to my experience precisely*. While psychotic, there was absolutely no doubt that what I was experiencing was true, and my feelings sincere. The latter certainly were: there were very real molecules in my brain producing aberrant patterns.

  Luckily, I regained my sanity afterward, and could only identify my previous psychosis by comparison – in hindsight.

  Yet, as you say, during the experience, I didn’t question nor doubt it. Luckily, my physicians – relying on the scientific method and standards of evidence, rather than the strength of my belief – deduced I was suffering a psychological reaction to medication and removed it from my treatment plan.

  Strength and sincerity of experience play no role in determining what is true. Especially when the experience in question is entirely internal and subjective by nature.

• [Anonymous]

  Counterpoint: “The Decline Effect”, a piece in the current New Yorker, about potential problems with the scientific method (which might explain this):

  http://www.newyorker.com/reporting/2010/12/13/101213fa_fact_lehrer?currentPage=all

• [Beelzebuddy]

  It also looks like the good people of the JREF have been tearing this paper a new one for months now.

• [Teller]

  This paper, like Assange’s forthcoming autobiography. will be “one of the unifying documents of our generation”.

• [Anonymous]

  Ahhh, more a priori thinking from people who claim to be critical thinking. You have results that “can’t be true”? Look for a flaw in the experimental setup or interpretation of the results, or change your mind about “can’t be true.” Deciding what is true and what is not true before investigating is as bad as consulting Aristotle about physics.

• [Anonymous]

  I quite enjoy this bit from the rebuttal published alongside it, laughing at the “but, but, quantum mechanics is weird toooooo” whinging:

  “Some argue that modern theories of physics are consistent with precognition. We cannot independently verify this claim, but note that work on precognition is seldom published in reputable physics journals (in fact, we failed to find a single such publication). But even if the claim were correct, the fact that an assertion is consistent with modern physics does not make it true. The assertion that the CIA bombed the twin towers is consistent with modern physics, but this fact alone does not make the assertion true.”

  Other than that, it argues that this article provides evidence that modern psych needs to update the requirements for statistical significance- which, coming from an unrelated field, seeing that they accept one-tailed t tests at p < 0.05 on sample sizes of 15 or so, is bleedingly obvious.

• [Ambiguity]

  Sorry — there was a parsing error in the above post owing to the < in the HTML. What the above should say is:

  “If p < 0.05 is acceptable in the domain, then it is acceptable.”

• [Roscoe]

  Just came across a reference to some of Dr. Bem’s other studies, and he’s not pitching outlandish claims… isn’t science supposed to study what we don’t yet know, not the other way around?

  He had another study, that’s been replicated with similar results, that I think has a lot to say about just how much we don’t know about the subtle powers humans have. Subject A sits in a room with a blood pressure monitor. Subject B is isolated down the hall. Subj A has no knowledge of Subj B. When B is told to think a happy loving though about Subject A, A’s blood pressure drops. When B is told to think angry hateful thoughs about B, his BP jumps in response.

  We’ve also seen the results of Robert G. Jahn’s experiments with the ‘plinko’ machines – where someone can sit in front a machine with falling ping pong balls. With no thoughts, it will make a pretty regular bell curve. When someone directs their thoughts to steer the balls left or right, there will be a measureable (though not drastic) shift of that curve the direction of that persons intentions.

  I’m amazing how the story about this kind of stuff becomes a story about people slamming it, rather than the content and results itself.

  Humans do have psychic abilities, they’re strongest in children before they’re taught it’s crazy, and everyone should embrace this as yet another of those fanstastic mysteries that still abound.

  PS. Science: have you figured out how gravity works yet? just cause you can’t say how, I still believe in it’s results!

  • [Brainspore]

    Yet oddly all those results disappear whenever the studies are double-blinded.

    • [The Life Of Bryan]

      Then clearly double-blinding has been scientifically proven to affect psychic ability!

      • [Brainspore]

        On a related note, my family hates watching “Peter Pan” with me since Tinkerbell always dies when I refuse to clap my hands.