The McGurk Effect

This video demonstrates how looking at someone's mouth movements affects the way we hear what they are saying. The man in the video is saying "bah, bah, bah," but when the same audio recording is played while he mouths out "fah, fah, fah," it sounds like he is saying "fah, fah, fah."

The McGurk Effect

Mark Frauenfelder is the founder of Boing Boing and the editor-in-chief of MAKE and Cool Tools. Twitter: @frauenfelder. Come and hear Mark speak at the ALA conference in Chicago on July 1.

MORE: Weird

More at Boing Boing

Ants and Stars: Bruce Sterling and Jasmina Tesanovic visit the Sardinia Radio Telescope in Italy

The Snowden Principle

Anonymous

Weeelll, the effect tends to work less well with subjects from countries in which foreign TV programs are dubbed, because people in these countries are accustomed to hearing something that does not correspond to what they see.
mermaid

Bet if you tried this with a Tagalog speaker it wouldn’t work.
Anonymous

Olive Juice lipreads as “I love you”
- Donald Petersen
  
  Olive Juice lipreads as “I love you”
  
  As does “elephant shoes.”
  
  I deal with ADR and sound sync as part of my work in TV post production, and it seems to me, too, more like the “fah” picture is slightly out of sync with the audio, as opposed to the B sounding like an F.
OldRipbeak

That’s bucking cool!
HarlanH

That’s not the McGurk effect. The McGurk effect is where you perceive a sound that’s neither what you see or what you hear. The standard example is when you hear “ba” and see “ga”, you often hear “da”.
- Anonymous
  
  Both of those count as the McGurk Effect. Consonants are composed of three phonological dimensions: voicing, manner, and place of articulation. When you’ve got both auditory and visual inputs, especially when they conflict, the auditory signal contributes the voicing and manner information, and the visual signal contributes the place because that’s much harder to tell from sound alone. The McGurk Effect is just the combining of these two signals, which sometimes results in a percept that’s different from both the auditory and visual signal, and sometimes that’s the same as the visual.
  For example,
  aud /ba/ + vis /ga/ = perceived /da/
  and
  aud /ba/ + vis /da/ also = perceived /da/
  
  -A cognitive scientist
- Xopher
  
  I’d be reluctant to argue with the UCR guy, myself, but the effect you describe makes sense too. /b/ is at the front of the mouth and /g/ at the back, so splitting the difference (/d/ is the closest stop to the middle of the mouth) is a fairly sensible thing for the brain to do.
  
  I have to watch with captions on, and this is related to the reason why.
Sparrow

I still hear bah, but I get a distinct impression that the ADR is off. I really notice this effect in movies when the overdub doesn’t match what was originally said or when the sound isn’t quite in sync with the mouth movements.
Anonymous

I need to figure out how to write to this man. My daughter and I, both neurologically typical, were astonished by this. But my high-functioning autistic son always heard “bah bah bah” no matter which face he saw. A possible test for autistic spectrum disorders? Or just something weird about my son, in particular?
- SamSam
  
  My reaction to the video being an empathetic sense of “feeling” the sound “eff” would make me say, as a non expert (but a BA in cognitive science, for whatever that’s worth) that it’s completely plausible that there would be a significant difference in people with autistic spectrum disorder. It’s worth researching more, and I bet this person would be interested in researching it if there isn’t anything already.
Anonymous

Bah, humbug…
Bucket

Hm.

I still hear ‘bah’ and get the feeling like the video is out of sync with the audio.

I also have a really hard time conversing in loud environments, even though a recent hearing test showed that I have pretty good hearing.

I wonder if the two are related.
Anonymous

Doesn’t seem to work for me. I hear the bah both times. Just doesn’t look like it is him saying it the second time.
Anonymous

Is this guy David Cross’s more erudite clone?
andmit

What does it mean if all you hear is “ba” the whole time? I didn’t get what was going on when my boyfriend first showed me this because I don’t hear “fa” at all. Even when he made me stare at his mouth. All I hear is “ba”. Someone said it means my brain isn’t working right.. O_o
mellowknees

I was really hoping that this was a post about the coach from Home Movies.

I am disappoint.
- Trent Hawkins
  
  As am I. I was hoping to see something about buying swords at 3am.
marc anthony

At 56 seconds, when the guy first starts changing the movements of his mouth, I detect a slight “f” sound, even when completely looking away from the video. I don’t think you can change the shape of your mouth without slightly reshaping the sound. The effect is increased with the visual, but I detect the change without it.
- Anonymous
  
  @marc anthony
  I agree it does seem to change… I can hear a “popping” noise when he does the Bah, and it seems the audio has been edited in the second loop to mute the “pop” of the b. Seems one time he actually does say “Fah” after all. I think the guys in the studio maybe took a liberty to increase the effectiveness of this video? But the point it well made.
  
  Which, like HarlanH says, is a bit misconstrued, because if you wiki McGurk Effect it relates that two nodes of communication can be misconstrued to create a third unrelated effect; which is much more interesting if you think about it, than just a confused “flip flopping” (or blip blopping?) between the two F and B nodes.
- Anonymous
  
  You do understand that both videos use the “bah” audio? Interesting that you hear a change that isn’t there. Or if you look away when it shows him saying “bah”, do you hear “fah” there too?
Robert

Betrayed! I closed my eyes and heard B both times. Then I opened them and heard F the second time. Betrayed, I tell you!
SamSam

That was great. When I was watching the supposed “fah fah fahs” after having been told what was happening, I was acutely aware of my own mind supplying the missing Fs. It was less an auditory sensation as a physical one — by seeing his mouth make an F sound, I was making an F sound myself. I was almost feeling what I would be feeling if I were the one making the Fs. I had almost the same sensation watching the video with the volume off, but not as strongly.
recoiled

I always thought this was called the Two Party Effect
RHK

So that’s how politicians do it!
- Sork
  
  “So that’s how politicians do it!”
  
  Read my lips!
yragentman

“… a co-hear-rent view of the world…” ugh
JohnBerry

Opening and closing my eyes really drove home the point – you can’t trust your eyes (or ears, or (sometimes) brains). Amazing!
shava

I have crosstalk deafness — a cognitive deafness that means I can’t parse one strand of speech in the presence of conflicting signals, like a person who can’t make out the separate instruments playing in a piece of music. As a result, in a place where many people are talking or there’s non-white background noise — a subway car, a networking meeting, the lobby at a theater — I have to lip read.

Whether it’s strong conditioning to lip reading, or the cognitive bits, or both, this illusion doesn’t work on me at all. Interesting.
rebdav

Woah!!
Its like my visual center snuck some shoes, a nail file, and a bottle of milk onto my auditory center airplane and demanded that we fly to Cuba!!
wylkyn

How weird. I didn’t experience the illusion. What I got was a feeling of dissonance. I was hearing the “b” sound and seeing the “f” sound, and my brain was setting off “this is wrong” alarms, even though I was trying to go with it. I wish that I had seen this before it was explained so my rational brain would have been taken off-guard. Just like when the hypnotist couldn’t put me under. Stupid rational brain always spoils my fun. :(
Crispinus211

A similar effect happens with the subtitled STSanders “shreds” videos — I know Mick Jagger is singing (for example) “You make the grown me cry,” but the dubbing and the subtitles trick me into something else.
Anonymous

@mermaid
I had a college professor who studied anthropology in Micronesia. His name was Bill but the local people there didn’t have a “b” or short “i” sound, and never ended words with a consonant. So “Bill” became “Peelly”. Even when he said his name, they would “hear” their version in their mind.
Bill Barol

Thanx for the nightmares, Mark!
dross1260

I prefer SeÃ±or Wences
Anonymous

Anyone also hear two people saying something when they do the split screen? I thought at first they had laid down two audio tracks over each other.
pepik

This strikes me as the result of a kind of error correction technique. Very often, it can be hard to discern the initial sound of a word or syllable starting with the f sound. Seeing someone’s mouth make the sound helps fill in the gap.

I have had the distinct experience of hearing sounds that don’t quite make sense, and having my brain go back and ‘fix’ the sounds into a word that makes sense in context. It’s something very cool that our brains can do, I think.