Hearing lips and seeing voices

This video explains the weirdness of the McGurk effect, a perceptual illusion demonstrating that understanding speech is not just about what we hear, but also what we see. You can learn more about the McGurk effect at Yale's Haskins Laboratories dedicated to the science of the spoken and written wordl. (via Imaginary Foundation)

Whistling Speech

I really love the research that they're doing over at Yale's Haskins Laboratories: instead of studying speech perception and production in terms of faithfully replicating alllll of the sounds we make with our mouths, (like the minute clicks, pops, and hisses of consonants), the team is proposing that all we need to understand speech is to track and re-create a few select resonances of the vocal tract. I like to think of speech production in this context as a series of bottles with varying levels of water in them--the mouth is one bottle that changes pitch resonance when you move it to open it or close it, the nasal cavity another, and so on throughout the vocal tract. It ends up sounding like a bunch of complicated melodies that are then combined into a complex micro-tonal harmony, a.k.a., we're all better at perceiving and making music than we think we are!

The examples below break it down into isolated sine-wave patterns that you can combine yourself to build a sentence. What do you think? How easily can you hear words emerge?


Tone combinations

If you like this, you can go here for more interactive demonstrations, or check out this great sine-wave-synthesized Robert Frost poem.

Thanks to Robert E. Remez, as well as Phillip Rubin and Jennifer Pardo at Haskins Labs for allowing me to embed their work here.

Coming up, I'll be writing about a cool ethnographic example of a language that actually uses something like this in practice!

Wendy Carlos and color perception

andy-gilmore.jpgIn addition to being one of the most historically significant pioneers in electronic music, Wendy Carlos is fascinated with how people see and hear. I am, too. She has been conducting experiments on color perception for over 50 years. Wendy created a cool little red/green color lightbox and a series of pages that show how two monochrome images can create full-color images when combined. She explains the origins of her interests:
Interestingly enough, most primates which evolved in Africa, Europe and Asia and environs posses a similar wide range as ours, while those which evolved later in "The New World" of the Americas usually have the narrower range of human color deficiency. The technical distinction is between: "trichromats (human and old-world primates)" and "dichromats (new-world primates and the common human color deficiencies)." Anyway, I built a lot of amusing devices way back in grade-school that allowed me to tinker with mixing various colors, both with paints (subtractive mixing of: magenta, yellow and cyan) and with colored lights (additive mixing of: red, green and blue). I read everything on color I could get my hands on, and with many years of more or less scientific experimentation, I thought I knew a bit about the subject. But I was wrong.
Experiments in Color Vision (Wendy Carlos)

Wendy's music (recommended: 'Tron' and 'A Clockwork Orange' soundtracks, Switched-On Bach, and The Well-Tempered Synthesizer)

Image: Another amazing Andy Gilmore design via bridbird.com.