Using gestures to interact with surfaces that don't have screens

A joint Disney Research and CMU team have produced a demo showing gesture controls on a variety of everyday, non-computer objects. The system, called Touché, uses capacitive coupling to infer things about what your hands are doing. It can determine which utensil you're eating your food with, or how you're grasping a doorknob, or even whether you're touching one finger to another or clasping your hands together. It's a pretty exciting demo, and the user interface possibilities are certainly provocative. Here's some commentary from Wired UK's Mark Brown:

Some of the proof-of-concept applications in the lab include a smart doorknob that knows whether it has been grasped, touched, or pinched; a chair that dims the lights when you recline into it; a table that knows if you're resting one hand, two hands, or your elbows on it; and a tablet that can be pinched from back to front to open an on-screen menu.

The technology can also be shoved in wristbands, so you can make sign-language-style gestures to control the phone in your pocket—two fingers on your palm to change a song, say, or a clap to stop the music. It can also go in liquids, to detect when fingers and hands are submerged in water.

"In our laboratory experiments, Touché demonstrated recognition rates approaching 100 percent," claims Ivan Poupyrev, senior research scientist at Disney Research in Pittsburgh. "That suggests it could immediately be used to create new and exciting ways for people to interact with objects and the world at large."

Disney researchers put gesture recognition in door knobs, chairs, fish tanks


    1. I don’t know about that. Microsoft doesn’t make hardware, so hardware really doesn’t embarrass them. Apple has been using multi-touch gestures without screens for years now in their trackpads. You could say that needing hardware like a trackpad as opposed to just using the body or something else hooked up to this is embarrassing, but that doesn’t quite mesh with the whole “total product package” philosophy. It does fit in nicely with the “as intuitive as possible” philosophy, though, so who knows?

      Edit: I stand corrected. That’s what I get for trying to stand up for Microsoft, I guess.

      1.  “Microsoft doesn’t make hardware”

        Tell that to the XBox guys.  Especially the guys who have spent some serious energy working on the Kinect.

        1. Not one of wikipedia’s best efforts. It reads like a marketing brochure. The bold text bothers me.

    1. “You watch this door, it’s about to open again. I can tell by the intolerable air of smugness it suddenly generates.” – Marvin

  1. Hahaha. Wtf with the “teach your kids to use the correct utensils” suggestion?
    It felt like a bit from “Look Around You”.

    1. All jokes aside, the concept could have a lot of applications for rehab after a stroke or brain injury, not to mention cognitive disability.

    2.  what it’s going to lead to is a generation of kind constantly poking inanimate objects in the hopes that it would tell them what to do.

  2. So now you get to look like a wizard when changing songs on your phone or making a call…

      1. At least in this case, I get the last laugh. The gestures I’ve been using all these years have saved me from repetitive strain injuries that many others have by now. Not to mention I’ve been getting much more work done in less time. Those wrist-crippled stubborn sumbitches should have listened to me.

  3. “Negative reinforcement” (the removal of conditions or stimulus as a consequence) does not mean “punishment” (the application of undesired conditions or stimulus as a consequence). They’re often related, but the term “negative reinforcement” has very specific meaning in psychology.

  4. This does seem like a logical extension of the touchscreen mentality. Creeps me out, though.

  5. Finally.  I’ve been waiting for an advancement in proper-food-implement-training technology.  This could be the real shit.

    1. Not to sound *too* cheeky though.  I mean, it seems like progress in capacitive sensing is pretty obviously sellable, with lots of obvious marketable applications.

  6. Child dips spoon in cereal bowl:  reasuring DING!   Child flings spoonful of cereal across the room.  Child dips spoon in cereal bown to refill:   another reassuring DING!  This will end well. 

  7. Unfortunately, I bet the first use Disney will put this to will be improved biometrics for park passes, or stern automated warning for kids that drag their hands in the water on a ride, or put their elbows on the table in the Prime Time Cafe.
    (which would be kinda cool if they had a Rosie the Robot roaming around the place).

  8. Instead of nonense gestures for controlling your media player/cell phone, why not have the device just recognize ASL gestures. That way everyone gets to learn ASL while they learn how to control their devices. Seems like a wasted opportunity if they just make up their own gestures to me.

  9. It would seem to me that based on the examples they give, that it is actually “just” measure the amount of skin contact with the contacted object. Perhaps different areas of skin have different, um, electrical properties but it is not so much gesture recognition in itself. I think the next step is to do a lot of interaction design type stuff to see what configurations of multiple zones could produce more meaningful information. Also the variety of gestures and body types, yadda yadda

    Also, couldn’t you make the “doorknob” itself have different resistive qualities across its surface in a pattern in order to increase the information while still only having one line of connection. Boolean logic in neural networks comes to mind.

    I’m sure they have thought of this stuff, it just seems like they spin it a really weird way.

Comments are closed.