Using gestures to interact with surfaces that don't have screens

A joint Disney Research and CMU team have produced a demo showing gesture controls on a variety of everyday, non-computer objects. The system, called Touché, uses capacitive coupling to infer things about what your hands are doing. It can determine which utensil you're eating your food with, or how you're grasping a doorknob, or even whether you're touching one finger to another or clasping your hands together. It's a pretty exciting demo, and the user interface possibilities are certainly provocative. Here's some commentary from Wired UK's Mark Brown:

Some of the proof-of-concept applications in the lab include a smart doorknob that knows whether it has been grasped, touched, or pinched; a chair that dims the lights when you recline into it; a table that knows if you're resting one hand, two hands, or your elbows on it; and a tablet that can be pinched from back to front to open an on-screen menu.

The technology can also be shoved in wristbands, so you can make sign-language-style gestures to control the phone in your pocket—two fingers on your palm to change a song, say, or a clap to stop the music. It can also go in liquids, to detect when fingers and hands are submerged in water.

"In our laboratory experiments, Touché demonstrated recognition rates approaching 100 percent," claims Ivan Poupyrev, senior research scientist at Disney Research in Pittsburgh. "That suggests it could immediately be used to create new and exciting ways for people to interact with objects and the world at large."

Disney researchers put gesture recognition in door knobs, chairs, fish tanks

• [koko szanel]

  Disney Research? really?? Expect perpetual patents soon.

  • [doug rogers]

    And they will certainly not be licensing the teledildonics applications.

• [Guest]

  Hey Microsoft, Hey Apple, the Mouse is now embarrasing you.

  • [tré]

    I don’t know about that. Microsoft doesn’t make hardware, so hardware really doesn’t embarrass them. Apple has been using multi-touch gestures without screens for years now in their trackpads. You could say that needing hardware like a trackpad as opposed to just using the body or something else hooked up to this is embarrassing, but that doesn’t quite mesh with the whole “total product package” philosophy. It does fit in nicely with the “as intuitive as possible” philosophy, though, so who knows?

    Edit: I stand corrected. That’s what I get for trying to stand up for Microsoft, I guess.

    • [Nathan Hornby]

      Microsoft does make hardware, in fact they make mice, as referenced in the comment you’re replying to :)

    • [Jer_00]

       ”Microsoft doesn’t make hardware”

      Tell that to the XBox guys.  Especially the guys who have spent some serious energy working on the Kinect.

    • [MarcVader]

       http://en.wikipedia.org/wiki/Microsoft_Surface …

      • [jerwin]

        Not one of wikipedia’s best efforts. It reads like a marketing brochure. The bold text bothers me.

• [doug rogers]

  aaaaand…. doors with personalities….

  • [Guest]

    “Absolutely ghastly. Just don’t even talk about it.”

  • [Dan Hibiki]

    “You watch this door, it’s about to open again. I can tell by the intolerable air of smugness it suddenly generates.” – Marvin

• [vattenpipa]

  Hahaha. Wtf with the “teach your kids to use the correct utensils” suggestion?
  It felt like a bit from “Look Around You”.

  • [Henry Pootel]

    All jokes aside, the concept could have a lot of applications for rehab after a stroke or brain injury, not to mention cognitive disability.

  • [Dan Hibiki]

     what it’s going to lead to is a generation of kind constantly poking inanimate objects in the hopes that it would tell them what to do.

• [cfuse]

  I don’t want a bowl of negative reinforcement for breakfast!

• [J Gostick]

  So now you get to look like a wizard when changing songs on your phone or making a call…

  • [jerwin]

    Daffy Duck will be thrilled.

• [Jethro Taylor]

  ASL to speech synthesizer, anyone?

• [Itsumishi]

  What comes up on the window when you slam the door violently? 

  *NOT NOW, THE BOSS IS IN A BAD MOOD*

• [Guest]

  If this is the future, wilderness is about to get a lot more important. 

• [Cowicide]

  I’m so glad the rest of the world is coming around to my world.  Welcome to my world.

  2003:
  http://forums.macnn.com/82/applications/176033/this-will-add-great-gui-functionality/

  Cowicide
  Yes, I’m smug (and bitter).. because I got a lot of shit from naysayers for well over a decade who are now gesturing their asses off (like me) now.

  • [Guest]

     I’ve been gesturing at the world for years.

    • [Cowicide]

      You were giving the world the middle finger before it was cool.

      • [Guest]

        point and shoot

  • [SmokingHeartDesigns]

    Yeah people don’t believe me when I talk about what’s coming either.

    • [Cowicide]

      At least in this case, I get the last laugh. The gestures I’ve been using all these years have saved me from repetitive strain injuries that many others have by now. Not to mention I’ve been getting much more work done in less time. Those wrist-crippled stubborn sumbitches should have listened to me.

• [keighvin]

  “Negative reinforcement” (the removal of conditions or stimulus as a consequence) does not mean “punishment” (the application of undesired conditions or stimulus as a consequence). They’re often related, but the term “negative reinforcement” has very specific meaning in psychology.

• [cramerica]

  By your command

• [bunnyvision]

  This does seem like a logical extension of the touchscreen mentality. Creeps me out, though.

• [Ken Williams]

  Finally.  I’ve been waiting for an advancement in proper-food-implement-training technology.  This could be the real shit.

  • [Ken Williams]

    Not to sound *too* cheeky though.  I mean, it seems like progress in capacitive sensing is pretty obviously sellable, with lots of obvious marketable applications.

• [David Smith]

  Child dips spoon in cereal bowl:  reasuring DING!   Child flings spoonful of cereal across the room.  Child dips spoon in cereal bown to refill:   another reassuring DING!  This will end well. 

• [sam1148]

  Unfortunately, I bet the first use Disney will put this to will be improved biometrics for park passes, or stern automated warning for kids that drag their hands in the water on a ride, or put their elbows on the table in the Prime Time Cafe.
  (which would be kinda cool if they had a Rosie the Robot roaming around the place).

• [Michael Crumpton]

  This is the most interesting advance in touch sensing in a long time.

• [bigomega73]

  Instead of nonense gestures for controlling your media player/cell phone, why not have the device just recognize ASL gestures. That way everyone gets to learn ASL while they learn how to control their devices. Seems like a wasted opportunity if they just make up their own gestures to me.

• [Neill “Dire” Mitchell]

  Electric Potential sensing has some interesting things to offer the Gesturing world.
  http://www.plesseysemiconductors.com/products/epic/videos/

  Disclosure – I work for the company linked above.

• [abraxiom]

  It would seem to me that based on the examples they give, that it is actually “just” measure the amount of skin contact with the contacted object. Perhaps different areas of skin have different, um, electrical properties but it is not so much gesture recognition in itself. I think the next step is to do a lot of interaction design type stuff to see what configurations of multiple zones could produce more meaningful information. Also the variety of gestures and body types, yadda yadda

  Also, couldn’t you make the “doorknob” itself have different resistive qualities across its surface in a pattern in order to increase the information while still only having one line of connection. Boolean logic in neural networks comes to mind.

  I’m sure they have thought of this stuff, it just seems like they spin it a really weird way.