Oz Ramos is the creator of Handsfree.js, a software library that you can use to add face-, hand- and pose-tracking to any app built in Javascript. It looks incredibly fun and I'm gonna tinker with it this weekend to use in some generative art thingies.
Ramos also working on MIDIBlocks — a Scratch-like interface that neophytes to coding can use to create their own commands that will be executed as you move your head around (which controls the cursor) and blink, raise an eyebrow, smile or do other common face gestures. Significantly, MIDIBlocks can control the mouse pointer on your desktop, so once it's turned on you can use it inside regular desktop and OS software.
In that video above he talks about it, and you can see the MIDIBlocks demo online here. You can go to the full set of MIDIBlocks commands for controlling your desktop or writing Javascript. And since you can turn on his facetracking while you're doing so, you can … use facetracking to program new facetracking commands. Very meta!
In fact, I successfully used the facetracking to navigate around his site — the "click" gesture is a quick smile — and use his calibration tool to correctly reposition my face in the camera-view.
The face-tracking control is a little tricky to learn, and it never got really comfortable for me — but then again I only tried it for a short period. At the end of the video, Ramos shows himself using it to play solitaire on his computer, and he's really whizzing around, so I suppose practice makes perfect.
