"Sixth Sense Technology" Will Be Open Source

My husband called me last night all a-twitter and once I got him talking slow enough to understand that he wasn't going on about "six pennies", I could sympathize with the high level of enthusiasm. Earlier this year, "Sixth Sense Technology" from MIT---basically, a visual interface system that allows you and the computer in your cell phone to communicate in some truly astounding ways---was a big hit at TED. This week, at TED India, inventor Pranav Mistry announced that the technology will be released as open source...in a matter of months.

"Rather than waiting for that time to come, I want people to make their own system. Why not?," Mistry says in an article on Rediff Business. "People will be able to make their own hardware. I will give them instructions how to make it. And also provide them key software...give them basic key software layers...they will be able to build their own applications. They will be able to modify base level and do anything".

Makers, start your engines.

Mistry to make digital "Sixth Sense" open source on Rediff Business
The importance of Sixth Sense going open source on zdnet

Sixth Sense augmented reality device goes open source on Singularity Hub (natch)


  1. this is neat — but does anyone really buy the notion of real-world gestural display tech? MIT’s media lab hasn’t turned-out too many real-world applications from their more neato-cool projects — “harder” concepts like MPEG formatting and even the $100 laptop make sense — and work. most of the experimental musical instruments, the mattel-MIT partnership — even LEGO and MIT haven’t gone as far as the demo videos (for those endeavors) would imply.

    there’s nothing like point-and-click in terms of instant — you can see what you’re shooting and look at the pix on the device immediately — this system pre-supposes that folks don’t want to use their devices (and instead wish to use the hand).

    the whole point is that the hand is being extended — think 2001 where the ape-man throws the bone up into the air…and the shot dissolves to a space ship. humans make tools. we like tools. we use tools. our tools rock. our hands are awesome — for using tools. the hands aren’t the tools outside the bedroom, massages and pinching salt. the rest of the time we’re pecking, grasping, turning, clicking and caressing our tools with our hands.

    ALSO — there is NO new tech in this concept — it’s all about existing hardware with some firmware updates being linked with existing devices all run on some custom code.

    making…that — whatever it is — (a bundle of hardware drivers?) open source is a good idea — folks WILL enjoy playing with this and seeing what they can hook-up. no doubt it’ll get used for pr0n and teledildonics.

    meanwhile, keep blowing our minds TED and keep promising us tomorrow MIT and keep posting fun stuff BB.

  2. This is great news! If companies aren’t willing to jump on a new technology, you can bet industrious nerds will. Open sourcing the technology means that somebody, somewhere will do the hard work to get the ball rolling. With the right license, it even gives companies an incentive to adopt the technology, since there won’t be any troublesome license fees to contend with. Assuming it’s not a license that makes their life harder, like GPLv3.

  3. eh well it’s kind of either this or he winds up sharing the rights to it with MIT… but still good for it to be out there.

  4. Hmm, didn’t Bruce Sterling once say that any design that begins, “the device hangs around the users neck” is likely to be bad design?

    @RevEng, the GPL (both v2 and v3) just means that companies can’t appropriate the work of others without compensation in kind. While that no doubt “makes things harder” for the companies, in the same way that having to pay for groceries at the supermarket “makes things harder” for me, on the balance, it makes things better for the technology itself and for the whole of society.

  5. It’s very strange that anyone sees value in this. A smartphone is honestly a superior technology in almost every way. Who would want a mobile device that you can’t use while walking (or moving your torso at all), requires you to depend on your environment for use of its display, and makes that display publicly viewable to anyone around you?

    I understand that cute nonsense sells people devices, but this is kind of a stretch.

  6. Okay, I have to admit to being another hater, but I have another reason:

    All of this has to be set up and monitored by the user. For example, for it to give me toilet paper recommendations, I would have had to tell it in software what I’m interested in when it comes to toilet paper. That also presupposes that I even have an opinion on it. And if I had an opinion on it, I’d probably already know what brand I wanted.

    Furthermore, where the hell does all this data come from? Is there a database of toilet paper manufacturing processes? How do I access it? Do I need to import that data into my own database?

    Buying the book: Okay, another terrible example, because no one buys books in stores unless they’ve already read about it on Amazon. Again, the actual research for purchasing decisions is always going to result in a product name, so it’s not confusing. If I’m worried about MSG intake (as I am–I just try to limit it; I’m not one of these food Nazis), it’s a hell of a lot easier for me to compile a mental list of the various names it sneaks by with in ingredients, and I can then read that list on the can. That’s why we have laws about labeling.

    The photo thing is pretty cool, but no one wants to put things on their fingers to do it, and, moreover, you can really look forward to some crappy shots that way, having no idea what the damned thing is taking a picture of.

    Beaming a numberpad on my hand is ridiculous. I love my iPhone, but even so, I miss having a real keypad. I certainly wouldn’t want to go one more step away to where I actually have to find something to shine a projector on and then touch buttons that appear on the back of the finger I’m using to push them (as happens with a projected image).

    Basically, the uses demonstrated for this are unnecessary, the implementation is unattractive, and the device itself isn’t really advancing the vision very far, since most of the work would be on the software end, getting all that data in one easy-to-read-and-access place–a very difficult, time-consuming, and expensive proposition.


  7. Here’s a little spoiler:
    projected interface technology has been dead all along!

    I’m not seeing anything too new about this interface only a nerd could love. ‘We’ll have a nicer design of this necklace-phone, then just paint your fingernails rainbow colors…’

  8. I think this is awesome.

    This would not replace your phone, it would augment it. For social acts like looking at a map or photos with a friend, this rocks.

    Use this with a pair of VR glasses or VR contact lenses (currently in development) and you can project the data onto your visual field in private.

    Currently with off-the-shelf tech this hangs around you neck. In a very short period of time the components could be in the form of jewelry (necklace, collar, earrings, rings, or all of the above) or embedded in your clothing (hat, shirt buttons, glasses).

    No need to create your own info database. Say you are shopping for toilet paper. Me, I’d subscribe to Greenpeace’s buyers guide. Maybe you’d subscribe to Better Homes and Gardens buyers guide, or the Feminist buyers guide, or Consumer Reports, or whatever.

    Same goes for people data: subscribing to various Reputation Networks gets you data about that person. I could also see most people subscribing to an instance criminal background check. What will this do to crime rates when your crimes *literally follow you around.*

    What will this do to rape and robbery rates when everyone is a 24/7 live feed to the internet?

  9. I like this, but I’d prefer a glasses-based interface that would let me see these things, but nobody else. Think about all of the information you don’t want projected in a public space.

  10. What i love about the haters is their always lack of imagination. In some of the shots he didn’t have anything around his neck.

    Open source allows those with imagination to run with it.

    How many TED talks have you given?


    Why do you read bOING anyway?

    I’m going back to my record player.


    1. I agree with you 100%! How can ANYONE not see the value of the Sixth Sense Technology!?
      Its freaking awesome! And to make it even more awesome, its freaking open source. This guy deserves to get a Nobel Prize!

  11. this does seem pretty cool at first glance, but not very practical…what happens when this is mass produced and half of the people in the store has one of them? projections everywhere overwriting each other! meeting someone? stand very still and keep it pointed away from his/her face so the projection does not blind them…why would someone draw a watch on their hand to check the time when they could just as easily look at their phone, computer, clock or, this may just be crazy talk, a real watch!

    seems to me the better, more practical thing to do would be to write “product recognition” software for a mobile phone with camera+internet capability, then display the info on your phone…or what about an app that could read the product barcode?

    you’ll get better picture quality on the mobile too…flat picture instead of undulating ridges of a person’s hand or shirt

    kudos to pranav for putting it all together, but sixth sense? how about nonsense

    1. i love the way you think. The people here are so used to the way technology is… now, that they’re either afraid or too stubborn to change and adapt to a more futuristic technological advance. Sure, projecting it for all to see… not the best idea, but at the same time, if you don’t want people to see it, don’t project it where everyone can see. He shows you that you can use a piece of paper or some other surface that you can hold close to you, and that’s no different than sitting on the subway with a laptop.

  12. some people have very short sighted imaginations… just read some William Gibson… especially “Virtual Light”:

    “Book overview
    2005: Welcome to NoCal and SoCal, the uneasy sister-states of what used to be California. Here the millenium has come and gone, leaving in its wake only stunned survivors. In Los Angeles, Berry Rydell is a former armed-response rentacop now working for a bounty hunter. Chevette Washington is a bicycle messenger turned pickpocket who impulsively snatches a pair of innocent-looking sunglasses. But these are no ordinary shades. What you can see through these high-tech specs can make you rich–or get you killed. Now Berry and Chevette are on the run, zeroing in on the digitalized heart of DatAmerica, where pure information is the greatest high. And a mind can be a terrible thing to crash…”


    and I’m sure some of Cory’s work features tech like this

  13. The device here is a metaverse interface. It is absolutely fantastic and i am so glad to see it in action. I spent some time working on the theoretics of this metaverse with some top flight folks such as Esther Dyson, Scoble, and a number of virtual world designers like Raph Coster.

    The exciting thing is the evolution that the technology can only get smaller and more intelligent and accurate as the years go forward. From the Cure-cat to this is a light year leap forward. Fantastic job from a fantastic school.

  14. That’s it…I’m finally giving up my motorcycle. Can you imagine how dangerous the streets will be when all the d-bags out there start using this tech while driving?

  15. This is really cool technology, but I’m not so sure it’s creators have a good target-use in mind yet. There probably is one, but it wasn’t in that video.

    manicbassman: I just read Virtual Light (the whole trilogy really) yesterday, and there is no projection-interface technology in it at all.

    Cory once wrote about an output-only light pen device, and pretty explicitly stated that it was going to have to work hard to find a niche.

  16. This is utterly brilliant; it’s been predicted in Dennou Coil, but to see a working prototype now is fabbo!

  17. People are missing the point.

    Their toy/prototype version has a projected display and it hangs around your neck, which is obviously not ideal or even really useful. It is a necessary step.

    Get to the point where this thing is integrated with your glasses or something in that line, then you have something truly groundbreaking.

  18. I do not lack imagination. I can imagine myself trying to use such a device and then giving up very quickly, as it will likely slow things down, not speed them up.

    I said the same thing about the MS Surface project, which is now quite old. I could see how it could be pretty cool for things like kiosks, etc., but no one wants to use a computer at home like that.

    And anytime someone brings up Minority Report as an example of the future of UI design, I chortle. No one wants to hold their hands in the air all day. The march of human-computer interface has been one toward less movement and more support as we have increased the amount of time we spend using them. Monitors should be at eye level; keyboards should be usable with straight wrists, etc.

    And what I meant when I said it doesn’t expand the vision, I meant the vision of the technology. For example, Steve Mann at the U of Toronto, who claims to be a cyborg because he has computer glasses, has had this kind of technology attached to him most of the time since 1980. It’s been around a long time, and even in the 1990s, we were reading about stuff like this that predicted we’d all be using it soon enough. We’re not.

    I see stuff like this a lot like flying cars, which were predicted for decades. It isn’t really a technological reason that we’re not all zipping around the skies instead of being stuck in traffic; it’s a logistical one. How do you get that many people up in the air safely? How do you make it so that a simple maintenance problem doesn’t send Joe and Jane Sixpack plummeting to their deaths with their 2.3 kids in the back seat? The changes in infrastructure necessary just don’t really balance the benefits of such a system.

    Obviously, this device doesn’t include such large obstacles, but I think it suffers from the same cost/benefit balance problem. Setting it up, maintaining it, having networks that let you stream all the time, having wireless providers that let you actually use Bluetooth for something other than headsets, each person carrying some piece of kit that constantly displays probably-needless information–whether it be projected or (as in the case of Mann) on glasses, the inevitable failure points (as it is, sometimes I can’t get Wikipedia to load when I actually do want to know something out and about) and delays, all to do something that isn’t really necessary, just doesn’t strike me as a very viable or likely technology.

    People don’t need this. Even as much as I like my iPhone, honestly, I used it a lot for research and stuff when I first got it, but already I pretty much don’t. It’s way easier to just make a frickin’ list on paper or on the phone of things I want to buy or rent or do… I don’t think anyone needs constant information like this.

    None of this is to say that it isn’t a really cool project and tech demo. I just don’t think it’ll make it much past that.

  19. strapping gadgets around your neck is so not elegant, im sorry.

    i think that instead of a projector it would be much more awesome if images were projected right into your optical nerve. Now that would really be portable.

  20. I do think human should try to move in to the new world like Pranav introduced.
    The world is changing. Technology is changing. And why human is not changing?
    From the demo of Pranav, he is showing some examples which could help us in our life. I believe this application of this sixth sense device may not only hang on neck, but can be on your glass / wrist?
    I can foresee the future will change a lot especially the way people communicate to each other if everyone is start using this device.
    Get ready and welcome the future.

  21. I think this is a cool idea. Yes, the technology is not new, and neither are the concepts but it is one more version. And it works. Many aspects of this should be improved (the projector on a necklace idea) and I am sure they will be. The next generation version will not look much like this one but that is the beauty of this. Flying cars don’t exist for the same reason that home made cars don’t exist (for the most part). It is incredibly expensive to make a car and even more so to design a new one. Buying one is cheaper and the seller has no incentive to design a new kind.

    Proprietary designs cost a LOT. Open source the designs and you will cut down the cost, allow access to everyone and Fred in the shed will make this better. As you said, the technology is not new and the hardware is inexpensive, so this can become real (after some iterations). I am looking forward to this.

  22. Are there any reports or SDKs out now since the mention of becoming open source? I’ve been looking at this for a few months now and was curious on his progress, especially with creating the sdks and/or guides he used in creating this version of the device.

  23. The whole point of the 6th sense technology revolves around two things – 1) Making something that is extraordinarily awesome, 2) Bridging the gap between the physical and virtual world.

    First point is fairly self explanatory.

    Coming to the second point – Ask yourself, how many hours a day do you spend your time on the PC/laptop/smart-phone ? Unless you’re really fortunate or unfortunate, we(most of us) do spend a fair bit of our “free” time usually at home or elsewhere where we have access to a computer etc.

    Why do we do it so regularly? – Because its Fun and We learn new things and it is EASY. I can go on and on about how much of a better view and knowledge of life we have because of YouTube, Facebook, Wikipedia, Google ETC.

    But imagine where ‘surfing’ through the net is actually like walking on the road and every ‘link’ is like an object on that road. For once we can have a choice whether we’d like to stay Indoors or Outdoors but with the same virtual facilities that we enjoy now while in front a computer but with the feel of the physical environment.

    I’m highly skeptical whether it’ll work, but I’m hoping against all hopes that it will improve to be sold to us.
    I have an iPhone and its the best, but the first touch screen phone I had sucked some a$$.

  24. Does anyone have a link or source as to where I can get a kit or instructions for making one of these devices? Perhaps someone makes and sells them ready to go? Thank you.

  25. I have checked out his website and apparently he was hoping to give instructions on how to make this over a year ago. Saying that it would only take about 2-3 months. Yet nothing has been updated on this at all yet so it seems like they either have hit a brick wall or they are not going to try and give this away for free.

  26. For those who they’d rather use their phone, consider this. You are constantly carrying your phone around with you everywhere you go. Phones come in various sizes with different perks based on it’s size. Really small ones are easy to carry but a pain to stare at and operate. Large ones are more user-friendly but they are prone to dropping or falling out of pockets.

    Instead of this crap where your essential contacting tool is this technological pocket sized brick, you have a small projector/camera. Immediately, screen size is optional. The tool you are operating doesn’t require that you touch it! This simplifies so much. Instead of your eyes being forced towards a tiny screen, you decide where your eyes go.

    This technology is like creating the whole cellular interface device around a person’s body. Instaad of a cumbersome machine that you are trying to command, you yourself are moving and searching through information.

    This technology skips past terribly overated touchscreen technology and turns anything in the world into a button.


    KARAN J.

  28. We do indeed love using our devices (tools), but how many people do you see now walking with their head downs because they have their eyes fixed on their smart phone?

    As a younger adult, when i go out with my colleagues/friends, i find that if the conversation goes dry, or someone is uninteresting in what’s going on in their surroundings, they disengage inside their smartphone/tablet world where they become isolated and secluded, and this is in a social setting!
    No my friend, these devices are single handily ruining social society.

    Even if we do feel more comfortable with these tools, I’m certain that this new sixthsense technology or something similar will be the future we’ll all end up adopting.

    Tablets/Smartphones have already started using augmented reality, and the possibilities for that seem limitless. (If you don’t know what augmented reality is, youtube or google it now.) But the idea of carrying a device, no matter how small or light they make them, eventually will wear away when it becomes easier to do the things they do without them. We don’t NEED something in our hands, we don’t NEED don’t more tools, we were already born with them.

Comments are closed.