MIT Researcher records 90,000 hours of home video, analyzes the hell out of it

[Video Link]MIT researcher Deb Roy's presentation was probably my favorite at TED2011. The highlight of his presentation was when he played an audio file of his son learning how to say "water" over the course of the research project.

MIT researcher Deb Roy wanted to understand how his infant son learned language -- so he wired up his house with videocameras to catch every moment (with exceptions) of his son's life, then parsed 90,000 hours of home video to watch "gaaaa" slowly turn into "water." Astonishing, data-rich research with deep implications for how we learn. Deb Roy studies how children learn language, and designs machines that learn to communicate in human-like ways. On sabbatical from MIT Media Lab, he's working with the AI company Bluefin Labs.

Deb Roy: The birth of a word



    Or actually, a guy called “Mr. Brainwash” already did it. Or something like it. See generally, Exit Through the Giftshop (Banksy 2010).

  2. He’s the founder and CEO of Bluefin labs, and although his experiment is cool, his talk ends up being an ad for their services.

  3. sorry but surely just the fact that he knew they were videoing all these 5 years did have an impact on how they taught the kid language. surely this will have distorted the research massively. of course it would be deeply problematic to film a similar situation without the participants being aware of the research being carried out but yeah i think he missed that in his little presentation.

  4. It’s an interesting video. I suggest that those that watch it then go watch Final Cut as the anti-unicorn-chaser.

  5. That’s incredible. The fact that he was able to collect such an absurd amount of audio/video data, and then actually instrument it and collect real, rich, interesting data from it, is amazing.

    That must give hope to all those doing Quantified Self-type recordings, where they are taking huge amounts of data about themselves. I always thought that most people would either be not taking enough data to make it interesting, or be taking too much to be able to analyze. But it looks from this as if you could never have too much data.

    I guess those of us who have gigs upon gigs of photos ever since getting digital cameras may finally be vindicated. Who knows what kind of data we might be able to pull from them, and what ways of visualizing them we might have in the future.

    1. I was part of a now defunct company, Zetera, that did “blocks over UDP” —you could connect arbitrary numbers of disks with IP adapters. Then you could construct arbitrary RAID topologies out of these networked disks.

      The desktop CPU was the disk controller, it had cycles to spare.

      We supplied Dr. Roy at least in part, at some point, with drives
      for free. PR no doubt.

      I stood in our room with maybe 60 TB under my control, this was
      a few years ago so it was maybe 60 boxes with a 500GB drive each.
      He had more I think.

      The blue LEDs used to blind me but made for good scifi pix.

      I was an ugrad at mit so I got what he was doing. He had privacy shields over all the cams that also shut off the mics.

      Its amazing that much time has gone by and his kid is now 5.
      Best of luck and I hope his research does well. I have one and found his development fascinating. (And you don’t need to
      pass an ethics committee to do experiments)

  6. 90,000 frigging hours!
    ~ i will never complain about working on a documentary project with a measly 1000 hours of recordings ever again

    (i wonder if he’ll market his software?)

    1. “i wonder if he’ll market his (analytic) software?”

      i just hope this is used for good and not for evil
      Big Brother would love this technology

      btw, there’s a unicorn chaser at the end to the video

      1. @millrick in reply to millrick

        All us decent technologists hope our stuff, some of which we
        realize may be dual use, is used for good.

        The guys who make battery chargers don’t want them clipped
        to dissident nipples, ok?

  7. I think the idea that the house was so wired that it could essentially “remember” everything for you was as interesting if not more so than the language research. Amazing stuff all around.

  8. What Deb has done is amazing. It was still a ton of work for them to transcribe and mine that *much* data. For us mere mortals I have found that continuous audio recording in combination with a button to click to ‘remember that’ is feasible.

    I have been liferecording for the last 9 months. So far audio only as it is the most unobtrusive for other people. But like Deb, I have found it amazing at catching my kids at their best moments, with things I would normally miss.

    Anyway, I have open sourced what I have done and it’s almost ready for mass consumption.

  9. This is great stuff. I grew up with Deb and it is wonderful to see that our ridiculous shenanigans during community gatherings has not damaged his intellect or self possession. As opposed to me….

  10. The “water” sound montage needs to be a hit record, soon.

    Seriously. Boy gaaaaa instead of lady g.

  11. I bought a little cassette recorder so I could save the sounds of my daughter while she was still tiny. I don’t think I got it in time, though. It used to be we’d go through the bedtime routine, and she’d lie quietly in bed and we’d go out of the room.

    Then the magic would happen. We’d hear her tiny sounds. She’d be talking herself to sleep in baby talk; probably a collection of sounds from China, because she’d been on the point of starting to talk just as we brought her home.

    I wish I could hear those little syllables again.

  12. Fascinating, inspiring, the implications can be terrible. This will no doubt be turned on us, we must turn it on our leadership classes as well least the lopsided applications of this technology establish the most horrible form of government mankind has yet to experience.

    1. In the US the two major political persuasions are looking at this closely. One will marvel in its wonder, the other will stop at nothing to harness its power.

  13. The presentation and underlying research and data-gathering logistics and analysis is incredible.

    That being said, the conclusions are weak to non-existent.
    Despite a dramatic and awe-inspiring presentation full of Matrix-movie like eye candy, all he comes up with is that: people talk about what they see on television on twitter.

    People talk about what they see on television.

    Am I the only one to see that this research fails to do anything but state an obvious observation?

    1. People talk about what they see on television. Am I the only one to see that this research fails to do anything but state an obvious observation?

      Um, no, I think you’re the one who failed to get the point. Yes, people talk about what they see on television. We know this. The researchers know this. That’s why they decided to delve into all that information and start to try and make sense of it. Start to try and structure it. Start to find out what the conversation is, and who is talking to who, and how different events affect the conversation.

      This will certainly be interesting to researchers, for whatever egg-headed reasons they have, but is even more important to people like politicians and the TV industry. As someone said above, this puts the Nielsen ratings to shame.

  14. One key point yet unmentioned is often that “pro-am” bloggers & c. are also on the payroll of the people producing content – having a plane for content and one for consumers in this way isn’t an objective picture of what’s the true ‘pulse’ of a nation. Perhaps this is doubly true with political content.

    1. I imagine they chose “water” just because it’s easier to track down. it’s probably one of the first words a kid wants to produce. It would be much harder to make the study on a word like “strawberry” or “anger”, just because a kid doesn’t have to say “strawberry” as often as “water” and becasue “anger” is a more abstract concept and it’s acquired some years later.

  15. The end was really scary. This takes Nielsen ratings beyond anything ever dreamed of. Seeing the impact of media and it’s feedback loops in a quantitative way with multidimensional visualization provides media creators the ability to tune messages to match expectation or optimize impact. Politics will never be the same. Media messages will dip to match the market just as speech patterns dipped to provide feedback to the infant’s word learning process. The echo chamber between mass media and social networks is becoming self aware. Those with access to powerful feedback visualization tools can manipulate messages to sell us anything. The business world and political world will run wild with it, empowering them to do whatever they wish, while telling us what we need to hear in order to get away with anything.

  16. That was pretty freaking amazing stuff.

    What’s with all the pessimism though, seriously, some of you here think that the big, bad, govt. is going to turn this sort of tech on us? One question, how? Are you just going to let the goons into your house to wire all this junk up, make them coffee and sandwiches while they work? And what about the collection, storage, and mining apparatus? Shit people, I don’t care how cheap a GB is these days it’s just not possible, yet. I love Orwell as much as the next guy (alright, a lot more) but I think the importance of what he was writing about was more the political and totalitarian not technological and analytical capabilities of evil tyrannies. Speaking to a matter of scope, it is unreasonable to even think that the govt. could monitor every persons phone conversations, as many tin-foil-hat-fraidy-cats often think is possible. Adapting that idea to all encompassing audiovisual surveillance is just absurd.

    Then again MIT does get a lot of funding from the military and spook community.

    Also, media manipulation is nothing new. Herman and Chomsky wrote about that in ‘Manufacturing Consent’ way back in ’88. And of course it has only matured. Still though, it’s not perfect and has little effect on rational people like many of us here at BB, or rather shouldn’t. I also realize that for the uneducated and irrational herd this creates problems.

  17. I think patterns exist all over around us, in a bean seed, in a mineral, in an atom, a molecule, in a galaxy and in animal behaviour too.
    We can use new technology to collect data (new or old) and analyze it and discover or have a better understanding of patterns. So, I think this talk was impressive because of all the cool graphics and images. It was sort of like watching fireworks. Also, the presenter spiced it with some emotional moments, what for? The aim was to give a cool talk, it’s just a show! have we learned anything new about how kids acquire language?

Comments are closed.