AI vs. IQ: IBM's Watson takes on the meatbags on Jeopardy

This week on "Jeopardy!," various champions of the popular television game show face off against the new IBM computer, Watson, which was developed to compete in human brain games. It's not a supercomputer, per se--some TV show hosts and bloggers have misidentified it as such--but it is a very interesting application of AI.

Over at PBS NewsHour, science correspondent Miles O'Brien did a fun piece where he challenged the machine to a "Jeopardy!" duel. The segment aired last night, and is embedded above (video link / transcript). As an aside, the piece includes often-Boinged personalities Marvin Minsky and Ray Kurzweil, and as a bonus, includes some wonderful old sci-fi film clips (can you ID them all?)

Below, watch Miles, Watson and David Gondek (one of Watson's creators at IBM) do battle over "unusual animal phobias, presidential tongue twisters and laundry detergent brand names." (video link). I don't want to spoil too much for you, but: Miles is a smart guy, and Watson pummeled him. It ends in a sad, sad zero on one of their scorecards.

If you've read this far, you'll also want to watch this NOVA feature on "The Smartest Machine on Earth." And you may want to revisit this excellent Boing Boing post last week from guestblogger Bob Harris on the science behind Watson's silicon brains.

Night two of the man-vs-machine challenge airs tonight on Jeopardy.

Boing Boing editor/partner and tech culture journalist Xeni Jardin hosts and produces Boing Boing's in-flight TV channel on Virgin America airlines (#10 on the dial), and writes about living with breast cancer. Diagnosed in 2011. @xeni on Twitter. email: xeni@boingboing.net.

MORE: Gadgets • Games • pop culture • Technology

More at Boing Boing

Eurovision 2013: An American in London

The technology that links taxonomy and Star Trek

SamSam

You can find last night’s entire show on YouTube: http://www.youtube.com/watch?v=4PSPvHcLnN0 and http://www.youtube.com/watch?v=CtHlxzOXgYs

I very scrupulously avoided finding out how Watson did until I could watch it at work this morning. (Really, who schedules an important event for the evening of Feb 14th??) Last night was only through round 1 of the first game.

In terms of AI developments, I would say that this is a significantly more important moment in history than Kasparov vs. Deep Blue. For some reason, though, I would bet that most people would regard Jeopardy as more trivial than chess.
- oasisob1
  
  “(Really, who schedules an important event for the evening of Feb 14th??)”
  
  A bunch of dateless nerds, I suppose. Go nerds!
Anonymous

It would be interesting to see what Watson prints out as it’s “average font” that is in it’s mind as the way to recognize all other fonts it hasn’t seen yet. That’s Plato’s idea of perfect forms, and what type designers in the 50′s tried to accomplish with Helvetica.

I’m sure the scientists can print out an image of that, but I doubt any of them would have thought to do it.
Anonymous

I can’t tell you how disappointed I am in Miles. He ruined Watson’s opportunity with the last tennis question to ask “What is love?”
UncaScrooge

Alex, I’ll take “Paradoxes” for $800.00
SamSam

I think they would say that buzzing in first is actually a pretty significant part of the game.

Well, indeed they would, and actually Watson is pretty great at that.

Wait, which side were you debating on? Buzzing in is a task that they made Watson do, instead of letting him do it by some other means. Is the problem now that he’s too good at that?

I was saying that the task of reading words off a blue screen is superfluous to the game, and that no one would consider that an essential task in playing “Jeopardy.” Likewise, Deep Blue didn’t need to use a video camera to view the board, and instead had an electronic representation. No one complained then, because they recognized that the video camera would have nothing to do with chess.

But apparently buzzing is important, so they decided to make Watson do it, and, guess what, he’s great at it. But now making him compete on a level playing field (making him buzz) is also somehow “unfair” in the minds of many, because apparently he’s too good.

Frankly, though, I wish they had included a video camera and forced him to do OCR (which is so trivial even my phone can do it), just to prevent people from fixating on that.
ncinerate

What bugs me really isn’t the “reading” of the questions, but the ability for the computer to rapidly “buzz-in”.

None of those questions were particularly hard – I knew many of them before the actually buzzing-in is allowed (as I’m certain both human contestants did). The problem is – the game lights up a little indicator behind the podium saying when you can answer, at which point you can press the button to answer. Watson gets this impulse electronically and can chime in instantly without the laggy “eye-brain-nervous impulse-hand” loop to contend with. Without delay in his system, it is nearly impossible to contend with him on this fact alone.

The simple fact is any good jeapordy player knows almost -all- the answers before that little light fires, so we’re contending with human reflexes vs a computer’s ability to instantly respond to an electronic stimulus. While it’s a remarkable show of the computer’s ability to answer the questions, in practice it’s just showing that a computer can flick a switch faster than a human.
- ncinerate
  
  In fact, just watching the video of watson from last night on jeapordy again – look at ken jennings. He’s trying to click on almost every single question but is unable to get his click in before watson.
  
  As the show continues watch him closely, he’s trying to click. Every single question – you can see him clicking away with no effect. Both human players are easily able to see the question, parse it, and answer it correctly before the actual “click-in buzzer” is turned on.
  
  Again – not discounting the amazing ability of the computer to parse the information and come up with an effective answer – but in the format of jeapordy the computer is given a HUGE advantage by the way the game works. This becomes more of a “john henry vs the track-laying machine” than a “kasparov vs deep blue”.
  - chgoliz
    
    I agree with this completely. I’ve known several friends who were on the show at various times, and they all said that getting the timing right on the buzzer is the only significant difference between the (human) contestants.
    
    In this case, the playing field is particularly lopsided. Electronic signaling is faster than mechanical signaling. The set-up is categorically unfair to the humans.
  - Guysmiley
    
    Errr, how is “Kasporov vs Deep Blue” any less “John Henry” than this? It’s brute force processing of a huge amount of prior data, and the fact that it works as well as it does is really impressive.
    - RedShirt77
      
      One is mechanical and one is intellectual.
    - ncinerate
      
      “Errr, how is “Kasporov vs Deep Blue” any less “John Henry” than this?”
      
      Actually, the answer to that is quite simple. Yes, in both instances a brute force computation method is employed, but in the Kasparov example both the human and the computer are facing off on equal grounds.
      
      Each player takes turns, allowed to complete their move within the time allotted. Yes, Kasparov lost to brute force computation, but it ws still at least a fair battle. Kasparovs ability to move a chess piece at superhuman speeds wasn’t a factor. He was able to think through his moves and play accordingly, he lost a legitimate match.
      
      In this jeapordy game the players face the same kind of brute force attack, but the nature of the game means the humans playing against him aren’t on a level playing field. Ken Jennings physically can’t click that button faster than Watson can register the electronic impulse and chime in. -That- is the reason he is losing. It’d be like competing with a gun in the “who can toss this lead across the room fastest” competition. Both players could clearly get the lead across the room, but the gun is going to do it faster.
      - Guysmiley
        
        You’re entirely wrong on that account, the computer still has to ring in with the same mechanical switch that the human contestants are using. The computer is searching through terabytes of data in the short amount of time it takes for Trebek to read the question. Nobody can ring in until the question is fully read, in fact if you try to ring in early it puts you in a time-out for a short period of time, so mashing the button early actually hurts you.
        
        This isn’t some side-show gimmick to sell IBM computers, it is actually an incredible demonstration of computing power.
      - ncinerate
        
        I wasn’t saying ken jennings was mashing the button prior to the light coming on – I was saying that he is clearly clicking it -after- the light comes on and still failing to chime in, because he’s SLOWER than the computer on the buzzer.
        
        Yes, the computer has to chime in with the same buzzer – and I’m certain that it is able to do so at a speed no human can match. From impulse to click is simply faster than a human being is capable of performing.
        
        The computer -and- the human contestants know the answers to most of these questions well before that little light comes on. The issue here is the mechanics of a human being buzzing in before the computer. Still not discounting watson’s amazing ability to parse the data and come up with an answer (it’s cool) but it’s -clearly- got a mechanical advantage that has nothing to do with knowledge.
- SamSam
  
  The simple fact is any good jeapordy player knows almost -all- the answers before that little light fires, so we’re contending with human reflexes vs a computer’s ability to instantly respond to an electronic stimulus.
  
  Wow, I really, really can’t believe so many people are using this line of argument, and are failing to understand what it is that Watson is demonstrating.
  
  What Watson is not demonstrating is the ability to click a buzzer quickly. That’s not important.
  
  What Watson is demonstrating is an ability — which not five years ago would have been laughed at — to come up with the correct answer within the few seconds allotted. Make no mistake, the fact that it can do this is truly remarkable. As I said, anyone with much knowledge of AI would have laughed at the idea that a computer could compete with humans at Jeopardy five years ago.
  
  So the challenge is, can Watson come up with the answer before Alex finishes speaking? If yes, it wins the point (because it’s fast on the buzzer). If no, it doesn’t, because the humans will.
  
  And again, note that the computer still isn’t perfect. Often it didn’t know the answer before Alex finished speaking. In those cases the humans won. But if it can work out the answer before Alex finished speaking, which is a ridiculously hard problem which hundreds of AI researchers have failed at before, then it gets to buzz in and win the points.
  
  I do see where everyone is coming from, though. The problem now is, given that we’ve seen the computer is so good, the humans can’t win on their own merits, they can only win when the computer doesn’t know the answer. Well, tough — that would merely mean that Watson has succeeded in this important demonstration.
  
  If you want to keep up the challenge, don’t give the computer stupid mechanical disadvantages, like making the buzzer harder for it to click. That isn’t the point. Instead, make the questions harder and harder, so that you keep pushing the dividing line between the kinds of questions a human can answer and the kinds of questions a computer can answer.
  - RedShirt77
    
    There seems to be this disconnect between two groups on this thread. Those that want to see a fair fight between man and machine, and those that want to revel in the computer science.
    
    If reflexis and fairness are irrelevant, why are there two people standing on either side of the silly avatar with buzzers?
    - Guysmiley
      
      Idiots abound, I see. When Deep Blue spanked Kasparov the echo was “yeah that was brute force, a computer will never play Jeopardy”, because natural language processing is so complex and difficult.
      
      Then they come up with a system that can play Jeopardy and WIN and people are tripping over their pouting bottom lips complaining about the “fairness” of the god damn buzzer? You people who are doing that are like the ones scoffing at the Wright brothers and their crazy bird contraptions.
  - chgoliz
    
    So the challenge is, can Watson come up with the answer before Alex finishes speaking? If yes, it wins the point (because it’s fast on the buzzer). If no, it doesn’t, because the humans will.
    
    Ah. Thank you. That makes it perfectly clear, to me at least.
SamSam

Those that want to see a fair fight between man and machine, and those that want to revel in the computer science.

Right, I got that towards the end of my big block o’ text. The AI-geek in me says that it’s great that people are complaining that the fight isn’t fair, because that means that the computer has already won.

My point at the end, though, is I think the right answer (though I don’t expect anyone ready that far). If you want to make the fight fairer, don’t mess with “silly” things like making the buzzer harder to click, or making the words harder to read. That’s like complaining that Deep Blue wouldn’t have won at chess if it had to physically pick up the pieces.

Instead, make the questions harder, so that the humans can compete in terms of raw brain power. Since we accept that there are still questions that humans are better at, start targeting those questions, and thus pushing the boundaries of what Watson can do.

In Deep Blue v. Kasparov terms, it’s like moving to Go (instead of messing with forcing the computer to pick up the pieces or use a camera).

If reflexis and fairness are irrelevant, why are there two people standing on either side of the silly avatar with buzzers?

Because that’s still the best measure of whether the computer can figure out the answer by the time Alex finishes speaking. If it can’t, those buzzers will mean that the humans will win.
- RedShirt77
  
  >That’s like complaining that Deep Blue wouldn’t have won at chess if it had to physically pick up the pieces.
  
  I am not sure how time factors into chess games, but my guess is that if deep blue had an arm there and would have had to coordinate itself to pick up the right pieces, it would have regularly gone over time and knocked pieces over. If there are penalties for such things, the computer probably would have lost.
  
  John henry on the other hand had no technical quip about his contest, he and the steam hammer or whatever it was had to perforn the exact same task.
  - SamSam
    
    I guess we’re really talking past each other here.
    
    Deep Blue was supposed to be a chess genius. That was his challenge. He wasn’t supposed to be a robot with co-ordination. That wasn’t what people were designing him for, that wasn’t his challenge.
    
    So it doesn’t matter whether or not he could pick up the pieces.
    
    Would a one-armed man not be competing fairly if he didn’t have arms and required someone else to move the pieces? Would a bland man not be competing fairly if he had his opponent’s moves read to him, and then spoke his own moves out?
    
    Those actions are not in the sphere of what we mean when we say someone is good at “chess.” Requiring that a computer must also pick up the pieces when playing chess is about as far removed from what “chess” is as requiring that they also be able to dance ballet.
    
    Likewise, those actions are just as extraneous as the action of physically reading a clue with your eyes or a video camera when engaging in a test of questions-and-answers. Would you begrudge a blind man who played Jeopardy and needed someone to quickly read the words and whisper them into his ear as Alex was speaking, so he would have the same amount of time to think as the other contestants?
    
    What are we trying to test? We are trying to test the ability of a computer to parse complex natural language, with puns and whatnot, and come up with a great answer. We don’t want to test if the computer is also good a ballet or if they also have hands like a human with which to push a buzzer, or if they also have eyes like a human to read with.
    
    If you asked anyone what a game of chess was about, no one would ever answer that it was about your ability to pick up pieces. If you asked anyone what the game of Jeopardy was about, no one would ever answer that it was about our ability to read words off a blue screen.
    - SamSam
      
      >> “Would a one-armed man not be competing fairly if he didn’t have arms and required …”
      
      I type too fast and my posts are too long. Watson would probably be better than me at making coherent arguments…
      
      Make that “would an armless man not be competing fairly if required…”
    - RedShirt77
      
      >you asked anyone what a game of chess was about, no one would ever answer that it was about your ability to pick up pieces. If you asked anyone what the game of Jeopardy was about, no one would ever answer that it was about our ability to read words off a blue screen.
      
      I think they would say that buzzing in first is actually a pretty significant part of the game.
      
      Don’t get me wrong, I get what you are sayin and it is amazing that a computer can do trivia. I just think it could do all the other things involved, but they don’t include it because I guess it was too much trouble.
      
      >Make that “would an armless man not be competing fairly if required…”
      
      If a no armed man requested that the other contestants get to buzz in only if he didn’t know the answer, wouldn’t that be unfair.
      
      If a computer knew boxing strategy and could relay that to a boxer, would you say a computer beat a human at boxing after the match?
    - RedShirt77
      
      >Would a one-armed man not be competing fairly if he didn’t have arms and required someone else to move the pieces? Would a bland man not be competing fairly if he had his opponent’s moves read to him, and then spoke his own moves out?
      
      Also, these examples still end up giving the player a disadvantage rather than an advantage.
jungletek

Agreed that Watson should either have some sort of simulated nervous system delay, or everyone should be able to chime in as soon as they know the answer.
hancocks

Watson: lousy at fly-fishing, however.
- Art
  
  It should look like the robot @ 2:28. Without a doubt.
  - Xeni Jardin
    
    I really loved the use of those vintage clips in the piece.
Anonymous

OK, this is weird. I had “Jericho” before i’d even finished reading the question. And i’ve been drinking (ok, and smoking a little). And i’ve never read the bible. Atheist, in fact.

Of course, i’m posting this anonymously, so my name may be Wintermute, for all you’se know. ~ring~ ~ring~

reCAPTCHA sez: breathers fascinating¹¹ (i’m not making this up)
adamnvillani

I competed against Ken Jennings on Jeopardy!, and let me tell you that by his 61st episode, his ability to time the signaling device correctly was a major factor in how he was able to keep winning. I got pretty close — we both got Final Jeopardy correct, but I was close enough that I would have beaten him had he answered “Guyana” instead of “Suriname,” (or, more to the point, if they counted spelling — he wrote “Surinam”). His advantage wasn’t entirely in the buzzer timing, as he’s very good at the game regardless, but it was a major factor in how he was able to keep winning for so long.
adamnvillani

Sorry, I never got to my point! My point being that I have to admit it was a bit satisfying to see Ken frustrated by his opponent’s skill with the signalizing device in stead of the other way around!
SAMO1415

Watson became self-aware on February 15th, 2011 at 10:55 A.M. PST…
Unmutual

Hey it’s Marvin Minsky! I’ve been to his house!

I don’t get to name drop very often :P
newsbeagle

Discover Magazine has a neat article about all the other games AI’s have been taught to play. Apparently computers are great at checkers & chess, struggle with the ancient board game go, and totally suck at Stratego, poker, and Risk.

http://discovermagazine.com/2011/jan-feb/14-who.s-smarter-human-computer-round-9-jeopardy/
febryle

Watson: “Alright Alex I’ll take obscure factoids for $100.”
Alex: “The answer is 919VBKXXJ425008.”
Watson: “Alex, what is the part number of the lateral drum stabilizer for a 2004 Kenmore dryer?”
Unmutual

They are terrible at Pictionary!
Moriarty

Bla bla bla. Where are the sexbots?
- ncinerate
  
  That does raise some interesting questions…
  
  I suppose the robotic technology exists to create something like a sex-bot. I wonder what the regulations on actually making and attempting to monetize one would be.
  
  Could you, for example, pimp out a sex bot? Could you set up a brothel with a handful of them? Would you be breaking some sort of decency law against prostitution or do those laws not cover sex with “objects”?
  
  Assuming there’s no legal ramifications, you could go simple for “Sex Brothel 1.0″…… The possibilities!
eezo

I love how Miles O’Brien smugly explains what cores and RAM are for those of us without “doctorates in computer science”.
- Xeni Jardin
  
  Give the guy a break. Many people watching this broadcast on TV won’t know what those terms mean, and it’s a polite way of explaining.
  - eezo
    
    I appreciate the need for explanation, but I grow tired of the way mainstream media delivers it. I think we’re long past the point where it’s appropriate to be cute and patronizing about computing concepts, as though they’re strictly the purview of propellerheads. These systems are a pillar of our civilization.
    
    I liked the piece otherwise. :-)
    - airshowfan
      
      I’m with eezo: Interesting and informative piece, but there were a few painful spots where a common concept was explained both unnecessarily and incompletely.
      
      I’ll generalize my annoyance (and I recognize that this is totally irrational): Whenever I’m watching/reading anything, and it defines something that I know the definition to, I feel slightly offended. It’s as if I was talking with someone and they assumed that I did not know something that I know, and this makes me feel “This person thinks I’m stupid” or “This person is treating me as though I were a child”.
      
      Of course, this typically happens when I’m watching reading something about a topic I know more about than do most people, and what I’m watching/reading is intended for an audience that has average or below-average (to be safe) knowledge on the topic. So as long as I stick to videos/audio/text meant for audiences as knowledgeable as I am, I’m fine… but that would restrict me to specialized content, and keep me away from most content in Time, NPR, PBS, the New York Times… even BoingBoing!
      
      Yes, most BoingBoing posts correctly guess what words I need defined, and what words I know. But once in a while, it will define something I already know. (I would scan the latest posts but I probably wouldn’t find an example, since this only happens, I dunno, a few times a year. I’ll think about it to see whether I remember any). Funny how, when an article/video/whatever FAILS to define a word I don’t know, I never mind it or remember it, I just look up the word. So I wonder why articles/videos seem to err on the side of defining too much rather than too little.
  - Antinous / Moderator
    
    Although, to be fair, those people tend to stick their fingers in their ears and hum loudly when you try to explain it to them.
RedShirt77

Faster than an Iphone! Just slightly.
Anonymous

Xeni,

I’m curious why you don’t consider Watson a supercomputer? I’d certainly consider any computer that has 15TB of RAM and 2880 cores and specialized programming that only runs on that specific platform a supercomputer.
Anonymous

I think Watson got “Ailurophobia” wrong because of the way the definition in the Wikipedia article is phrased (with a dash), perhaps?

http://en.wikipedia.org/wiki/Ailurophobia
plaintext

I don’t see what the big deal is. Any computer worth it’s AI salt would simply answer, “What is the answer to this question?” every time and save a lot of cycles.
Anonymous

What, a lamb isn’t considered a sheep? It’s like saying a baby isn’t a human.
Zan

Nice to see that Mile O’Brien has a new permanent science correspondent gig.
GuyInMilwaukee

Meatbags!? Xeni, I prefer Ugly Bags of Mostly Water, if you please.
RedShirt77

Why doesn’t the computer have to read? the People do.
- SamSam
  
  What do you mean? The computer does have to read. It’s just reads a lot faster at it than the humans. It’s still shown the sentence at the exact same time.
  
  Are you saying that it doesn’t have to read the words with a video camera? Why should it have to do that? My phone can read and words from its video camera, and even instantly translate them. That’s a trivial problem compared with what Watson’s doing.
  - Mark Dow
    
    It gets a text message. It should have to use OCR because OCR was one of the first great AI challenges that was solved, and so is not considered AI anymore.
  - RedShirt77
    
    “Are you saying that it doesn’t have to read the words with a video camera? Why should it have to do that? My phone can read and words from its video camera, and even instantly translate them. That’s a trivial problem compared with what Watson’s doing.”
    
    Yes, because the people have to do that. we call organic cameras eyes. it takes time to allign the eye properly, to process the words. recognize them and analize the meaning.
    
    Maybe with a computer that size it takes a second to do that, but its a heck of a lot harder then having the entire question emailed directly into your brain.
    - Guysmiley
      
      “Emailed directly into your brain”? I’d argue that both the players and computer are getting the questions delivered in the most preferable way possible. Humans are getting it visually (which we excel at) and the computer is getting it electronically (which it excels at).
      
      People pouting about the method of question delivery are totally missing the point of why this is so impressive. You could dedicate one single server in that data center to do OCR from the video feed. Yawn. Trivial. The amazing thing here is the NLP (NOT speech recognition). NLP is really, really, really hard.
      - RedShirt77
        
        Meh, I know there are probably a series of wonderful things going on inside the computer, but if the little experiment is to see if a computer can compete and win in a human game, i don’t think they should create a computerized version of the game for it to play, it should have the same requirements as a human.
        
        Lets not kid ourselves here, the main point of this is entertainment.
      - Guysmiley
        
        “Lets not kid ourselves here, the main point of this is entertainment.”
        
        Spoken like someone truly ignorant to the capabilities and limitations of computer science.
      - RedShirt77
        
        Right, they put massive expensive computer on national telivision strictly for the scientific learning involved…
        
        Spoken like someone truly ignorant to the capabilities and limitations of capitalism.
      - Mark Dow
        
        “Lets not kid ourselves here, the main point of this is entertainment.”
        
        Yes, IBM’s been a laugh minute for a hundred years now.
      - RedShirt77
        
        You know I assume they sat down with a large some of money to do computer research and looked at the options: Work on cure for cancer, track pedophiles, find osama bin laden, create the best WOW character ever, cold fusion, or Win a game show.
        
        Then they thought which of the following;
        1. What would be the most scientifically important option be?
        2. What would my grandmother find cool?
mercator

Watching round one last night, didn’t it seem strange that the first time watson got to pick, it picked the daily double? I call shenanigans!
Anonymous

As a non-American, I’ve never really understood the gimmick of Jeopardy. Why must you answer with a question? Quite a lot of the clues don’t seem to work if you do that anyway.

I’d like to see Watson take a crack at University Challenge next.
chgoliz

So now I’m thinking that one way to test the boundary of what a computer can do would be to formulate questions that take advantage of one of the more human components of Jeopardy Q/A’s: the subtle hints in the text that help the contestants put together disparate pieces when rote memorization doesn’t bring up the answer on its own. Often, those hints aren’t factual or directly on-topic: they’re linguistic plays-on-words that wouldn’t help a linear-process-type “brain” figure out the answer. Sifting through encyclopedic knowledge can be done faster and faster simply based on RAM, but tying together the puns and other word associations would mean the computer is “thinking” an awful lot like a human brain. It’s not that the questions would be factually harder, or more obscure…it’s the process to get to the answer that would be.
Mark Dow

This city was mistaken for Chicago by a computer just before human game shows became redundant.
Chairboy

My understanding is that with its 2800+ processing cores, this machine is theoretically able to run Crysis at near full resolution.