Virtual actor looks eerily lifelike

The person in this video looks real, but she isn't. She's a digital replica of the actress Emily O'Brien created by Image Metrics and Paul Debevec at the University of Southern California's Institute for Creative Technologies (ICT). They demonstrated the high-definition animated face at this month's SIGGRAPH 2008 conference. From the press release:
Emilyimagemetttt ICT employed its high-resolution face scanning process to capture O'Brien in 35 facial poses directed by Image Metrics. This newest process from the ICT Graphics Lab places the actor inside a sphere of LED lights, illuminating the talent with a set of polarized spherical gradient illumination patterns while a pair of high-resolution digital cameras takes around 15 photographs in under three seconds.

These patterns allow the shine of the skin to be photographed independently from the main skin tone so that precise colors and characteristics can be calculated at hundreds of measurements per square millimeter. The resulting CG models provide unprecedented detail of natural facial expressions - down to skin pores and fine wrinkles - with perfectly aligned shading information that allows photo-real faces to be rendered under any illumination and viewpoint with standard rendering packages.
The Emily Project video, Image Metrics press release (Thanks, Tara McGinley!)

Previously on BB:
Computational photography


  1. “The person in this video looks real…”

    I beg to disagree. The MOUTH is especially creepy – some of her teeth float outside of her head.

  2. so, if this is out in the public sphere, imagine what the spooks in the alphabet agencies have….

  3. I agree with Bardfinn (except for extraoral dentition) — the mouth doesn’t look right to me. Definitely getting better, though.

  4. Is just the face rendered, or are the clothes, hair, and everything else rendered too?

    I think the face is in that area where you can’t really tell if it’s off, or you’re just being overly critical.

  5. Only the face is rendered. But, the motion tracking is very good. However, the face is softer than the detail you can see in the live action of the hair and blouse.

    The technology is impressive, but they are leveraging off the verisimilitude of the live action plate.

  6. This has made the blog-rounds in the last week and it should be clarified that this is merely a facial replacement. Everything outside the eyes, nose and mouth is still the real actress. An interesting experiment, but quite deceiving it its presentation, and really not so amazing in the world of facial animation.

  7. Okay – The teeth don’t float outside the face /visibly/ – Watch the whole clip, remember where her teeth are in the mouth. watch it again. Be amazed as your mental model of teeth dissolves and reforms as her cheeks create and destroy virtual teeth! The inside of her mouth is larger than the outside of her head.

    Also: Lips should tilt down when the head does.

  8. When they screw with the filters at the end, it’s super creepy.

    I just don’t understand why they felt they had to make this. Is it a “because we can” thing?

  9. Wait

    What is the point of this.

    According to what she says in the video. THe actor performs the entire sequence and then they turn that digital video into digital animation. and then they can lay different textures on that.

    So this is essentially a virtual make-up generator.

    Doesn’t create a unique animated character, simply maps and recreate a video image.

    I am sure it is an advancement, but it is a boring one.

  10. This isn’t really for mass amusement. If you’re wondering what the point is, you’re being a little short sided.

    In the world of CG you see tech demos like this all the time. It’s showcasing innovations so that software/hardware developers and/or studios will hopefully adopt the technology into their development schedules/pipelines. Every impressive CG effect, from volumetric liquids to realistic hair and fur had such boring tech demos before the effects became commonplace. Just because they were attempting to create a photo realistic sequence doesn’t mean that you couldn’t use the same tech to do practically anything that face capturing is already doing (like the super punch shot from the end of the last Matrix movie).

  11. I think this falls into the “because we can” category. Since it’s only the face that’s captured and comped back onto the real actress, how does this benefit any kind of fx production? It’s still a video of an actress, not a 3d character.

    I honestly don’t see where this fits in an fx pipeline. If the face animation has to be comped back onto the real actor, what kind of creative choices does that offer? Most anything you might want to do with it can be done more easily and cheaply through makeup fx.

  12. I’d be much more impressed if they showed us the actress’ face to compare with the one we see in the video, to get an idea of how much they’ve managed to change. Double points if the “actress” has a thick beard in real life.

  13. “This isn’t really for mass amusement. If you’re wondering what the point is, you’re being a little short sided.”

    CGI not for mass amusement? me, shortsighted or sided?

    “Every impressive CG effect, from volumetric liquids to realistic hair and fur had such boring tech demos before the effects became commonplace.”

    The super realistic fur on the CGI Lion attacking a guy is somewhat less impressive if they must film a Lion attacking a guy to create it. Unless someone comes up with a movie Idea containing a pack of albino Lions, this advance might just be pointless.

  14. Guys, did you miss where she said the performance can be used to drive any facial rig? I would imagine the subtleties of the performance could be mapped to everything from Jabba the Hutt to a simulated Heath Ledger. Right?

  15. I’ll second (or is it fourth or fifth, by now?) the sentiment of wanting to see the original footage to compare with. If they could take the face from one shot and paste it onto a body from a different scene, or modify proportions a bit to make the face look significantly different, or even apply significantly different lighting to the two scenes, I could see how this is useful. But being able to take a video and do nothing more useful that reproduce that same video…seems pointless to me.

  16. “THe actor performs the entire sequence and then they turn that digital video into digital animation. and then they can lay different textures on that.

    Doesn’t create a unique animated character, simply maps and recreate a video image.”

    It’s incredibly useful for videogames, which from what she was talking about are what this is being targeted at.

    Think of all the work Valve put into making facial animation algorithms for the Source engine. Now think of how much time they could have saved if they were able to just take the animations they needed directly from the actors. Instead of creating the animations from scratch, they get them pre-made from this company.

  17. Those that think that it looks ‘creepy’ are not being ‘hypercritical’. The human eye-brain combination has, for millions of years, been evolving the ability to discern minute facial features as indicators as to friend, enemy, lover, parent, liar, etc., and it has become very good at it. The fact that these ‘animations’ look creepy is just your brain telling you that there is something wrong with the way this face looks. When computer animators get to the point where these animations don’t look ‘creepy’ anymore, then they will have achieved their goal of ‘natural’ looking CGI faces.

  18. I don’t think it looks creepy, but on the other hand I don’t think motion capture looks creepy. Justin Timberlake on the other hand … well, talk about looking almost human.

    Anyway, I think these type of things should be subjected to a blind taste test of sorts, to see if people can really tell the difference between the fake Emily and the real one. I bet if a lot of people were shown a picture of someone performing and were told it was computer generated, they would say it looked fake.

  19. I actually saw Paul Debevec giving a talk at Adelaide Uni a few weeks back. He showed us the process for this (along with heaps of other cool CG work).

    Here’s a shorter, earlier version of the same talk.

    For reference, Paul is one of the main people responsible for HDR and image based lighting.

    Anyway, as others have noted the context is important – this is an early demo of the next stage of a technique they’ve been refining for several years now (remember Doc Oc’s death scene in Spiderman 2? His face was mapped with way).

    Also, in the best tradition of computer science, they’re looking for (and finding) shortcuts and cheats to get better, faster results – photo-mapping the face rather than raytracing the whole scene.

    I’d thoroughly recommend seeing Paul talk if you get the chance. Some of it went over my head, but if you know a bit about CG and want to know where it’s headed, it’s brilliant.

  20. What can this be used for?

    1) When political leaders are sick/injured/dead, they can use this tech to appear on TV to convince the populace that they are fine.

    2) TV studios and the recording industry could create “stars” that will not need to be paid. These virtual people would be “driven” by low-paid actors, whose faces would be replaced.

    3) If you kidnap somebody, but accidentally killed him in the process, you could is this to convince the victim’s family that he is still alive.

    4) No longer would it be necessary to torture anyone to force them to confess to crimes they did or did not do. You could just fake the confession.

    5) For tech support. In 5 years, you would be using video-chat instead of telephone calls. But we need to show something on the other end. And instead of talking to a call-center in India, you would be talking to a non-existent person in a computer.

  21. When I was watching, the only thing I could think was, “Still not out of the uncanny valley yet. Still looks weird around the mouth and eyes.”

    Then they showed the real actress and all became clear… She actually looks like that (except for the makeup).

    They did a very passable recreation of a woman who has an unusual look (large teeth, slightly flattish face) and it made the whole thing look fake. Her distinctive features draw attention to the very parts of the face that we expect to look fake in CGI.

    If they did it again with someone who has a more standard “television face”, it might be really impressive.

Comments are closed.