How an AI neural network brought Luke Skywalker's voice to The Book of Boba Fett

At the end of season 2 of The Mandalorian (which aired in December 2020, so spoiler alerts are meaningless but here we are), a DeepFaked CGI version of Luke Skywalker makes a surprising appearance to recruit the young Grogu for his upcoming Jedi revival scheme. It was both a cool and creepy moment, with the impressive digital imaging nestling way too comfortably in the heart of the Uncanny Valley. It was so ineffably unsettling that a random YouTuber tried to fix the scene himself — and did such a good job that Disney hired him.

When CGI DeepFake Luke Skywalker returned in The Book of Boba Fett, his robotic performance was noticeably more convincing. But there was something about his voice that was still … off. (In my group chat with my friends about the show, it was the first thing I commented on.) There was apparently a good reason for that: it was made entirely with AI. From Esquire:

Hamill didn't record lines for The Mandalorian, according to Jon Favreau. In Disney Gallery: Star Wars: The Mandalorian, he revealed, "Something people didn't realize is that his voice isn't real. His voice, the young Luke Skywalker voice, is completely synthesized using an application called Respeecher." 

Read this horrifying description of Respeecher from a sound editor who worked on the series, Matthew Wood: "It's a neural network you feed information into and it learns," Wood says. "So I had archival material from Mark in that era. We had clean recorded ADR from the original films, a book on tape he'd done from those eras, and then also Star Wars radio plays he had done back in that time. I was able to get clean recordings of that, feed it into the system, and they were able to slice it up and feed their neural network to learn this data."

Whereas Luke's physical performance was created by blending Mark Hamill with a younger actor, Disney apparently decided that it wasn't worth their time to hire noted VoiceOver artist Mark Hamill to voice a character played by Mark Hamill, and outsourced the job to a computer instead.

This is both creepy, and impressive, because it won't be long until that VoiceOver technology climbs out of the Uncanny Valley.

The Unbridled and Mildly Horrifying Evolution of CGI Luke Skywalker [Brady Langmann / Esquire]