State of the Art: The Truth about Digital Doubles
Why and how to animate characters that are indistinguishable from real humans.
Why and how to animate characters that are indistinguishable from real humans, according to Cubic Motion. By David Barton
To create a truly photorealistic human character, whether it’s for a feature film or video game, every element must be at the absolute highest possible quality. We’ve all seen countless human beings throughout our lives, so each and every one of us is an expert at judging whether something is realistic, lifelike and believable. When one element doesn’t match up to the rest in a digital creation, whether it’s the level of realism in skin textures or the cadence of movements, everything falls apart. One subpar component will disrupt the entire illusion.
That’s precisely why utterly lifelike human characters, or digital doubles, are a relatively recent phenomenon. It’s an accumulation of techniques and technology, both baby steps and massive leaps, all piled together, polished and brought in sync to enable truly revolutionary results. Digital humans can blend in seamlessly amongst real people, and video game characters can look and act more like a person than just an intricate mesh of polygons.
It’s all about immersing the viewer or player in the story and performance. The level of added immersion provided by a photorealistic character is dramatically higher than one that doesn’t quite hit that high bar. You believe the characters now. No longer do you watch a video game cinematic and let it wash over you, or just give it a pass because it’s a video game. Now you’re fully captivated and immersed, watching a human being perform in front of you.
Capturing the Human Face
Delivering that level of fidelity starts with a photogrammetry scan, capturing the intricate details of the actor for a static asset — from skin textures to eye shapes and everything else that defines the human face. But an indistinguishable model that doesn’t move is effectively useless, so you have to build a system to animate the character.
Facial Action Coding System (FACS) movements have been the industry standard for capturing any kind of expression that the human face is capable of producing, but we can add significantly more nuance now with video-based facial capture. With head-mounted cameras, either single or multiple for added depth, we can get closer to the real thing. But it’s just a performance until you can solve it and turn it into data to drive the digital character.
That’s a very simplified way of looking at a process that is incredibly intricate and extensive. However, it’s one that is well worth the effort when you see the kind of results that we at Cubic Motion have contributed to some of the biggest games in recent memory.
We handled the facial animation for Sony’s God of War revival that recast hack-and-slash hero
‘Creating photorealistic
characters allows us to
tell new and better
stories, deliver previously-impossible
performances, and
deliver ever-growing
immersion.’
Kratos as a damaged father, who must trek with his son to honor a loved one. It’s because of that level of animation that players believed those characters and the others around them, garnering widespread praise and Game of the Year awards for the PlayStation 4 smash.
Capturing those kinds of facial performances also elevated Insomniac Games’ Spider-Man, an epic-scale, open-world superhero game that’s grounded in the emotional moments between Peter Parker and his flame, Mary Jane Watson. That hyper-lifelike facial animation helped keep such a fantastical tale feeling realistic, and earned a Visual Effects Society Award nomination for Outstanding Animated Character in an Episode or Real-Time Project for well-known Marvel nemesis Doc Ock.
Cubic Motion’s computer vision technology allows us to build highly-accurate models of whatever is visible by tracking all facial features in a very dense manner. That’s hundreds of data points in each frame, using multiple cameras to reconstruct that into 3D data to maintain all of the depth with the performance.
Creating photorealistic characters allows us to tell new and better stories, deliver previously-impossible performances, and deliver ever-growing immersion. From improving game engines and rendering techniques to enhanced lighting and animation, the results just keep getting better and better. We’re already crafting the next wave of standout performances, and given how far photorealistic humans have come in just the last five years, just imagine what we’ll be able to do five years from now. ◆
As a producer at Cubic Motion, David Barton specializes in high-fidelity facial animation for AAA games, feature films and special next-gen projects. He has delivered hyperreal characters for clients such as Sony, Ubisoft, Electronic Arts and Activision.