Palo Alto firm takes AI to the movies
Inside an old Palo Alto auto body shop, Stefan Avalos pushed a movie camera down a dolly track.
He and a small crew were making a short film about self-driving cars. They were shooting a powder-blue 1962 Austin Mini, but through special effects the rusted relic would be transformed into an autonomous vehicle that looked more like the DeLorean from “Back to the Future.”
Stepping back from the camera, Avalos referred wryly to the movie he was filming as “Project Unemployment.” The film was a way of testing new technology from a startup called Arraiy, which is trying to automate the creation of digital effects for movies, television and games.
This new type of artificial intelli-
gence, which is also being developed by Adobe and in other technology industry research labs, could ultimately replace many of the specialists who build such effects.
“This is no joke; it will put people out of work,” said Avalos, a Los Angeles filmmaker who also runs a visual effects house. “The artists are safe. But it will replace all the drudgery.”
For three decades, computer-generated imagery has transformed how movies and television are made. But building digital effects is still a painstaking and tedious process. For every second of movie time, armies of designers can spend hours isolating people and objects in raw camera footage, digitally building images from scratch and combining the two as seamlessly as possible.
Arraiy is building systems that can handle at least part of this process. The company’s founders, Gary Bradski and Ethan Rublee, also created Industrial Perception, one of many robotics startups snapped up by Google several years ago.
“Filmmakers can do this stuff, but they have to do it by hand,” said Bradski, a neuroscientist and computer vision specialist with a long history in Silicon Valley. He has worked with companies as varied as Intel and augmented reality startup Magic Leap.
Backed by more than $10 million in financing from Lux Capital, SoftBank Ventures and others, Arraiy is part of a widespread effort spanning industry and academia and geared toward building systems that can generate and manipulate images on their own.
Thanks to improvements in neural networks — complex algorithms that can learn tasks by analyzing vast amounts of data — these systems can edit noise and mistakes out of images or apply simple effects and create highly realistic images of very fake people or help graft one person’s head onto the body of someone else.
Inside Arraiy’s offices — the old auto body shop — Bradski and Rublee are building algorithms that can learn design tasks by analyzing years of work by movie effects houses. That includes systems that learn to “rotoscope” raw camera footage, carefully separating people and objects from their backgrounds so that they can be dropped onto new backgrounds.
Adobe, which makes many of the software tools used by today’s designers, is also exploring machine learning that can automate similar tasks.
At Industrial Perception, Rublee helped develop computer vision for robots designed to perform tasks like loading and unloading freight trucks. Not long after Google acquired the startup, work on neural networks took off inside the Mountain View tech giant. In about two weeks, a team of Google researchers “trained” a neural network that outperformed technology from the startup that had taken years to create.
Rublee and Bradski collected a decade of rotoscoping and other visual effects work from various design houses, which they declined to identify. And they are adding their own work to the collection. After filming people, mannequins and other objects in front of a classic “green screen,” for example, company engineers can rotoscope thousands of images relatively quickly to be added to the data collection. Once the algorithm is trained, it can rotoscope images without help from a green screen.
The technology still has flaws, and in some cases human designers still make adjustments to the automated work. But it is improving.
“These methods are still rough around the edges — there is still a long tail of things that can go wrong in unpredictable ways — but there aren’t any fundamental roadblocks,” said Phillip Isola, a computer vision researcher at the Massachusetts Institute of Technology and OpenAI, the artificial intelligence lab created by Tesla CEO Elon Musk and others.
Avalos said he thinks this work could ultimately supplant work done by his own effects house. But he is comfortable with that. He already farms out many of the more tedious tasks, via the Internet, to workers in other countries.
As hardware improves, these methods will help generate preliminary effects right on the movie set, giving filmmakers a better idea of what the finished product will look like, Bradski said. The Mill, an effects house from London, recently used Arraiy’s technology to create a system that could instantly apply effects that change the look of an automobile.
If tech companies can help automate some of the grunt work involved in creating special effects, creative people will have a chance to try new things, said Pasha Shapiro, a filmmaker and special effects artist who has also worked with Arraiy.
“Some work is so tedious that it is not practical,” he said. “That is where technology can help even more.”
“The artists are safe. But it will replace all the drudgery.” Filmmaker Stefan Avalos