Real-time storytelling
3D World talks to Epic Games, Unity and 3rd World Studios about leading the real-time revolution across film and television
Epic Games, Unity and 3rd World Studios discuss leading the real-time revolution across film and television
The use of game engine technology in narrative film or television isn’t a new phenomenon, “Unreal Engine has been driving previsualisation and virtual production on feature films for years,” says Marc Petit, general manager of Unreal Enterprise at Epic Games. “The first use was by ILM on Steven Spielberg’s 2001 film A.I.”
More recently the technology has been utilised by Framestore for on-set virtual productions on the likes of Blade Runner 2049 and Christopher Robin. It’s also been employed by previsualisation company Halon for their work on War For The Planet Of The Apes and by ILM for final effects shots in Rogue One: A Star Wars Story. VFX house Method Studios has even used Unreal Engine as the basis for a whole new virtual production pipeline on Robert Zemeckis’ upcoming film Welcome To Marwen.
Petit continues: “Most of the major studios are leveraging realtime tools in their pipelines as part of previs, tech vis, on-set virtual production or post-production. Each year we are seeing Unreal expand into more and more areas of filmmaking, as decision makers see the creative and financial benefits of visualising an entire film earlier in the process. Real time will also be embraced by mid-budget and indie filmmakers – even though they may not be working with CG characters or environments, they can still enjoy creative and financial benefits in visualising and validating their ideas earlier in the process.”
Unity has also begun making waves in the entertainment industry, as head of Made with Unity, Isabelle Riva, explains: “The gap between real time and offline has been closing over the past few years, and it’s never been smaller. Unity’s advancements in rendering technology enable filmmakers to achieve a high-quality look in any vision or style. It’s already a secret weapon for directors like Steven Spielberg, Denis Villeneuve and Jon Favreau.”
There’s never been a more promising time for real-time technology on the big screen, but it’s also having a profound effect on smaller ones too. Effects-heavy shows are making use of it for extensive previsualisation, such as The Third Floor’s work on Game
Of Thrones. Montreal-based studio Digital Dimension has gone one step further by using it to create an entire animated series, Zafari. Petit continues: “Unreal is also proving to be revolutionary for enabling real-time mixed reality broadcasts, most notably by The Weather Channel and FOX Sports.”
The Unity engine has also been utilised on various television projects, as Riva explains: “Mr
Carton was the first animated series fully produced and rendered in Unity, for French television. This last year has been a bit of a gold rush in terms of amazing
“We are seeing Unreal expand into more areas of filmmaking, as decision makers see the benefits of visualising an entire film earlier in the process” marc Petit, general manager of Unreal enterprise, epic Games
content, like with Baymax Dreams, three shorts broadcast by Disney Television.” Meanwhile, Pakistani animation studio 3rd World Studios has been busy creating the first animated feature to be made entirely with Unreal Engine. “Allahyar And The Legend Of Markhor is about a young boy’s adventure through the wilderness of northern Pakistan and his relationship with the indigenous wildlife that he encounters,” explains Uzair Zaheer Khan, CEO of 3rd World Studios as well as the film’s director and producer.
Established in 2016, one of 3rd World’s initial agendas was to produce international quality content at an extremely competitive cost, through innovative approaches, shorter production cycles and smaller teams. Allahyar And The Legend Of
Markhor combines this with their ideological mission to influence audiences through entertaining films that highlight some of the issues plaguing us today, such as the lack of wildlife conservation, global warming and intolerance.
“Markhors are the largest species of wild goats in the world,” says Khan. “Not only are they endangered, due to poaching and a loss of habitat, but they also happen to be the national animal of Pakistan. The aim was, that if a child watches the film, they should never grow up to be someone who considers hunting a sport and would end up treating animals with the respect they deserve. We were fortunate to receive a lot of appreciation and support for our efforts, not only from the local audiences, but also from organisations like the World Wildlife Fund. ”
He continues: “Originally, we had only played games made with Unreal Engine. When we started exploring it several years ago, we didn’t understand its true potential. Even though it was regarded as one of the most powerful game engines around, it was considered fairly complicated. It was only when we dove deeper into it, that we realised its exceptional, robust and versatile toolbox. We quickly became very excited about its potential.”
At this time Epic Games had already created various shorts inside Unreal Engine, most notably the kite demo, which showcased impressive capabilities in cinematic production. Alongside this was an increasing number of cinematic video games being made with the engine. This encouraged 3rd World Studios to explore it further in film production.
“All assets and animations for the film were created using industry-standard software, with Unreal being used for set assembly and rendering out the final frames. We used all the latest tools that Unreal provided for landscape sculpting, foliage painting,
dynamics and its powerful post-production features,” adds Khan. The fact that Allahyar
And The Legend Of Markhor was 3rd World’s first film brought endless challenges for the studio, let alone the fact that they were switching to a game engine for film production. “The approach involved a steep learning curve,” Khan admits.
“Artists and engineers with experience in Unreal were practically non-existent in Pakistan,” he continues. Instead the core of their team had experience creating in engines such as Helios, Torque 3D, id Tech 4 and Unity. Khan continues:
“We had to switch and train our entire team to work in Unreal from scratch. Figuring out an efficient, working production pipeline also took a lot of effort. We faced quite a few technical limitations and had to learn, look out for and create a lot of outside-the-box, custom solutions for our pipeline.”
Despite the challenges that arise from uprooting the more traditional production pipeline, there are a number of advantages for filmmakers willing to embrace real-time engines. Petit outlines one huge benefit: “Directors can realistically visualise an entire scene up front, instantly iterating on different lighting choices, colours, blocking, framing and more, meaning their vision is locked in much earlier in the process. This drives both time and cost savings in everything from production design to VFX.”
On the animation side, offline rendering has traditionally eaten up numerous resources, with each small change to lighting or effects taking hours, if not days. Realtime rendering not only allows animators to iterate on the spot, it also enables significant cost savings in computer power and energy output. This is something Khan is keen to reinforce: “Quick pre-visualisation, cutting down hardware requirements and render times, Unreal’s ability to handle extremely large scenes and the power to make adjustments at the last minute are just some of the advantages.”
“It is a very non-linear way to tell a story,” adds Riva. “Small teams working on story development or animated shorts can iterate with all of the
“directors can Realistically visualise an entire scene Up front” marc Petit, epic Games
departments at their fingertips. Unity also places compositing up front in the workflow, creating a mission-control style platform where better judgement calls can be made on animation, timing, lighting and camera positions. Larger teams can enjoy the speed and efficiency of real-time rendering, widening the window for more iteration and creative risk. Why not try it, if it only takes a few seconds?”
According to Petit, real-time engines are fast catching up to the level of quality provided by more traditional methods. “We’re already seeing Unreal driving final on-screen pixels for blockbuster films like Rogue One where ILM developed a real-time rendering process for the droid K-2SO. This will only become more common as the technology continues to improve and people dive deeper into the tools.”
Khan is sceptical that larger studios will fully embrace real time: “They have their own tools and pipelines in place. However, smaller production houses with limited budgets have already started looking into it. We are starting to see animated content coming up through game engines. The shift in focus from game engines towards building better and easier tools for creating such content ensures that they will be industry-standard tools for cinematic content in the future. It’s great to see this community developing and growing right in front of our eyes, when just a couple of years ago it was relatively non-existent.”
Elsewhere Unity is already becoming a vital part of the pipeline for various studios. “MPC and other post houses are even integrating Unity into the suite of tools used on big budget films
“small teams can iterate With all of the departments at their fingertips” isabelle riva, head of made with Unity
like Aquaman and Justice League, or for companion experiences to future-proof IP, like with Pixar and Magnopus who produced Coco
VR,” says Riva. Whatever the future holds for real time, its potential in the film and television industry is unparalleled, and filmmakers are just beginning to scratch the surface. “From truly immersive mixed-reality broadcast segments, to crossing the uncanny valley with photorealistic digital humans, the image quality that creatives are able to output with Unreal is unprecedented, and it is only getting better,” says Petit.
Petit concludes: “Beyond final imagery, it also has the potential to reinvent the entire content creation pipeline. Once finalquality images can be achieved at any time during production, the creative process will shift to the front end. Imagine on a Cg-heavy action film, cinematographers being able to nail their lighting, and visual effects artists being able to craft their monsters, right there while everyone is still on the green-screen set. That will be an incredible moment of coalescence.”