Everything In-camera
Where is virtual production going, and how can you ride the wave? We chat to Nic Hatch, CEO of Ncam, to find out
CEO of Ncam, Nic Hatch, answers our questions about advancements in virtual production technology
If you read the headlines (especially Mandalorian-related ones), it can seem like virtual production is a wholly new concept rather than a constantly evolving methodology that’s been with us since, at the very least, Digital Domain’s work on The Jungle Book. However, what used to be relegated to previz is starting to touch nearly every department. With a little bit of planning and client sign-off, nearfinal VFX is making its way into film and episodic productions much earlier in the process, giving rise to greater collaboration between the artists and supervisors on both sides of the spectrum.
And yet, for all this momentum, virtual production is actually a fairly young process with limitless room for growth. The question is, where will it go next and what will ultimately drive it in that direction? To answer this, we sat down with Ncam’s CEO, Nic Hatch, as it’s quite literally his job to think about how productions are going to blend real-time 3D, XR and live-action content now and in the future.
3D World: Virtual production is all the rage right now, especially with COVID-19 changing how we work day to day and, even though people are making a push for it, is it ready for primetime?
Nic Hatch: The short answer is ‘yes and no’. In some instances virtual production works perfectly, and in others it’s a bit like the Wild West. Virtual production techniques have been around for a long time, but not many studios were using them. Then, when the pandemic hit, everybody started rushing to figure it out. But it’s a complex puzzle. Some of the big pieces exist, e.g. LED walls, game engines and the type of in-camera VFX that Ncam Reality enables, but other pieces are either brand-new or non-existent.
Virtual production is still very much in the early adoption phase. It’s been used successfully on a handful of productions, but
every time is like a new science experiment. The off-the shelf tools aren’t all there, so studios need to create custom solutions to make it work for them. That’s what we’re seeing now – the pandemic created a need for wider adoption of virtual production, but not every studio has the means, team or tools to put it into practice.
3DW: What are virtual production’s current strengths and weaknesses?
NH: When you can see everything in-camera, you can make better decisions. From a filmmaker’s perspective, you’re aligning everything through the camera again, which restores some of the creative freedom they had before visual effects came along. And if we design the tools correctly, we won’t have to give up anything. You’ll have ultimate flexibility – so you can get the shots you want on set, but also tweak them easily in post.
Arguably the biggest struggle is the idea of changing your entire workflow. It used to be that filmmakers would come to set with just an idea; in virtual production, you have to come with most of your 3D content already finished. Which means a lot of the creation process has to be done weeks – or even months – earlier than people are used to. It takes a lot of planning. But in the end, it gets everyone on the same page much faster.
3DW: What do artists need to learn if they want to stay on top of this?
NH: Real-time engines are a key element in virtual production, allowing flexibility and quicker turnaround times. There’s also a shortage of talent that can apply these tools to a VFX workflow. If I were coming into VFX right now, I would want to know all of the tools that would enable me to deliver quality work. Right now, there is a gap here – current VFX tools aren’t designed to be used on set, or in real time. Will tools be released that can bend game engines to their will? Yes, I believe we’ll see lots of apps and interfaces that are built around or on top of real-time engines. But right now no one tool can do everything, so artists need to have a breadth of knowledge. We will always need great talent to get great results.
Additionally, since virtual production is still in the early phases, the number of people who really understand the complete puzzle and how all the pieces fit together is limited. So that’s an area where artists can definitely make themselves stand out.
3DW: At this point, what are the virtual production essentials when it comes to gear? Do you see that staying pretty consistent for a while or changing as new tech gets developed?
NH: Some of the classic production tools like high-quality cameras and lenses will always be essential. That’s a big reason we’ve specialised in tracking technology – an in-camera view makes a huge difference for creatives on set, but you need to align the real and the photorealistic to make that happen.
Mocap will continue to be a big piece of the puzzle, as well as real-time compositing and visual effects. LED walls are just getting off the ground, but they will be widely used over the next
few years, and will continue to improve in both quality and functionality. Right now, virtual production can be a very expensive undertaking, because the off-theshelf tools don’t exist – but we expect they soon will.
3DW: The Mandalorian has really become a bellwether project for virtual production. Do you think the techniques being used, like the LED screens, should be looked at as the gold standard now or are there many ways to get to an excellent result?
NH: Everyone is excited about LED walls, and they are wonderful if you have the resources. But what people may not realise is that they have their own limitations. Improvements we’ll see in the next few years will include upgrades in resolution, brightness and especially in the way we control them, allowing teams to drive content in an easier, more affordable way. However, it’s very unlikely that we’ll see every studio with a 360-degree LED wall!
It’s also not just about the wall. You still need to have great content to feed into the wall. We’re always going to need systems to reconcile things like camera positions, lighting changes, etc. So motion capture and camera tracking are still a huge piece of the puzzle.
That said, LED walls are a big improvement over green screens. Green screens are a necessary evil, but they can destroy creativity from a filmmaking perspective. With LED walls, actors don’t have to imagine what’s behind them.
Directors don’t have to imagine what a background or effect might look like. There really are a multitude of benefits to LED walls. And you don’t even have to put the capital down to buy one at this point; you can rent them.
3DW: Is virtual production only going to be accessible to the major studios for a while or is there room for everyone? If the latter, how can smaller studios take advantage of this moment?
NH: We’re going to see a rapid democratisation of virtual production technology. At Ncam, we’re talking to people about all types of budgets and productions, from blockbusters to independent projects. We are also seeing the barriers to entry lowering,
which is great because smaller studios are likely to benefit more. Virtual production techniques can help save a great deal on both production and post-production costs. The more planning and heavy lifting you do up front, the more you can save on the backend. You can bring locations to you, shoot any time of the day or night. The sun never sets on an LED wall – unless you tell it to.
3DW: How close are we to seeing final VFX coming out of virtual production workflows?
NH: We’ve already seen it. There are plenty of projects where you can’t make changes; for instance, in broadcast and news the content goes live to air. For film and TV we are seeing it a little bit, but it’s few and far between. That is the ultimate goal, of course, and we’ll see it happen more and more as time goes on and studios change their workflows to adapt. We’ll likely see the biggest adoption in television and episodic first, because the repetitive nature of that industry (i.e. filming multiple episodes and seasons) means the tech will pay for itself faster.
3DW: How are game engines supporting this effort? What could they develop to be even more useful for film, broadcast and live event audiences?
NH: There’s work to be done on the artist/user interface, but the platform is there, and you can do incredible things with it. The classic thing you hear is that game engines are designed for games, and not necessarily conducive to creating other content like film and television. That said, we’re seeing a lot of effort from nearly every part of the industry. Epic is heavily invested in expanding their toolsets, for instance. Look no further than their Megagrants programme, which is already empowering companies to create the next generation of real-time production tools.
3DW: A lot of people think of virtual production as a VFX tool, but it’s also being used to help reshape news broadcasts. Why is this method so effective?
NH: Any time you need to create and show content in real time, virtual production is a great tool. Broadcasters have long used green screens. Soon these will be replaced by LED walls, which offer more flexibility and enable news teams to create content with better storytelling in mind. The bar has definitely been raised for creating realistic virtual sets, which are scalable and moveable, with relative ease. With virtual production, all the limits broadcasters used to face are gone.
3DW: Where is virtual production going next?
NH: Now that the early adoption phase is finally opening up, there’s a real need to make virtual production more approachable. We need to make this technology more affordable for more people. Adopting virtual production is quite a change, so we need to make it easier and more scalable.
Virtual production offers huge efficiencies. If your virtual sets look completely realistic in real time, your post-production time either doesn’t exist or is maybe 10 per cent what it used to be. No more huge render farms churning away; you do it all on set. This means quicker turnarounds, faster approvals, and big savings on costs and reshoots. Who wouldn’t want that?