Oped: Augmenting AR
James Burrows, technical director at Immersive Studios, on how we’re sidestepping the limitations of handheld devices
James Burrows of Immersive Studios talks AR experiences
“QUITE SIMPLY, THIS PROCESS RAISES THE BAR FOR THE VISUAL QUALITY”
By now, we’re all well-versed in what augmented reality can do. It lets us place virtual objects in real environments and interact with them through a handheld device. Ever since Pokémon Go’s launch, the capabilities of augmented reality have become more and more sophisticated. But there are still limitations – and visual quality is one of them. That’s why, at Immersive Studios, we’ve been experimenting with ways to augment the visual assets in AR experiences without overloading the devices used to view them. Technical director James Burrows explains.
STRIVING FOR MORE
A little while back, Nvidia shared a demo for its 5G-based Cloud XR where it streamed highly detailed CAD models to VR and AR devices. What was striking was seeing AR being able to support the level of detail in a CAD model. At Immersive, we create AR experiences for a range of uses and invariably come up against the same balancing act: how to get the best visual quality for the assets while keeping the app’s size down. It’s always a bit of a compromise, although AR platforms are evolving all the time. However, we wanted to push the boundaries a bit further to find a way of building AR experiences with high-fidelity, Pc-grade graphics ourselves. So we did.
PUSHING THE TECH BOUNDARIES
We devised a way of using video streaming to enable a central PC to stream high-end assets to a handheld device, where they can be viewed in real-time in AR – essentially taking the job of rendering 3D assets away from the device and giving it to a more powerful computer instead. In a nutshell, the phone sends positional information to the PC, which renders the frame in a game engine, that frame data is then encoded into a video stream, which is transmitted back to the phone and composited against the phone’s camera video feed.
WHAT DOES THIS MEAN?
This means that for any situation where you need a desktop-class GPU to get the best results, you can now do it in AR on a cheaper device. This opens up AR as a platform for complex 3D models that mobile hardware wouldn’t usually be able to render.
SO WHAT ARE THE BENEFITS?
• It looks better – Quite simply, this process raises the bar for the visual quality, and even the interaction available, for AR – opening it up as a more useful tool capable of showing highly realistic models, in turn making the experience more immersive.
• Stream to multiple devices – With a central PC streaming the graphics over Wifi, it’s possible to stream the same feed to multiple devices without the need for multinetworking, allowing multiple users to see and interact with the same session.
• Practical – This process can make it practical for modellers to preview their assets without having to build a whole app or push it to the App Store. Instead, using PC rendering, they can stream their models to a device and drop it into the real world easily.
• Smaller app sizes – With the onus of rendering assets removed from the device, the app size will come down, making it both more appealing to download and enabling you to have more fun with your experience.
• Easier to update – If your app’s content is streamed from a central PC and the mobile app effectively works as just the viewer, then you don’t have to update the app itself if you need to make content changes. Instead, you can update via the central source.
• Level playing field – Devices of differing levels of power and capability can all access the same graphical experience so long as they’re able to render a simple video feed.
At the moment, this process requires a good Wifi connection so currently it will be more feasible in setups where you can control the local Wifi network and how people connect to it. However, with the upcoming arrival of 5G and Edge streaming, it’s feasible this way of running complex AR experiences could become much more common. In the meantime, we’re enjoying experiencing AR that looks really, really good.
Find out more at weareimmersive.co.uk