Seeing is believing

Elly Strang talks with RealityVirtual's Simon Che de Boer about creating slice-of-life, hyper-realistic VR experiences

How did one New Zealander go from seeing his house and all his prized possessions go up i n fl ames, to becoming one of the world’s most soughtafter virtual reality directors and cinematographers?

Elly Strang sits down with realityvirtual. co founder Simon Che de Boer to get to know the man who says he’s i n the business of ‘slice- ofl i fe’ virtual experiences that are hard to distinguish from the real thing.

With a shock of white hair tousled around his face like a lion’s mane, black clothing from head to toe and a pair of rare Dillon Optics diffused aviators perched on his head, Che de Boer is a striking individual to first lay eyes on. Guessing what career he’s in is an interesting challenge for strangers to grapple with: when he’s traveling through Arab countries, they flock to him and ask for his autograph, suspecting he’s a Targaryen from Game of Thrones. When observing the large, heavy black cases he’s lugging around behind him, you might think he’s a rocker in a band. But what’s inside the case is actually tens of thousands of dollars’ worth of heavily modified photography and virtual reality equipment.

Che de Boer’s hyper-realistic VR cinematography has accrued rave reviews from his industry counterparts around the world, from being called “some of the most exciting work I’ve ever seen” by Dunedinbased Animation Research’s Ian Taylor to Santa Clara based NVIDIA Rick Champagne stating “The realism and detail in Simon's work helps us see the beauty in the world we live in again”.

But what might astonish viewers of his creations is Che de Boer is actually considered legally blind under New Zealand law.

He took an Uber to and from our interview as he can’t operate a car and when the aviators come off, his eyes are noticeably crinkled as they swivel around, straining to take in the elements of the world around him. If you observe him editing his work on a computer, his nose is almost brushing against the screen as he takes in every infinitesimal detail.

But it’s this so-called disability that, if viewed in a different light, has led to a super-ability.

Che de Boer says his vision has made him develop an extremely good environmental spatial awareness and memory, as he maps out places in his mind carefully before becoming confident in them, and it’s this skill that helps him produce such detailed work.

“So unless someone decides to put 5000-yearold gold statue somewhere in the middle of the ground without telling me, I'm fine,” he says.

This specific scenario didn’t actually happen, but a recent trip to Egypt to shoot Nefertari: Journey to Eternity – an educational VR experience that lets viewers explore the ancient Egyptian queen’s tomb – did get Che de Boer’s heart racing when he found out the team who hired him didn’t take out liability insurance.

“You're working with these priceless objects and you know, you're freaking out,” he laughs.

He says the other advantage of this condition is he doesn’t get lost in the details of the reallife environment he’s documenting, as he simply doesn’t have that luxury.

“Where others are bombarded with so much detail, they might have a narrow vision of what they're actually doing. The reason the detail is so good in our experiences is because in the real world when we're scanning them, I can never quite get close enough to truly appreciate it. ” he says.

“If I was to lean into the Mona Lisa the way I'd want to, I’d be right up against that masterpiece, moisture breathing all over the damned thing and you know, I’d be arrested. With virtual reality, the level of intimacy is compelling as these limitations are not in play anymore.”

Rising from the ashes

Unique vision aside, Che de Boer’s deep dive into the virtual reality world was really spurred from pure tragedy. Prior to RealityVirtual, Che de Boer says he was a 31-year-old in a happy place, living in an apartment in the heart of Auckland city for eight years for which he had extensively renovated. He had his then four-year-old daughter and his partner Chantelle with him. He played drums, sung in a band and had a few “geeky” hobbies on the side that involved playing around with computers.

On the morning after Simon’s 31st birthday, an Airbnb guest staying in the apartment spotted smoke and flames starting to take hold of the place. “Is this normal?” he asked Che de Boer, who replied no, and swiftly went into fight-or-flight mode, climbing half-naked up a ladder and fleeing to a spot on the roof that was safe from the flames. He and fellow tenants were later rescued by firefighters, but his home – and all of his possessions – were not so lucky.

The fire was classed as an arson, and the timing of events was suspicious considering a five-year-long custody battle over Che de Boer’s daughter was about to come to a close. Four days later, the court threw out the case due to him not having a home and he lost custody.

It was after this that the great personal losses kept coming. Che de Boer and his partner Chantelle’s relationship deteriorated due to the multitude of stresses, and later that year she took her own life.

“I went through hell. Literally everything in my life went from on top of the world to utter shit. You know, I lost my daughter, lost my partner, lost my house in a period, in just a short period of time,” he says.

“It almost broke me. Well, it probably did break me and made me like some sort of weird mad genius, as I had to absorb myself in something. It was a catalyst to force me to just really put all my mental effort into something.”

That something, it turns out, was virtual reality and visual effects. When returning to his beloved creative medium of music reminded him too much of what he’d lost, he decided to pivot into another sector and leave that chapter of his life behind him.

The reason the detail i s so good i n our experiences i s because i n the real world when we're scanning them, I can never quite get close enough to truly appreciate i t. At l east i n virtual reality, one can. If I was to l ean i nto the Mona Lisa the way I'd want to, I’d be right up against that masterpiece, moisture breathing all over the damned thing and you know, I’d be arrested. With virtual reality, the l evel of i ntimacy i s compelling as these l i mitations are not i n play anymore.

While suffering from post-traumatic stress disorder, Che de Boer came up with the idea of creating a virtual replica of his house so he could regain what he’d lost and return home. He locked himself away from the world with a computer and a first generation Oculus headset and began researching how to recreate real world environments in virtual realities.

One talk he had come across prior that proved influential was a 2008 TEDTalk showcasing Microsoft Photosynth, a now defunct app that allowed you to crudely navigate through your photograph collections in 3D space.

“This was the first time I’d seen such magic, I just didn’t realize at the time what it was,” he says. “I remember all that time back then wanting to take hundreds of photos of my house, I just couldn’t afford a camera.”

This introduced Che de Boer to the magic that can be created with photogrammetry – the 3D reconstruction of real-life assets by using photos – and down the rabbit hole he went.

While at first he was researching whether he could use old family photographs to rebuild his burnt-down home, he wound up spending 18 months absorbing all the information he could on how to go beyond photo-realism of locations to creating a sense of actual realism, using photogrammetry, visual effects signal processing, virtual reality and game engines.

It turns out he was onto a good thing. After posting early works of his photogrammetry-to-VR pipelines on YouTube, calls began to come in. Investment giants like Chinese-based Shanda Group were all of a sudden enquiring about his intellectual property. Che de Boer realised he needed to take his discoveries more seriously and begun extensively documenting some more real-life locations in even more detail.

But it was not without trial-and-error. He says in the initial research stages, he crashed, invested or lost approximately $20,000 worth of drones while shooting locations. But he was undeterred, as the timing was aligning for everything else.

“The computers had just got fast enough, while the algorithms that other people were inventing for the initial point cloud gathering had just kind of hit a holy grail of being able to do larger scenes,” he says. “So that's when it was like, how cool would it be to just do whole environments?”

Given his past tinkering, it’s little wonder Che de Boer fell into such a high-tech space. Back before Alexa

and Siri were a phenomenon, around 2008, Che de Boer had hacked together different bits of software to create a fully natural language, home automated PC. The talking computer was called Brian – with a posh British accent and all – and he’d converse and deliver any whim of information to him and his amused daughter. A stripped down version proved a hit on TradeMe, so he sold a couple off to fill the coffers.

But still, what he wanted to achieve with VR still seemed like a few years away, technology-wise, until he was introduced to a piece of software called RealityCapture. This proved instrumental in helping him create truly large scale environmental scans.

Combining these different, incredibly complex technical elements – photogrammetry, visual effects signal processing, automatic extrapolation and game engine software – is ultimately the secret sauce behind RealityVirtual. Che de Boer says RealityVirtual has utilised these different techniques and “put them on steroids”.

“We have derived delighting techniques, deep learning techniques for super-enhancement and much more. Our experiences are a magnitude higher in detail than that of the nearest competition due to our clever data and memory management methodologies.”

Evercoast and 8i co-founder Sebastian Marino, who has worked on the production of the likes of Star Wars to Avatar and won an Academy Award for technical achievement, says Che de Boer is working a very nascent field with technology that is not yet mature.

“What he does is not easy,” Marino says. “He has the vision and confidence to bring together disparate ideas such as photogrammetry and machine learning, which are individually complicated enough to make most people's heads explode, and he's able to use these as tools to create beautiful art that looks effortless in its realism. What a great thing to give to the world.”

Che de Boer puts this down to his mind being a sponge for different ideas.

“I routinely describe English as my second language, and abstract thought as my first,” he says.

“The first TEDTalk I ever watched was this great TEDTalk called ‘When ideas have sex’, and that’s kind of the philosophy behind it – mixing things together, trying to find someone else you can talk your madness to and hopefully they grasp the concept,” he says.

As well as this, he says that part of the reason he broke new ground in the VR industry is because he had no legacy structure to follow, so he didn’t think what he was trying to achieve in terms of hyperrealism was crazy.

“I found solutions to problems that people had been having in regards to managing such large datasets. It was my thing. You get all these billions of points and you’re like, ‘I want to have that in real time’, and people are like, ‘No, you have to scale it down to 1/1000th of that’. I didn't know that because I had never been classically trained in the visual effects industry.

“In VR, your mind expects to see every blade of grass, every bit of etching in the wood and all the imperfections and all the rest of it. And so, I just basically ran into a problem and just refused to have it be a problem. And as a result, I found clever ways of data management and memory management, which I can't talk about in detail because that would be giving away a massive chunk of our secret sauce. That's one of our many secret sauces, and that's why we're interesting, we're constantly trying to make our own pipeline obsolete.”

He expects other players will eventually catch on to RealityVirtual’s IP.

“They’re going to work it out very soon I think, but honestly – we don't care, because we will be doing the next thing,” he explains.

It almost broke me. Well, i t probably did break me and made me l i ke some sort of weird, mad genius, as I had to absorb myself i n something. It was a catalyst to force me to j ust really put all my mental effort i nto something.

Come near 2019, RealityVirtual has several high profile jobs behind it. Some have been publicised, and some are still subject to discretion, but RealityVirtual has found its sweet spot: documenting cultural heritage sites that are under threat so they’re preserved digitally for future generations.

One of its recent projects, Nefertari: Journey to

Eternity, takes viewers up close-and-personal in VR to an ancient civilisation that’s not experienced in many people’s lifetimes: Queen Nefertari’s tomb in Egypt. Documenting Nefertari took Che de Boer 36 hours of travelling to Egypt, and about six to eight hours to photograph 6.4 billion points of colour.

He says the project has achieved something incredibly important: collecting a realistic data-set of a priceless heritage site that has previously been closed to the public due to degradation, so ‘virtual tours’ can fill in when the site gets closed off to the public in future.

Most of the reviews on the STEAM site are in awe of the realism of the experience, with one of the highest rated reviews saying: “It looks absolutely amazing, really good graphics. It almost feels like you are there. I can just imagine a few more years down the line where you won't be able to visually tell the difference. This is another step towards that realization.”

For the month of August and July, Che de Boer says Nefertari was the 11th highest ranking STEAM game based on reviews and user feedback.

“Not the 11th highest VR game or educational app, or subcategory, literally the 11th highest experience full stop on STEAM, and that’s a massive platform,” he says.

“It shows an extremely strong demand for ‘sliceof-life’ experiences, and for the ability to be somewhere and experience something that very few people get the privilege of actually experiencing. That’s extremely important, as it shows we can move away from first person shoot-them-ups and get excited about ‘being there’ sensations. The industry is finally seeing VR as more than an extension to gaming. We have to treat VR like its own medium, it needs its own rules.”

While not speaking of specific projects, Che de Boer says each international gig that RealityVirtual picks up can pay anything between US$50,000 to $250,000, and the jobs are becoming far more frequent now per annum – with more lined up for 2019 than any previous year.

But he says the company still lives from gig-to-gig – or in other words, “extremely Dutch”.

“I just don't know what would happen if we found ourselves in the position of really having $20 million in the coffers. Like, that could just be disastrous or extremely fun,” he laughs.

It also still operates like a start-up: the team, which has now expanded to include creative technologist Dan Monaghan and machine learning expert Miles Thompson, all still work out of Monaghan’s lounge together in Wellington. A stint at working in an actual office proved ill-fated when Che de Boer was caught trying to have a cigarette out of the window.

“I just can't do offices, and especially a shared office,” he says. “I prefer somewhere cosy, creative, where one can chill and have a red wine. Also, as we are developing IP, we realised that this way was safer.”

RealityVirtual is considering floating an ICO or investment to scale up when the timing’s right. Currently, its operations have been running strictly on cash-flow funding research, but many of its core technologies are breakaway products and could be considered separate entities for investment.

Chasing a small pot of gold

A report by the New Zealand VR/AR Association last year forecasted that New Zealand’s cross reality sector (VR and AR) will reach over $320 million in annual revenue and double the number of people employed in the sector currently to 2200 people by 2019.

Yet despite the groundswell of interest for this new medium – and RealityVirtual now getting global recognition for its groundbreaking work (as this issue goes to print, Che de Boer is keynoting at the Cairo International Film Festival) – in 2017, Che de Boer was almost ready to call it quits in the VR industry. He was the primary caregiver of his daughter Anika, supporting them both with a benefit, and had burned through a lot of cash trying to make it work. He’d knocked on the door of government organisations many times, asking for funding and support, to no avail.

In a last-ditch attempt, he decided to get on a plane to Los Angeles and try make a name for himself through rubbing shoulders with Hollywood’s elite. Surprisingly, it wasn’t too intimidating a prospect for him, either. Che de Boer says he feels less awkward mixing with the people in the film industry in Los Angeles than he does with the people living in his hometown in New Zealand.

“It was like returning home to the mothership,” he says.

It proved to be a good call, as this led to his first big break. US video game giant Epic Games – makers of the now wildly successful game Fortnite – gave him a cheque to go make a demo of whatever he wanted to do.

But Che de Boer says his concern is an international gaming juggernaut helped the company get off the ground, rather than a New Zealand organisation.

“We're really stifling a hell of a lot of innovation here, in my opinion,” he says, “If it wasn't for Epic Games who literally just gave us, I think it was NZ$35,000 – no contract, no nothing, just, ‘What's your bank account number?’ then we would’ve gone bankrupt a long time ago. Everyone here’s just chasing this little tiny pot of gold and getting quite bitchy with each other. So it's a lot of underselling each other and as a result, then everyone gets paid way less.”

RealityVirtual was recently hired to scan the New Zealand House of Parliament in addition to indirectly being hired to document the Treaty of Waitangi, but he still feels as though the company still isn’t being prioritised for local gigs.

“We find ourselves locally being put third or fourth in line, while the companies that spend all their money on marketing dollars get all the praise. It seems around here it’s not about the quality of content, but more who you know,” he says.

Now that it has received worldwide acclaim, funding has

proven a little easier to come by – but this irks Che de Boer.

“This is when I go into my plug about how delighted I am, but honestly, fuck off, you took four years. And it's piddly,” Che de Boer laughs. “No, look, I mean I'm happy that New Zealand Film Commission recently got behind us. Good on them.”

“But at the start – Callaghan Innovation, Te Papa, Heritage New Zealand, NZTE, ATEED – the list goes on and on, we tried for years and years locally to get some kind of initial funding of any nature and got no favours whatsoever,” he says.

“It feels like incubators are just setting up incubators. We need to take a leaf out of Epic Games' book and just fund start-ups directly without the bureaucratic bullshit.

“It amazes me, the attitude to local talent. A great example is Air New Zealand hiring CyArk – a San Francisco-based company

– to scan our nation's marae, yet not approaching New Zealand companies, despite the fact we had already done two amazing experiences with indigenous Māori that had world rapport.”

While funding for VR projects in New Zealand is slim pickings, the New Zealand Film Commission Interactive Fund grant gives a grant of up to $25,000 for the concept development of original narrative-based interactive and games content, or up to $50,000 in exceptional circumstances.

I j ust don't know what would happen i f we found ourselves i n the position of really having $ 20 million i n the coffers. Like, that could j ust be disastrous or extremely fun.

Reality check

While cultural preservation is the focus of RealityVirtual, it has a promising future ahead if it gets a spin-out company off the ground that is called deepPBR (Deep Learning Physically Based Rendering).

Che de Boer says through his work, he’s been undertaking a bold new task: extrapolating the fundamental principles of reality. Using machine learning, he’s trained a neural network to learn the concept of light, its material properties and its 3D geometry all from one single image, which could be the holy grail for the visual effects industry.

“In photogrammetry, when you go out and do a scene, you're stuck with the environment that you're given and how the light and the weather was that day,” he says. “That's a bit of a problem, because if you want to relight it from day to night, or if you want to, you know, create variations in weather, you basically can’t. So what we've been doing is we taught a generative adversarial neural network to break down the fundamental principles of reality just by looking at the image in itself, such as how it would look like if there were infinite lighting variables: what is shiny, what is matte.”

He says the machine has acquired enough data through RealityVirtual’s work over the years to now have learnt the fundamental behaviour of light and material properties, and has the potential to do what would take a team of 100 VFX artists and photographers months, with a single individual and a DSLR camera in a matter of hours.

“You've got the, you know, incredible realism of capturing perfection in the real world and then you've got to benefit of being able to re-light it, which has not been the case previously. Not easily, anyway,” he says.

“We've done a thing called hybrid lighting over the years, and it's kind of like a pseudolighting, but it's not based on true physical values. So the new technique, which we are kind of piggybacking on, they call it physical-based rendering, and it’s basically the game engines trying really hard to basically base all materials on reality.”

This means details such as roughness, smoothness and whether an object is shiny or matte can also be remodelled in the environment, as can taking a scene from day to night. Organisations like Weta Digital, Ubisoft, Autodesk, Unicef and Unesco have already shown interest.

But Che de Boer says the only problem with heading in this direction is that it's going to make all of their other work potentially defunct within the next few years.

“So that's quite crazy, you know, we’re literally, potentially killing off our old business model, but hey, welcome to progress.”

And progress is paramount for someone like Che de Boer, whose short attention span and insatiable hunger for information means he

flips careers often. The everdeveloping VR industry and his developments in the deep learning space holds enough excitement to keep him stationary in one role for now.

However, he says he wants to keep innovating until he makes himself obsolete.

“It's an interesting idea that you're only as good as your last project, so every time we do a new project, we are trying to see if there's new ways to do things,” he says.

To continue on his quest to digitally preserve heritage sites for years to come, Che de Boer recently formed a strategic partnership with fellow Kiwi abroad Professor Sarah Kenderdine, head of digital museology at EPFL, a prestigious research university in Switzerland.

Together, they’re looking to virtually re-create the Christchurch Cathedral in its pre-quake state, in addition to several other culturally significant locations.

“We are building the framework and automation tools to essentially democratise the process of backing up the planet,” he says. "That is the holy grail.”

If all goes well, Che de Boer says the entire RealityVirtual enterprise will be uprooted and taken to Europe.

But for now, Che de Boer says the future is bright – and extremely detailed.

We are building the framework and automation tools to essentially democratise the process of backing up the planet. That i s the holy grail.