Desktop apps 360 VR videos
Discover why Adam Oxford thinks 360 video is finally starting to find its feet.
It’s a nascent art form, but Adam Oxford thinks 360 video is starting to find its feet, so rush out and capture your own.
Looking back it seems strange, but some of the first “killer apps” for smartphones were augmented and virtual reality (AR and VR) in nature. As early as January 2009, just 18 months after the original iPhone appeared, there were already several star chart maps that used the built-in GPS and accelerometer to track the night sky using the phone screen.
It seems strange because ever since then, AR and VR have been the Next Big Thing. Always exciting, always promising… but never quite compelling enough to hit the big time. PokemonGo! came and went, leaving a million tedious “viral” marketing imitators in its wake. Together, AR and VR have had so many false starts that not even Mark Zuckerberg’s preposterous entrance to Mobile World Congress last year, in which 1,000 journalists were convinced to don VR specs so that they could picture themselves sitting in the seat they were already sitting in could convince the world it was ready for VR again.
So AR and VR still feel like novelty subjects, despite predating the smartphone era. Which is a shame, because they’re a lot of fun to play with. Especially in their most basic form, the 360-degree video.
Generally speaking, we can divide the world of immersive video into three distinct areas. Augmented Reality (AR) is when virtual information is layered on top of the real world. Like PokemonGo! this is typically done in real-time using the camera on your phone and its screen. Apple and Google are putting huge efforts into this area, and Apple’s ARKit – part of the latest iOS toolkit – is described by developers as something of a breakthrough in ease of use.
Virtual reality, meanwhile, is a completely computer generated world projected onto the screen of your phone or into a headset. Not only is it stereoscopic, so that it appears 3D, it’s also fully interactive. Think games like Elite:Dangerous played with an Occulus Rift headset.
Immersive video
What we’re interested in, however, is 360-degree video. Like VR, this is completely immersive, but it’s not interactive. You can watch it on a computer screen, with a viewer that enables you to move the camera angle around, or you can use a headset or Google Cardboard-style viewer for full immersion. It might also be fully 3D if it’s recorded with two cameras for stereoscopic effects, but overall it’s the least explored of the three categories.
There have been some interesting experiments with 360-degree video: from action cams strapped to the heads of mountain bikers or downhill skiers, to a series of films shot for behind-the-scenes glimpses into BBC documentary making. For the most part, the aim of these videos is to give the viewer an immediate feeling of doing something else – they’re rough, unedited and don’t even attempt to portray something as nuanced as a story.
There are rare exceptions. The creators of Mr.Robot edited together a sequence of scenes from the first series as
360-video, for example. Unlike most 360 videos, the camera doesn’t always put the audience in a first-person view – but rather enables them to explore a scene while it’s playing out. It’s an intriguing – if uncommon – experience.
There’s clearly potential for 360-degree video – it’s a logical progression for filmmakers who want to control what their audience sees – but it’s still more-or-less unexplored territory and ripe for an auteur to show what can be done.
Capturing 360-video
There are three different ways to capture 360-degree scenes. Early pioneers built rigs upon which they mounted multiple cameras and stitched the results together in desktop editing suites, but things have progressed since then.
The easiest way to record 360-degree video is with a dedicated camera such as the Ricoh Theta S or Samsung Gear 360. These have a wide-angle lens on the front and back that gives an almost-but-not-quite 180-degree from either side of the camera, and the two video feeds are stitched together into the spherical view on the camera itself. Professional 360-degree cameras often have a third lens pointing upwards to eliminate blind spots.
Dedicated devices are still quite pricey, so cameras that work with your phone are a good second best. We used an Insta360 Air, which plugs into the USB port on a smartphone. It’s a bit more cumbersome to use than a dedicated camera, but almost twothirds less expensive. The end results are excellent and the phone app stitches the video together, enables you to watch other 360-degree videos and exports the final movie clip in a format we can use on the desktop.
Finally, you can shoot 360-degree still images using your phone or camera by taking lots of still pics from all around you and stitching them together using imaging software. It’s a great (and free – see boxout) way to get started, but it’s not video and so is no good for our purposes. There are barely any applications for Linux that can handle 360-video properly. Blender, for example, can create 3D worlds and export fly-throughs as 360-video, but it’s not so handy for adding elements to films you’ve already shot. Fortunately, 3D video can be edited and cut together in almost any regular non-linear video editing suite. However, you’ll need to install separate software to preview your edits, and there’s a few choices you can opt for. The open source Valiant360 project ( https://github. com/flimshaw/Valiant360) can turn your browser into a responsive 3D video viewer if you’re keen on customising it to use as a local player. For something easier and more off-theshelf, the folks at GoPro have a .deb package of GoProVRPlayer ( www.kolor.com/gopro-vr-player), which works flawlessly on most Debian derivatives. For an open source player, however, Version 3 of VLC player does support 360-degree video, although it doesn’t have the same viewing options as GoProVRPlayer. It’s not included in distribution repos yet, though. To get the latest build ( VLC 3.0.0-git Vetinari) in Ubuntu, for example, you’ll need to add the “nightly” PPA: sudo add-aptrepository ppa:videolan/master-daily && apt install vlc . Overall, GoProVRPlayer gave us the best playback quality and in-video controls. Once you’ve cut your video, if you want to share it with the world you’ll want to upload it to Google, Facebook or Vimeo et al. All support sharing and viewing 360-degree videos with their own in-browser players.
Astral projections
To start editing your video you’ll need to get it off of your phone and onto the desktop. This will differ from camera to camera, but the principle is usually the same. In the Insta360 app, for example, you’ll need to export the video to your Videos gallery using the in-app conversion tool under “Share”. You may find that there are a few
options for the exported file – particularly relating to the viewpoint that might be “Tiny Earth”, “Spherical”, “Fish-eye” or “Equirectanglar”.
If these are options, it’s the last that you want to select. This is the format that’s most common and easiest to work with, and exports the 3D video as a regular 2D MP4 file. It’ll be in a 2:1 super-widescreen ratio and you want to select the highest resolution you can (4K is usually an option).
On the desktop
Equirectangular projections flatten spheres into a rectangle, keeping relative dimensions accurate. So when viewed, the world they portray looks like two fish-eye images stitched together. When curled into a ball everything regains its proper proportions. The best examples of equirectangular versus other projections involves maps of the Earth.
Think of the way the globe is divided up into lines of longitude and latitude. As you get closer to the poles, the physical size of each box bounded by a line of longitude and latitude gets smaller, because the curve of the Earth makes longitudinal lines converge and means there’s less distance for each line of latitude to cover.
This presents a challenge to cartographers. How do you represent a curving line as a rectangle on a flat map? Some maps – such as the famous Mercator projection – do this by drawing each “box” from the globe as an equally sized box on a map. In doing so, they exaggerate the size of landmasses, so that tiny Greenland can appear geographically similar to Africa, which is big enough to consume China, the USA and all of Europe.
Equirectangular maps, meanwhile, try to compensate by shrinking the size of the longitude/latitude boxes appropriately. They have a more compressed look at the top and bottom and make landmasses near the poles look smaller than they might on a globe. If you’ve exported your video correctly, it should look a bit like that.
The edit suite
With your 360-degree videos in a rectangular aspect ration and common digital format, you can start actually cutting them together in any video editing suite. It’s really that straightforward – the only thing you’ll need to make sure of before you start is that the camera’s starting orientation is the same in every clip you edit. Viewers can look up and down once the video begins, but switching the ceiling and the floor halfway through will be very disorienting.
Once you’ve put together the movie you want, though, things get a little bit more tricky. It’s best to render the final cut as an MKV or MP4 file, because they’re widely supported in terms of software and cloud sharing services. Indeed, some online services such as YouTube will only accept 360-degree videos with the MP4 file extension.
Because the 360-degree format is new, however, and not really supported by any desktop editors in Linux, your final file will be missing some metadata that identifies it as a 360-degree film.
The GoProVRPlayer doesn’t care – it’ll twist any file that you give it into a 3D viewpoint – but YouTube and other online services do. If you don’t add the right metadata (similar to EXIF information in a still image), they’ll only play the file back in its flat equirectangular format if you don’t identify it correctly.
Thankfully, there’s an easy way to do this: Google’s engineers have created a tool called the SpatialMedia MetadataInjector. Grab the file from its Github archive ( https://github.com/google/spatial-media/tree/master/ spatialmedia), and extract it to a suitable location in your Home folder. Say ~/Injector.
You’ll also need to install Python from your distribution repos if you haven’t done so already, with the command sudo apt install python .
Once you’ve extracted the SpatialMediaMetadata Injector, using it is simple. Open a terminal and type the following command: python ~/Injector/spatialmedia -i ~/ pathto/input.mp4 ~/pathto/output.mp4 where input.mp4 is your edited video and output.mp4 is the corrected version with the new metadata added.
And that’s more or less it. Making a 360-degree video is much the same as making a regular 2D video, with just a few things to watch for before and after the editing process. The next step is to think about how to capture and map the sound with multiple mics in 360-degrees as well. But that would be a whole other walkthrough…