Mixed results
Does Quest 3 mark a tipping point for mixed-reality videogames?
John Carmack has never been one to bite his tongue. Since his resignation from Meta last year, however – accompanied by an internal announcement, leaked to the press, that he “wearied of the fight” – he has had no reason to hold back with his opinions on its strategies. In late September, as his previous employer prepared to ship the first Quest 3 units, with MR (mixed reality) the headline feature, Carmack took to Twitter. “I remain unconvinced that mixed reality applications are any kind of an engine for increasing headset sales,” he wrote. “High-quality passthrough is great, but I just don’t see applications built around integrating rendering with your real-world environment as any kind of a killer app. I consider it [an] interesting and challenging technology looking for a justification.” The Id co-founder capped off the post in the most John Carmack way possible, offering a cash bet to anyone who could prove him wrong.
Carmack’s words go through our head as we first place the Quest 3 onto it. The hardware, though, offers a pretty strong counterargument. The passthrough feed from its dual RGB cameras is sufficiently high-fidelity and responsive to make a plausible stand-in for actual reality. Handy for a quick glance at our phone or PC, yes, but it also makes it possible to move to another room – or even another floor – without removing the headset, something we’d never have dared attempt with its predecessor.
The hand tracking, meanwhile, feels something like magic. Extending a digit out in front of your face, it’s easy to scroll through app listings as you would with a touchscreen – except that here your finger is resting on thin air, meaning that you have an extra axis of interaction at your disposal. Without thinking about it, we instinctively grab at a menu to bring it closer, and it’s quietly thrilling to discover that it simply works. Finer control can be hit or miss, especially when dealing with something that’s not immediately in front of you, requiring some shaky pointing off into the distance. But at its best, this is exactly the kind of naturalistic interaction that has always been core to XR’s promise. At this point, though, we’ve barely left the menus. So, to Carmack’s point: what of the experiences made possible by this technology?
Quest 3 arrives without a launch lineup in the traditional sense of the phrase. With the exception of Meta’s own First Encounters (essentially an MR tutorial-cum-tech demo), the small cluster of games that accompany its arrival are all Quest 2 compatible, as will be the case until at least 2024. Fortunately, a number of games from the existing library have received Quest 3-friendly updates, working together to provide a handy overview of what MR gaming looks like right now – although technically speaking calling it MR gaming may be misleading.
MR works by taking a live video feed and overlaying it with computergenerated imagery – much like AR, then, but with the vital difference that the feed isn’t treated as a flat background but rather a 3D space into which these virtual elements can be integrated. A wall, for example, might host a portal to another world, while that coffee table you’re always bumping your shins against in VR sessions is not only visible here but a surface upon which a virtual boardgame can be played. By this yardstick, though, many of Quest’s first wave of MR games are, in fact, headset-based AR – a point that’s first brought to our attention by the developer behind one of them.
Clockstone’s Lego Bricktales arrives this December, a couple of months after Quest 3’s launch, but it was the very first game shown during the hardware’s unveiling, at Meta Connect back in September. “Quest 3 understands your space, so you can play with the world around you,” Mark Zuckerberg promised on stage. “You can solve Lego puzzles, or build your own creations on any flat surface in your room.” For now, though, that isn’t quite the case, as Clockstone co-founder and Lego Bricktales project manager Matthias Hilke explains: “If MR is to be understood as the interaction of a game with the real world, then actually, currently, we were not really able to implement the one feature that we had in mind, to snap the diorama onto the table.”
This has long been a stalwart of Meta’s marketing of this concept: the understandable, trailer-friendly fantasy of a boardgame or Lego set that can be plonked down on the table without requiring any of the tidying up afterwards, or shelf space to store it. “Meta asked for this early on,” Hilke adds, “but we had to fight more basic issues.” When those couldn’t be ironed out before the certification deadline, Clockstone’s hopes shifted to adding it to the game with a post-release update.
The feed isn’t a flat background but rather a 3D space into which these virtual elements can be integrated
Not that this dampens our experience too much. The diorama approach has always suited VR beautifully – think of peering down into the miniature fantasy world of Moss, or turning your head to follow Astro Bot through Rescue Mission’s
platforming sequences. But stripping away everything but the object in front of you really emphasises its scale, which fits the source material beautifully. We lean in close to study the intricate Lego brickwork that makes up its levels and coo appreciatively, as you might at a real display of master-builder prowess.
Bricktales represents one of two major schools of MR design here, also on show in the hectic multiplayer arena battles of Bam and in Demeo, a well-established VR take on D&D-style tabletop games that sticks to a stricter definition of MR, its board adhering to any flat surface in front of you. A recent hand-tracking update, meanwhile, lets you manually reposition miniatures and roll dice, which rattle convincingly along the table – and often off its edge, onto the floor below. There are clues, though, as to why Bricktales
might not have launched with these features in place. The board’s placement can be distractingly skittish, something that isn’t helped by the gesture controls shifting its location whenever a hand is clenched into a fist. Using your fingertips to grasp such small objects, meanwhile, exposes every tiny inaccuracy of the tracking, leading to frustrating moments when our hand seems to pass through the model we’re trying to pick up, or places it down a square short of the enemy we were intending to engage.
There’s much more room for error in the other school of MR design. As with the dioramas, this builds on an established VR template, in the vein of Beat Saber and umpteen virtual reality shooters, where you stand and move around a limited space while poking at your surroundings in some way. This can be with your fists, as in the Punch-Out-like boxing game Knockout League, or with a sword, in Broken Edge, which opens up a rift on a nearby wall to its pastel Moebius-art reality, from which your rival duellist emerges. Or, indeed, with
a pair of maracas in the case of Samba De Amigo: Party Central.
While it doesn’t do much to reinvent Sega’s rhythm-game series, Party Central does incorporate its MR elements with a certain flair. Video filters alter the appearance of your surroundings, inverting their colours or redrawing the outlines of nearby furniture in thick neon. At one point, the ceiling of our living room is torn away to reveal a giant Mecha-Amigo, its head poking into our space. Then its colossal legs begin kicking holes in the walls and floor. It’s a joyful way of reinventing familiar surroundings, albeit fairly shortlived: while all songs can be played in MR, the real showcase is the Multiverse mode, consisting of three stages and lasting under 15 minutes.
This is indicative of most games’ MR offerings for Quest 3, which either simply replace backgrounds with camera passthrough or else offer a limited sideshow to demonstrate what could, hypothetically, be done with this technology. But there are exceptions, most notably Espire 2,a Metal Gearstyled stealth action game that has bolted on a suite of bespoke MR missions that reimagine your living space as infiltration zones. One mission adds a blast shutter to a nearby wall that, when opened, reveals a snowy base outside with guards to snipe; another has us crawling along the living-room floor in order to duck a Mission Impossible-style laser grid.
This mode, more than anything else we play, highlights one strength MR has over VR. One of the latter’s biggest obstacles is having to work around the everyday objects in your play space. Espire 2, though, makes a boon of them, whether by scattering documents over a table or having you duck behind the sofa for cover during a gunfight, having fed the furniture into its procedural level generation at the outset of every mission. If that sounds like a development headache, then Michael Wentworth-Bell, founder of Espire 2 developer Digital Lode, confirms it. “This was the most technically difficult project we have worked on, primarily because every single person’s room is unique,” he tells us. “A player may have a ton of free floor space, but their walls are blocked by shelving units all around. Or they may have bare walls, but much of the floor space is taken up by a king-size bed.”
“This was the most technically difficult project we have worked on because every person’s room is unique”
The technical ambition pays dividends here, but – as with Demeo – also exposes some of the hardware’s limitations. While Meta’s Scene APIs allow developers to treat real 3D space as level geometry, this isn’t entirely an automated process from your perspective as player. You must set up your room manually, drawing out bounding boxes for every piece of furniture. Push aside a sofa for a VR session and you will need to repeat this process; play in a different room and you’ll need to start over entirely, which overwrites the original, since only one room can be stored at a time. Tapping into one of Espire 2’s headline features – multi-room MR to allow missions that let you traverse a base the size of your entire home – requires unlocking the device’s developer mode and taking advantage of various workarounds.
As a launchpad for Meta’s MR vision, then, Quest 3 falls a little short. Of course, that’s the nature of any nascent technology: it’s normally the second or third iteration that nails it, provided the dream makes it that far. This will depend on consumer adoption along with Meta’s will to keep pushing MR as a selling point, but also on developers’ continued interest in pursuing it. How do they feel about their position right now?
For Clockstone, which dabbled in VR back in the HTC Vive days before deciding “the market was not ready yet, and we would not follow that road any more,” the jury is still out. “There are two parties in the team,” Hilke says. “Some really do not want to do another VR game; others think it would be cool to continue here.” Digital Lode, however, has long been entrenched in the XR space – and as far as Wentworth-Bell is concerned, MR is just another “Wild West frontier” for the studio to explore: “Working on Espire’s MR mode, the process felt invigorating, like VR did ten years ago. Everything was fresh, no rules were established, and nobody knew exactly what’s fun yet – until you try it.”
Digital Lode is keen to apply everything it has learned during this process, Wentworth-Bell tells us: “We actually prototyped seven new MR experiences throughout 2023. There are three in particular at various stages of preproduction and prototype phases. It’s been a very tough year for the games industry – with luck, we will develop one of these MR experiences into a full product.” It’s likely to remain a side feature, he acknowledges, since relying exclusively on MR risks condemning developers to a niche of a niche. But Wentworth-Bell clearly believes that there’s room left to be explored here. Perhaps he could be the one to take Carmack up on that wager.