FANATICAL ABOUT FRACTALS: DISCOVER MANDELBROTS
A string of recent VFX blockbusters have gone ‘fractal-crazy’ – Ian Failes explores the art and science of these mathematical functions
Ian Failes explores the world of complex fractals, from Mandelbrots to Mandelbulbs, speaking to VFX studios about their blockbuster hits
Plenty of filmmakers are always looking for ‘organic’ forms to represent alien worlds or magical moments in their movies. So it’s perhaps unsurprising that they would seek to embrace fractals. After all, fractals tend to look like naturally occurring and infinitely repeatable objects, yet can often be simulated with mathematics.
And so it is that several recent
films, including Doctor Strange, Suicide Squad, Guardians of the
Galaxy Vol. 2, Lucy and Annihilation have adopted fractals – especially three-dimensional ones – to help tell their stories. And they’ve seen use in immersive projects too, where fractal simulations can help realise complex forms for users to explore.
3D World asked some of the visual effects studios tasked with making complex fractals – particularly Mandelbrot and Mandelbulb sets – how they went about tackling these tasks for some impressive VFX.
Fractals of the galaxy
“A fractal is Amazing in the sense that it’s just A tiny piece of simple code” Kevin Smith, visual effects supervisor, Weta Digital
Put simply, fractals are complex, which is exactly why Weta Digital looked to them as inspiration for the Planet Ego sequences in
Guardians of the Galaxy Vol. 2. The studio carried out some early tests embedding implicit functions into its proprietary Manuka pathtrace renderer via plugins, with promising results. It matched the client concept art – which specifically featured 3D Apollonian gasket-like shapes – but it was soon realised that relying on purely
mathematical functions could be limiting.
“A fractal is amazing in the sense that it’s just a tiny piece of simple code, but very small changes to the inputs of fractals tend to result in unpredictable, large-scale changes in the output,” outlines Weta Digital visual effects supervisor Kevin Smith. “They’re very chaotic. This was problematic for us as I couldn’t sit in a review with a client and ask them to give notes on a piece of code, much less one that’s completely unpredictable. We knew that whatever methodology we chose needed to be art directable.”
So Weta Digital considered modelling by hand, but again abandoned this approach due to the infinite detail required (“Also, the models supervisor yelled at me when I brought it up,” says Smith). That left the studio with new requirements: defining an arbitrary shape that was art directable but could still match those gasket shapes, and achieving a digital environment that felt like it had infinite practical detail.
The R&D team devised a method that let artists use curves to define an axis and a profile in Maya, and then code that would use a custom sphere-packing algorithm to boolean out spheres from the
“Visual effects for me has Always been About the combination of Art And science” Kevin Smith, visual effects supervisor, Weta Digital
first shape, to give the appearance of an Apollonian fractal in a usercontrolled volume. “For the detail,” explains Smith, “instead of trying to add infinite fractal minutia, we used an in-house piece of software called Genesis to essentially spraypaint little instances of fractal geometry all over the resulting shapes produced by the first tool. The layout department came up with Genesis brush presets that used combinations of the tiny instances with different scales to essentially make pseudo-fractals. This let us add a lot of detail without incurring an infinite cost. It also helped us age the main sections of the environment, since the Planet Ego in the film was very old.”
Then, a stumbling block. The Maya plugin could not quite achieve all the shapes in the concept art. Weta Digital needed a way to generate some of the more esoteric forms from the art that had originally come from Mandelbulb software, but attempts thus far had required prohibitive amounts of memory without the required resolution. The solution, devised by senior modeller Pascal Raimbault, was to generate a 4K turntable of the relevant areas in the Mandelbulb software – instead of geometry – and then feed those renders to Weta Digital’s photogrammetry software.
“It totally worked,” exclaims Smith. “It produced sharper, cleaner, higher-resolution images than we were getting with voxelisation, and allowed us to build a library of shapes we could use to dress in detail that was not just close to the concept but exactly matched it.”
“Visual effects for me has always been about the combination of art and science,” adds Smith, “and it was great to be able to take a purely mathematical concept like fractals and not only make something new and different, but to use it to help drive the narrative of an awesome movie like Guardians of the Galaxy Vol. 2.”
Fractals as Weapons: doctor strange
An earlier Marvel film, Doctor
Strange, was also one that adopted Mandelbrots and fractal shapes in its art direction. Indeed, the Mandelbrot was one of the key weapons used by the film’s villains to re-shape the world and reconfigure locations. Among other VFX studios, Framestore was called upon to craft such locations with Mandelbrot properties, including for a fight in the Sanctum Foyer where a corridor is made infinitely long.
To do that, artists first modelled each set. They then used Sidefx’s Houdini to ‘slice up’ each set into much smaller objects. “Rigging created a new workflow for the animators to rig every object themselves,” explains Framestore visual effects supervisor Alexis Wajsbrot. “It was important that they were able to animate and have the power to move, pivot or even duplicate the geometry if needed. We also created a toolbox for the animator, to be able to animate a large number of objects – sometimes up to 5,000 – with a mathematical function to create waves and any cool motion we felt was appropriate.”
The next step was to apply the Mandelbrot noise to the CG set. “We tried to implement multiple Mandelbrot functions including the ‘Mandelbox’ and ‘Mandelbulb’,” says Wajsbrot, “but we decided to only use the ‘Mandelsponge’ which was less organic, more geometric.
“After trying different techniques, we decided to apply the Mandelbrot at render time using an Arnold shader we developed to get the maximum amount of detail. To drive the shader, FX TDS used Houdini where they had a volume representation of the Mandelbrot. When we were happy with the
parameters, we baked attributes into the geometry itself so that the shader was reading at render time.”
Wajsbrot says the level of detail that could be achieved using the Mandelbrot function was high, but further art direction was necessary to make it work for the film. “It’s very easy to make something that looks complex and cool, but it’s a lot harder to nail a specific look or behaviour. In the final shots, I would say the animation is [the] main component, and the Mandelbrot is only the last 20 per cent of the image that completes the magic.”
immerse yourself
Away from feature films, Framestore has also been exploring how fractals can form part of immersive entertainment, specifically for virtual and augmented reality experiences. Their CORAL app, which is the brainchild of executive creative director Aron Hjartarson and senior creative developer Johannes Saam, is one such project.
The experience is essentially a journey into fractal shapes (including coral-like structures), controlled by the user. It was made in Unity and supports Vive and Oculus headsets as well as a nonvr desktop mode. “We’ve played with scale by shrinking the users to relatively microscopic sizes and back depending on how close they get to each fractal,” says Saam. “With the right controller the user can morph certain parameters bespoke to each fractal, creating a mesmerising kaleidoscope of transforming shapes.”
A lot of R&D went into making this kind of fractal exploration possible in a VR/AR experience. “The base rendering system uses raymarching as its core concept,” describes creative developer Patrick Beavers. “However, it uses multiple hierarchical passes to be able to converge to a solution faster, adding more detail by increasing the resolution iteratively. Additionally, given the fact that this experience is designed for VR, we developed a technique that reprojects intermediate results from one eye to the other, thus reducing the amount of calculations that are needed to render the shape of each fractal.
“Other techniques we use to accelerate the rendering are fixed foveated rendering, in which the screen has more resolution closer to the centre of the eye, and vignetting, which helps us directly discard pixels that are on the periphery. As for rendering each fractal, we tried to stay away from conventional rendering techniques and used different tricks that we developed to make interesting visuals while keeping good performance.”
The immersive experience is still under development. Framestore has also considered turning CORAL into a music visualiser or potentially using the fractal landscapes as a foundation for a rhythm game. “We would love to allow users to explore with their friends in multiplayer, and it will of course continue to be an outlet for real-time rendering research at Framestore,” notes creative developer Mariano Merchante, “but ultimately CORAL will always first and foremost be a playground of discovery, and we want to make sure we maintain the spirit of that no matter what we do going forward.”
“coral will Always first And foremost be A playground of discovery” Mariano Merchante, creative developer, Framestore