Digital Domain was the lead visual effects house on Speed Racer. In the following interview with Kim Libreri, DD's master of technology you will discover more about his long term collaboration with John Gaeta, the conception and development of the techniques and tools required to create Speed Racer as well as the history of pan and tile panorama techniques since Mission Impossible.

Can you tell us about your collaboration with John Gaeta on the Matrix trilogy and Speed Racer?
John and I started working together at a company called Mass Illusion in the mid 90s. We collaborated first on the film What Dreams May Come, where we investigated laser scanning and photogrammetry techniques that where used as a basis to create the movie’s ‘Painted World.’ After that we started work on The Matrix. I was head of technology for the company and the ‘Bullet Time’ supervisor for the picture.

Ever since then John and I have collaborated on every Wachowski directed movie. He provides the wild creative ideas and my team and I work out the techniques that get the images on the screen. For the first Matrix film the idea was ‘Bullet Time’, next it was building an entire visual effects studio, Escape, to handle the synthetic human effects for the second and third Matrix films. This time, for Speed Racer it was the cars, virtual race tracks, and establishing the techniques for the ‘bubble’ photography.

What was your role on Speed Racer? What were your and Digital Domain’s contributions?
I was one of two VFX supervisors, along with Mohen Leo, from Digital Domain – who also worked on the second two Matrix movies. We were the lead visual effects house on the film, involved from day one in helping to develop the overall look and the processes for bringing it to the screen. Pretty much the same role I’ve had on all of the Wachowski movies working with John.

Our team at Digital Domain developed, among other things: a large proportion of the color pipeline, the panoramic acquisition software; the compositing software, Nuke; the car animation system; the on-set compositing system, Sparky; the on-set motion base car simulation techniques; the look and methodologies for the cars; the initial bubble look and techniques; and of course we created the Thunderhead and Grand Prix races.

How did you work with John to develop the visual style and technique of the cars, races and backgrounds for Speed Racer?
As a team, for two months we had a lot of discussions and looked at tons of examples of tracks and cars to figure out how to make this different from any other racing movie. We wanted to make this movie as refreshing and new as the original Speed Racer was in the 60’s. Inspirations came from everywhere – anime movies, car commercials, and racing video games such as F-Zero, Wipeout and TrackMainia. In an homage to the original anime, we wanted to see what happens when you ignore the laws of physics. Would a modern audience still be able to believe in the world?

It was a very iterative process. John and Dan would review a reference, and talk about what we could do to make the look more interesting. We experimented, did a lot of tests. Initially we tried a bunch of random ideas and would go forward with the ones that worked and abandon those that didn’t. Overall it was a very organic, iterative process. Refine, refine, refine – that’s the process of art. Most movies don’t have the luxury of a pre-production test phase before the script is written, but because Larry and Andy wanted to develop a brand new look for Speed Racer we were fortunate to have this opportunity.

Eventually we came to the decision: do we want to make a movie with real cars that are built to run on real tracks, or bypass that and go for completely stylized tracks and cars akin to a cartoon? We chose the latter.

We did a great deal of visual testing, and in eight weeks put together some really cool ideas into a complete edited sequence that did what we set out to do – create something that looked like no other movie had looked before.

This test sequence provided Larry and Andy with a foundation for the writing of the race sequences in the movie.

Then the challenge became how to achieve the look in a full-length movie, not just a test.

We set out first to develop the techniques to create the cars and synthetic race tracks, and put together a team of Digital Domain animators, artists, engineers and computer scientists lead by Johnny Gibson, Darren Poe and Erik Gamache. For the cars we recruited Richard Morton from our commercials division, to tap into the talented artists who were expert in car commercials and knew how to build, paint, and light amazing-looking cars and environments. The R&D process to assemble all the tools we needed took about a year.

Once we had achieved the technology to create realistic looking cars and tracks and worked out the color palette, we needed to then make it look not photorealistic, but highly stylized. We worked a lot with interesting lighting concepts, and looked to the expertise of digital effects supervisor Darren Poe to re-colorize the lighting and cars to make something no one had seen before. This helped establish the ‘photo anime’ look of the rest of the movie.

Can you talk about how you, as the technology visionary, led the work of the creative team?
Let’s look at that in terms of how we approached creating the cars. First, we looked at real cars and how they interacted with light. We looked at car commercials, and noted how they stylized lighting and motion to make the cars look faster and cooler than they are in real life. You’ll notice that cars are almost exclusively defined by reflections. If those reflections move fast and look interesting, they can help to convey a higher sense of speed. Between Richard’s team and Darren’s, the compositors created a series of colored light tunnels that basically look like a tunnel covered in millions of LEDs switching on and off. These were mapped to virtual cylinders that we bent and placed over the tracks, which allowed the cars to move within a tunnel of lights. These would then reflect on and illuminate the cars, enhancing the speed and adding visual complexity. If a red streak went past the car, you would see in the reflection that same red light.

Essentially, we looked at reality, looked at commercials, picked the best, worked with our software engineers to implement the code we needed, then lit the cars to give the illusion of high speed.

click here to view video

Let’s switch gears and talk about the moving pano techniques in Speed Racer – the ‘bubble photography’ you referenced as being a key component for this film.

When you light a car in the computer, traditionally you’d use standard computer graphics (CG) lights. We wanted to light the cars as a director of photography would, which requires high dynamic range image-based lighting, ray tracing and global illumination. The Digital Domain Mental Ray rendering team created all the relevant technology and because all of our tracks were synthetic, based on CG models that we built, we could put a virtual camera anywhere in the virtual track stadium. To light the car, we’d put a pano (360º) camera at center of where car would be, then drive it down the track for the length of the shot. This created an animated virtual spherical panorama (bubble) of our synthetic race track. That allowed us to place any car into the relevant bubble and it would be lit as if it were in that environment.

click here to view video

When do you think the technology will be available to shoot video bubbles in HD (30 to 60 bubbles per second)?
There are some video devices available now, but they’re not full HD. Absolutely within five years we’ll see full HD cameras being used to shoot HD bubbles. Within two years we’ll likely see some proprietary systems/custom applications.

Can you give us a little history on your use of pan and tile photography in visual effects work?
The first time I ever used pan and tile panorama techniques was in 1995, for Mission Impossible. There was a scene that took place on a train where the production had built a train carriage interior and put bluescreen outside of the windows. We needed to track the movie camera and replace the bluescreen with moving footage of the English countryside. Quicktime VR had just been announced and we thought that we could use similar techniques to produce backgrounds for this sequence. We took three VistaVision cameras and mounted them as close together as possible with overlapping fields of view – we couldn’t get nodal, of course, but the objects were far enough away that it would not matter. I wrote software to stitch together the three camera views into one continuous panorama and implemented a virtual camera pipeline that ran in our compositing system, Cineon.

During the first Matrix movie, John and my team, including George Borshukov and Dan Piponi were creating virtual environments for the ‘Bullet Time’ shots using still photography and image-based rendering. These environments were so successful that we wanted to formalize the process for future movies, especially The Matrix sequels.

We got our chance on Mission: Impossible 2. The film required us to create a virtual version of Sydney, Australia where Tom Cruise leaps out of a building and parachutes down. We shot him on greenscreen and synthesized the environment around him. Obviously we couldn’t have helicopters near the buildings, but we could shoot panoramas with still cameras and project those onto photogrammetrically reconstructed geometry of the buildings. We ended up making a virtual version of downtown Sydney. The challenge was, at the time there were no high resolution digital cameras or high quality panoramic stitching software. We used the Canon still cameras that we’d used in The Matrix bullet time shots and repurposed them to shoot tiled panoramas on Manfrotto geared heads. We shot on film, structured our choice of lenses, developed photogrammetry techniques and built our own stitching programs.

Each single tile was made up of three bracketed exposures, so for every ten tiles (thirty exposures), we had to bring the camera back, reload it, slate, and put it back exactly where it had been before we ran out of film. It was incredibly tedious and very easy to mess up. Imagine having to do all this while being safety cabled to the top of a skyscraper. Fortunately in the end, the shots looked great, and we won a scientific and technical achievement award from the Academy for our virtual filmmaking system.

This was the first widescale use of panoramic photography techniques in a movie. When we saw how well it worked, we took it to the Matrix sequels. We refined it further, used lens calibration, digital cameras, and had custom nodal pan and tilt heads built by our friends at Innovative Arts (the creators of the original bullet time rig). We ended up shooting close half a million individual photos of Sydney (again) for the Matrix sequels. We used the technique for all cityscapes and virtual sets in Reloaded and Revolution, including the burly brawl, the sequence with thousands of Agent Smiths.

Those films were the greatest use of panoramic photography on a movie – until Dennis’ bubble photography for Speed Racer. Right at the start of that movie, John had a flash of inspiration – “instead of building sets for all the environments in Racer, what if we used Matrix-like panoramic backgrounds?” That’s exactly what we did.

Please tell us more about the special hardware and software you created for Speed Racer.
Digital Domain’s Brian Smith developed a custom real-time on-set compositing system specifically for Speed Racer, called Sparky (after the Mach 5 engineer). It consists of a normal Intel PC with an NVidia 8800 high end gaming graphics card, a Blackmagic HD video capture card, and our proprietary software system. It allowed us to take a live HD greenscreen element of an actor, load up the panoramas shot by Dennis and painted by Lubo, and do real-time compositing of the actor into that environment. It handles multiple bubbles and digital stills up to 8K resolution, lets us change the focal length, load multiple greenscreen elements and place them in the film frame in arbitrary positions, all in real-time. It gave the directors and DP real-time lighting reference and the ability to create a collage that they could use to compose the virtual items and environments that would be behind the actors while they were shooting.

What future projects do you have on the horizon?
Right now Digital Domain is working on Mummy: Tomb of the Dragon Emperor, which opens in August, The Curious Case of Benjamin Button, which comes out at the end of the year, and a new project using game engine technology to make an animated movie. And of course there’s more in house that we can’t talk about yet.

Can you give us some background about yourself and your passions?
That’s simple. I’m just a geek who likes computers, cars, and visual effects.

Links:
Digital Domain's website
VRMag's Speed Racer coverage
Speed Racer official site

Related articles in this issue:
WHEN CINEMA MEETS VR - JOHN GAETA TALKS ABOUT SPEED RACER
SPEED RACER'S WORLD UNIT LEADER DENNIS MARTIN
LUBO HRISTOV VFX ENVIRONMENTS ART DIRECTOR ABOUT SPEED RACER
DIGITAL EFFECTS SUPERVISOR JAKE MORRISON ABOUT SPEED RACER