One of the world's leading expert on using spherical photography in virtual locations for movies, Dennis Martin - prior to Speed Racer - has worked on the Matrix trilogy.
In order to work on the Speed Racer movie, Dennis has left his job at Industrial Light and Magic; packed his luggage and said goodbye to his girlfriend and his dog.

Six months away, with a mission to accomplish: to take more than 10'000 VR panoramas in several countries, from France to Morocco; from Italy to Austria.
Back on his side of the ocean, he spent the same amount of time in Los Angeles, to post produce the bubbles which were the locations for Speed Racer, what he considers as being his greatest project so far.

Discover - in this interview - how Dennis and his team worked, all the processes he was responsible for, and his passions, dreams and pastimes...

Dennis Martin, Morocco

How and when did you become involved in Photography?
I first became interested in photography when I studied architecture at University of California, Berkeley. Most of my design projects started with a site analysis where I would take photos, measurements, and make drawings of the proposed site. My first panoramas were actually a bunch of developed film photos shot handheld, and pasted over each other to show the site on the final presentation boards for my design studios. As I became increasingly focused on creating accurate representations of the building sites, It was a natural progression for me to get into 3D computer graphics to create photo-realistic computer models.

While I was in college, Paul Debevec was a computer science professor at UC Berkeley, and he and his students were working on projecting high dynamic range images on 3d models to get very realistic 3d animations. I saw some of his presentations on HDR images, and in particular his campanile movie where aerial kite photography was projected on a computer model of the UC Berkeley campus. I was very inspired by these projects and began to use photography with computer models to create my own 3d computer animations. So it was really through my bigger interest in creating virtual environments that I became interested in photography.

Can you tell us more about your professional/private background ?
I currently work in visual effects, and have been for the last 10 years. I’m presently with Digital Domain, and have worked at Industrial Light and Magic, and Esc Entertainment. My degree is in architecture so design is the foundation of most of my work. My expertise is very technical however: focused on techniques to recreate existing scenes with 3d laser scanning or LIDAR, photogrammetry, 3d projections and rendering. Working in visual effects is the perfect field for me because it allows me to use my technical and design background.

How did you get into the visual effects industry?
After I graduated from UC Berkeley I met Ben Kacyra who is now known as the father of LIDAR scanning. At that time, Ben's company called Cyra Technologies was about 12 people and they had a few working prototypes of the Cyrax laser scanner, and were just beginning production of the first commercial long range laser scanner - the Cyrax 2400. The Cyrax scanner fired a pulsed laser several thousand times a second to create highly accurate computer models of sites and structures. It had a range of over 100 meters and accuracy of 6mm. My job was to help guide the hardware and software development, and to prove applications for laser scanning. This was really an amazing job for a couple reasons: one because Ben Kacyra was one of the most inspiring and visionary people I had the opportunity to work with, and two because I was able to travel internationally to work on many diverse projects in different fields including engineering, archeology, architecture and visual effects. LIDAR was a direct replacement for stereo photogrammetry so I also learned about photogrammetry and alternative ways to document and create 3D reconstructions of real world scenes. Cyra was eventually acquired by Leica Geosystems and now LIDAR is the accepted standard for high accuracy reality capture in many fields. Ben Kacyra and his wife Barbara now run a non profit organization called Cyark that documents world heritage sites around the world using LIDAR scanning and pano photography: they have a very cool website that I’m sure your readers would find interesting: cyark.org.

A previous VRMag article about Cyark projects can be found here.

click here to view Lidar reality capture video

I saw LIDAR as a great tool for capturing geometry of sets for visual effects companies. I convinced the visual effects house Tippet studios to try LIDAR scanning on starship troopers locations as one of the first pilot projects for LIDAR in visual effects. Gradually LIDAR became a more accepted technique, and I got a call from Kim Libreri who was working with John Gaeta and Dan Glass on the Matrix sequels, to LIDAR scan the Burly Brawl set: the courtyard where Neo fights 100 Agent Smiths in the Matrix Reloaded. After scanning the set, Kim convinced me to join the virtual background team at ESC Entertainment.

How did you become interested in VR photography? Was there anyone in particular that influenced you to choose this particular path in photography?
Working with Kim Libreri on the Matrix sequels really began my serious work with panorama photography and photo real 3d reconstructions, and Kim has remained one of the mentors in the industry ever since. Kim is most well known for being the visual effects supervisor on the bullet time sequences in the original Matrix. Kim and his team had spent months in Sydney - Australia - shooting panoramas from tripods, helicopters, cranes suspended off of buildings, and cranes going down the middle of streets. They literally shot tens of thousands of bracketed images to recreate all of the scenes in the Matrix. Only at that time there were no PTGui or autopano stitch programs, so we had to build all our tools from scratch to combine the bracketed images into high dynamic range (HDR) images, and to stitch and render HDR panorama images for projecting on 3d geometry. We also had to build a photogrammetry pipeline to recreate the city accurately in 3D from the photos. Our R&D team, led by Dan Piponi made most of the tools we needed to create these virtual backgrounds.

To complicate matters most of these panoramas were shot on film; meaning each roll of developed film had slightly different color variations; thus we had to create a color calibration pipeline where each roll of film had its own color calibration. Also remember film has no exif data, so we had to rely on notes for all the camera parameters. We also shot lens grids for every lens that were used to undistort the images. So in the end we had a giant database to store all the data needed to process all the raw images. One thing’s for sure stitching panoramas is a lot easier now with digital cameras and the latest stitching tools.

Our first panorama stitching tool had no solving capabilities. All we could do was move tiles around and make them semi transparent to make sure they lined up. Once we lined them up we could then render a high dynamic range stitched image which is a feature only recently added to our current commercial stitching packages. It was a painstaking process to lineup an entire sphere of images manually, and the first couple times you tried it seemed impossible. Naturally, the stitching quickly became the hazing task for our most eager entry level interns: “ here’s 50 images to stitch try get them to me by next week”.

At that time it seemed there were a couple interesting papers every year about automatic stitching but none that actually looked like they would work in a production pipeline. Then came David Lowe’s paper Distinctive Image Features from Scale-Invariant Keypoints from the Univerisity of Vancouver. I quickly got a copy of Alexander Jenny’s autopano implementation of the sift algorithm and was amazed at how well it worked. It was quite exciting as automatic stitching was obviously on its way. In my opinion the sift detector is one of the biggest advances in computer vision of the last decade.

You are one of the leading experts on using spherical photography in virtual locations for movies, can you tell us more about this particular branch of VR photography ?
When shooting panoramas for film there are really two things we are usually after. One is reflection environments for cg objects and characters: this is really a replacement of the good old chrome ball shooting we used to do in visual effects. Even a quick sphere shot on a fish eye lens gives you much more detail than one shot of a chrome sphere providing more realistic reflections. These spheres are shot bracketed and high dynamic range, but can be low – medium rez and don’t have to be stitched perfectly. Often we will try to follow the director as he shoots, and shoot a quick reflection sphere at every location that will need cg elements from the shot cameras exact location.

The second and much more difficult type of panorama shooting is used for 3d environment reconstructions. In these cases we rely heavily on previs and storyboards to know what we need to shoot. Often for 3d reconstructions we are shooting partial spheres just to cover the areas we need for a sequence, making sure we have the resolution needed to cover specific shots. Also complicating matters, we usually need to shoot different environments that will be pieced together to make a custom environment. This process involves matching daylighting scenarios and a lot of matte paint and compositing touch up at the end. Many times I will do 3d daylight simulations to figure out exactly what time I need to shoot a scene.

Can u tell us about the most interesting projects you realized before Speedracer?
Some of the interesting projects I worked on that involved 3d environments before Speed Racer were the Matrix sequels, Catwoman, the Island, Mission Impossible III, and Transformers. Mission Impossible III is worth mentioning as we used image based modeling and rendering for all the backgrounds in the Shanghai sequence. This is the sequence where Tom Cruise swings from one skyscraper to another and parachutes down to the ground. We spent 2 weeks in Shanghai shooting from cranes off the tops of buildings. We lucked out on this shoot ,as a typhoon came through when we arrived, which cleaned the smog out the air and we got amazing clear photos of the Shanghai skyline.

For Speedracer you were leading the "World team" , and also the Nr. 1 VR photographer. Can you tell us about this incredible project and your involvement?
So first let me say that a project like Speed Racer doesn’t come along very often. I first heard about this project when I met John Gaeta at a café in Marin, California. He had been working with Dan Glass and the Wachowski brothers on how to make Speed Racer a movie that looked like a cartoon that had come to life. The key concept was to make the entire movie using spherical photography as the backgrounds. John and Dan had already spent a lot of time doing tests and conceptual proofs for this idea. The concept was simple: use the techniques of traditional anime artists, only replace the drawings with photos and realistic matte paintings. Driving sequences could be done with multi planar animated layers, and conveyor belts of repeating elements. The idea was not to make photo real backgrounds, but to make an anime-like world that would become the unique visual language for speed racer’s world. This concept combined with inspiration from the original cartoon and the imagination of the Wachowski brothers was a project that I could not turn down.

For me personally, it meant quitting my job at Industrial Light and Magic that was 10 minutes from my home in San Francisco. Then spending 6 months based in Berlin, Germany while shooting, and then another 6 months working on post production in Los Angeles... Leaving my fiancée Janice Kefer and my dog Gehry at home in San Francisco was probably the hardest part. Luckily Janice encouraged me to take the job and supported me 100% throughout the project.

As excited as I was about the project, the breadth of work was frightening. In my previous projects, I would normally have been responsible for a couple visual effects sequences which would involve a few weeks of on set shooting then a small team doing post production for several months. Also for this project John and Dan wanted to bring a lot of the tasks that would normally be post production into production: we would have to stitch and render all of our panoramas in tandem with shooting. Using a custom real time compositing system called Sparky created by Brian Smith at Digital Domain, the directors would be able to see our backgrounds on monitors as they shot the actors on green screen. So the world team was always under the gun to have locations shot and stitched into 360 panoramas (we called them bubbles) before the directors shot those sequences on the green screen stages.

The biggest challenge as always was the logistics of traveling to many different countries to acquire the images. The production hired Neil Ravan and his assistant Lulu Morgan as the Plate Unit supervisor, and Neil essentially was the production manager for the World Team. It was a great partnership as I could focus on the training of the crew and the technical aspects of the shoots while Neil oversaw our operation and made sure everything came together on schedule.

The shoots involved locations in Austria, Greece, Morocco, Italy, France, Switzerland, and Germany. We had just under 6 months to get a team and pipeline put together and shoot all the locations.

Dennis Martin, Grossglockner, Austria

click here to view Casa Cristo - Mountain Rally, Grossglockner, Austria
© 2008 Warner Bros. Ent. All rights reserved

Casa Cristo - switchback - passo dello Stelvio, Italy
© 2008 Warner Bros. Ent. All rights reserved

click here to view Casa Cristo - Switchback, Paso dello Stelvio, Stelvio, Italy
© 2008 Warner Bros. Ent. All rights reserved

click here to view Casa Cristo - Rendevouz, Lienzer Dolomiten, Austria
© 2008 Warner Bros. Ent. All rights reserved

click here to view Cortega Bracciano Castle, Bracciano, Italy
© 2008 Warner Bros. Ent. All rights reserved

Dennis Martin, Death Valley, California

click here to view Casa Cristo - Desert Rally, Death Valley, California
© 2008 Warner Bros. Ent. All rights reserved

click here to view Casa Cristo - End of day 1, Siena Square, Italy
© 2008 Warner Bros. Ent. All rights reserved

Casa Cristo - cruncher block's view - Santorini, Greece
© 2008 Warner Bros. Ent. All rights reserved

Cortega - aquaducts - via dell' acquedotto Claudio, Rome, Italy
© 2008 Warner Bros. Ent. All rights reserved

Casa Cristo - Victory lane - Brandenburg gate, Berlin
© 2008 Warner Bros. Ent. All rights reserved

Can you tell us about the members of your team?
In Berlin during production we hired a 6 person team: Stefan Erhard and Moritz von Lupke as lead photographers, Katja Fischer as my assistant / stitching and mapping expert, David Miller as our on set photographer, stitcher and image librarian. And Hayden Topperwien as our lead image stitching machine. In between shooting, all of the photographers would stitch images, so everyone was cross trained and could assist on shoots or in the office stitching bubbles.

Stefan Erhard, Stelvio, Italy

Moritz von Lupke, Stelvio, Italy

Katja Fisher, Sanssouci palace, Potsdam, Germany

In post production, back in Los Angeles, we assembled a new team for some local pickup shoots and for final HDR image stitching. Our LA team consisted of David Miller as photographer and image stitcher, Jill George as image stitcher / matte painter / photography assistant and Karla Ventocilla Curby as image stitcher and compositor. Patrick Kearney, who already had multiple visual effects production responsibilities, was our World Team Production Manager and made sure our whole operation went smoothly and our deliveries were on schedule.

The World Team also worked directly with the matte painters who would ultimately take the bubbles and make the final look for the shots. Leading the matte painters was Lubo Hristov and his company Christov FX. Lubo was the visual effects Art Director for the World Team and worked directly with the Wachowski Brothers, John Gaeta, and Dan Glass to establish the final look for the bubbles.

Can you tell us about some the specific challenges encountered while shooting on location?
Well there were many challenges that came up like the scary old hydraulic-fluid-leaking cherry picker I had used in Morocco; Neil having to shoo off wild dogs in Mac Ness; Moritz breaking his wrist on a Segway in Paris; bad weather in the mountains of Austria; locations contracts falling through from time to time...but all in all it was a pretty smooth shoot, thanks to Neil and the locations team.

One good example of our many challenges was the Ben Youssef Madrassa in Marrakesh, Morocco.
First of all when we arrived the owner of the Madrassa changed her mind about letting us shoot there. She pretended like she had never heard of us even though we already had a verbal agreement with her. Neil would call her and she would say “I don’t know who you are” then hang up. So we looked for alternative locations while Neil kept trying to convince the owner to change her mind. Neil’s tenacity paid off and in our last two days the owner agreed to let us shoot. While we were shooting, Neil and I would usually break away from the team in the evening and go scout the next day’s shoot to make sure everything was ready. So we went to scout the Ben Youseff Madrassa and found out there were no lights at all in the courtyard, and this needed to be a nighttime shoot. We had no lighting equipment with us as our goal was always to shoot locations as they existed.

Casa Cristo - square - La Madrassa Ben Youssef, Marrakech

Casa Cristo - Victory Lane - La Madrassa Ben Youssef, Marrakech

So Neil hired a local electrician named Jaguar to help us light the Madrassa. We basically had him get about 100 garden lighting fixtures with 150 watt flood bulbs and had him wire up the square for the next nights shoot. He finished just before shoot time at sunset and we discovered Jaguar had used fixtures rated at 50 watts for all the 150 watt bulbs. They seemed to work fine but we were worried about fire danger so we had to hire some locals to basically sit and watch each fixture just to make sure they didn’t burst into flames as we shot. Luckily they all worked fine and we got the shots as planned.

Which were the technical specs for the panos required for Speed Racer?
The resolutions required varied per shot, but most of our interior spheres were shot with 24mm or 50mm lens and were rendered at around 20,000 pixels wide. For sequences like the mountain rally, where we were cutting and pasting a lot of elements together, there were a lot of high rez panorama elements shot on long lens 200 - 400 mm that were not full spheres. We would basically look at the previs animations of the sequences and break the shots down into layers and try to shoot specific elements for each layer. The key was to make sure there is enough resolution in the pano to zoom into the photo as much as needed in the shots. In many cases we would also try to get a more generic library of elements that could be used in different situations.

What VR equipment did you use in the creation of Speed Racer'spanoramas?
Of course getting the equipment together is always one of the most fun parts of a shoot. There’s no better feeling than having the perfect kit while you shoot. So for speed racer we shot mostly with the Canon 5D, and Canon’s top quality L lenses. We also used primes as much as possible. For manual pano heads we had a manfrotto, 2 360Precision Adjustes, and my personal rig which has a novoflex panorama click stop base in the horizontal axis and really right stuff rails and panning base in the vertical axis.

For automatic rigs we used 2 Clauss pan and tilt heads sold by Rodeon. These are lightweight and run on Bluetooth enabling us to put the camera on a crane and run it from a laptop. We ran usb cables with repeaters to get the images from the camera directly to the laptop. Brian Smith at Digital Domain wrote the custom software we used to run the Clauss heads. His software allowed us to see the images stitched on the laptop as the images were being shot. We could also pan around the sphere while shooting, and select any tile to reshoot, when needed. The coolest feature is that the software will combine the bracketed images into HDR images in real time on the gpu, and it will highlight any part of the image that is out of range. The software would also automatically stop the camera up or down to shoot additional brackets if any part of the image was out of range, making sure we always had full dynamic range needed on any tile.

team walking the crane in Grossglockner, Austria

We also had this great lightweight crane we rented from ABC cranes in Germany. It extended about 6 meters and could be carried by one person (not counting the 75 lbs of counterweights). We used the crane to extend the camera over the edge for shots like the top of the tower in Siena square in Italy.

What software packages did you use?
We basically had a 2 pronged attack for stitching and processing our images. We shot everything raw with half res jpegs on the camera. We first would make a quick stitch of the jpegs for every single setup and render a 6k spherical jpeg and a qtvr for every single position. After specific positions were selected by the Wachowski Brothers or John and Dan, we would then batch process the bracketed images into HDR exr files, then fine tune the stitches to perfection, and finally render high resolution HDRs. I wrote many python scripts to automate most of the batch processing, and we had a small farm of computers running every night to create the HDRs and final renders.

For the initial automatic stitches we relied mainly on Autopano Pro. And for the final exr renders we used PT Gui.

Which were some of the final statistics for the amount of images you shot?
So in the end we shot over 10,000 individual bubbles. That amounted to around 10 terabytes of data including the processed HDRs.

Can you tell us more about the professional services you offer ?
Currently I offer consulting services for environment reconstructions for visual effects and pretty much anything involving panorama photography. I hope to branch out a bit in the future to do projects outside of visual effects industry.

What advice would you offer those considering entering the VR photography field as professionals?
My advise is to batch process as much as possible.

Future plans and projects you can talk about?
My immediate plan is simply to take some time off. Speed Racer was one of the most ambitious projects I have ever worked on, and the last few months I was working 60-90 hour weeks. I hope audiences enjoy the amount of work that all the talented artists put into the film, and enjoy it as much as we did creating it.

Dennis Martin
Email:vbshooter at gmail.com
Phone: 415.812.9665

Links:
DennisAMartin.com
VRMag's Speed Racer coverage
Speed Racer official site

Related articles in this issue:
WHEN CINEMA MEETS VR - JOHN GAETA TALKS ABOUT SPEED RACER
LUBO HRISTOV VFX ENVIRONMENTS ART DIRECTOR ABOUT SPEED RACER
DIGITAL EFFECTS SUPERVISOR JAKE MORRISON ABOUT SPEED RACER