Greg Downing has recently developed a technique for creating image-based 3D models from panoramas, and then texturing them, allowing the capture of complete environments with a few panoramas that can then be used to "walk" around in, as if in a game.

He took three omni-directional panoramas from inside the Tribunal Plaza in Nice, France, and then used these panoramas to create a photo-grammatically accurate 3D model of the Plaza. Texturing the model with the photos created a very realistic 3D environment that can be explored at high frame rates in real time. The entire model of everything in the plaza is 15,000 polygons, which can very easily be rendered in real-time on consumer video cards.

Greg spent four weeks on this project, including the time to photograph, stitch and develop this process.

This project is a breakthrough in panoramic image modeling. What equipment did you use?

I shot the three panoramas with my Nikon CoolPix 5000, and a Kaidan multi-row head using my iBook as portable storage. I then used REALVIZ Stitcher on a 1.7 MHz PC w/ 2 gigs of RAM and a Geforce 3 video card to stitch the images.

Thirty-two images gave me very high-resolution spherical images to work with, 16,000 by 8,000 pixels. This resolution was necessary to create an accurate 3D model and has the added advantage of allowing the option of producing very high-resolution renders. I can render movies of this project at up to HDTV resolutions.

I then used Image Modeler 3.0 to calibrate the images and did all of the 3D modeling and texturing in Maya 4.0.

I also used several custom scripts that allowed me to work from panoramas rather than still images (Image Modeler does not support complete panoramas), created a custom interface that had multiple panorama viewer windows for modeling in Maya and allowed me to automate the texturing.

What were the major problems you encountered in the various phases of production, shooting, post-production and assembling and web delivery?

A necessary part of doing innovative productions is working around problems. I encountered several new challenges in production, post-production and delivery.

Production:
Unfortunately when I shot the images I only had a 128MB Compact Flash card (my 512 chip took forever to ship from the U.S.) so I had to download to my iBook between panoramas. This delay caused changes in the lighting between panoramas that I had to compensate for later.

It was raining when I was shooting, this was good for ensuring that there were not too many people in the square but it is difficult to shoot while holding an umbrella and having to clean the lens between shots. I chose to shoot on an overcast day because I was concerned that the scene would have more dynamic range than I could capture with a single exposure. Texturing a model from multiple photographs has the same issue with exposure lock that shooting a single panorama has; if the exposure changes between panoramas there is a clear difference between the buildings shot and textured with different exposures.

Post:
Because I was inventing the process while doing the project I ended up changing the process several times. More than a week into the process, I realized that much of what I was trying was too frustrating and going to be too slow. It was a hard choice but I had to go back to very close to the beginning and start over. I took the time to develop it such that the next time I do this process it will be streamlined; unfortunately, I ended up missing a milestone because of this, but in the end I think it was worth it.

Another difficulty I faced was the learning curve of Maya. It is a very powerful 3D package, the more you use it the more you realize that anything you can imagine, and several things you didn't yet imagine, are possible. I was familiar with the functions of 3D packages from the development work I did on Image Modeler but this was my first project with Maya, so it took a while to learn where everything was and to script the missing functions that I needed as part of my workflow. Luckily, I had access to some people that were very knowledgeable about Maya and could answer many of my questions.

Delivery:
When it comes to delivery I was pretty lucky. VRMMP needed only a Maya file to begin their immersive virtual reality project. They will be compiling the model into a standard Virtual Reality SDK called Vega from Multigen. This will allow them to use 1028x768 active stereo goggles, head mounted motion tracking and VR gloves for navigation.

As far as delivering the material to the rest of the world goes, I still have some work to do. The challenge that I faced was that I want to be able to show the world that this is working at very high quality in real-time. Because I wanted to announce it right away, I figured the fastest thing to do would be just to shoot video of me manipulating the model in Maya, which uses Open GL for its 3D representation. For people working in 3D this is proof that it will play well in any 3D engine. I then edited the video with iMovie on my Mac. I also took half day to relearn enough Macromedia Flash to build some images that would show the renders with the underlying geometry, for those who didn't have enough bandwidth to download the movie. I still plan on doing some higher quality and resolution movie renders and I also want to post some type of web3D format. I have investigated a few formats so far, I don't really like the navigation controls for VRML, and in order to customize them you have to use JavaScript, so I may just end up using Shockwave. I am looking into a few web3D QuickTime components as well.

The whole project took four weeks. Can you describe the production timeline and milestones to give us a feel for the scope of the project?

I spent 3 days on the panos and photo editing, a day calibrating, two weeks modeling/texturing, and a week refining the model and textures.

First step was to shoot, stitch and retouch the three panoramas that were the basis of the project. I spent a little longer than usual on this process because stitch errors could have wreaked havoc on the project. I probably spent about three days on this step.

I then spent a good part of a day in Image Modeler calibrating the images. In this project, calibration meant figuring out exactly where the images were taken relative to one another in space and the panorama rotation on the x, y and z axis. By adding 7 to 12 points to the same features in all three of the images, I was able to determine how far apart the panoramas were from one another and the precise angular relationships. I also added a number of 3D points to the edges and corners of a few buildings; I used these for reference when building my 3D model.

I then spent most of the two weeks modeling everything that could be seen from the three panoramas in square. As I mentioned, I was still on a bit of a learning curve when it came to the modeling step. That often meant I took three steps forward and two steps back. I spent a few days creating the textures and I had to redesign how this worked several times before I came up with techniques that would work under all the circumstances in the project.

The last week was spent refining the whole project. I fabricated a bit of information that wasn't in any of the panoramas by duplicating parts of the different buildings to make new unique buildings. During this time I also added several purely 3D elements that did not photo model so well, such as the railings on the balconies and the multi-layer textured ground plane. The ground plane was made from a repeating pattern I made from a small selection of eight bricks I got from my photographs with several layers of alpha, and more detail repeating at different rates on top, so that it looked less like a repeating pattern. Using the photos for the ground plane would not have worked well since I didn't have an aerial photograph and it would have been a waste of texture memory.

How does your model play on consumer video cards?

I haven't done thorough testing of different video cards yet. I built and have been playing the project using a NVIDIA GeForce 3 video card. It not a professional video card, it was intended for those who wanted to get the most out of video games a few years ago. However older, slower video cards and cards with less video memory will either not successfully play the project at this quality, or will require lower quality textures to play at reasonable frame rates. One nice feature of my project is that it is scaleable. I can recreate it to fit any specification. If the issue is texture memory, it is as easy as resizing the photos I am using for textures; I could update the project in a mater of hours. However, if the target hardware cannot render as many polygons, I would have to know in the beginning of the project before I begin the modeling process.

Where do you see the most interesting application fields pertaining to production costs?

I think the most interesting applications for this will leverage the real-time and real-location capability of this technique. It is much faster than traditional modeling techniques and the result is much more realistic, especially for architectural subjects. It has the added advantage of being able to make fairly accurate models that you can make measurements from. It works less well for organic subjects, such as trees, or subjects that have more complex reflective, highlighting and light remittance properties, such as a human face.

The industries that will be interested in this are: Game Companies; film special effects; virtual sets with blue screen for television; pre-visualization for film (this is used to determine camera angles, movement and placement before actually shooting on location) and has the added advantage of being able to measure, i.e. the director would know ahead of time which crane they needed to fit in a small alley and still reach the 3rd floor window; Virtual Reality and simulation training; interactive museum exhibits; and web3D games and educational tours.

What are you headed next (i.e. when can we be expected to be impressed again with another breakthrough)?

I’ve been studying high resolution High Dynamic Range Imaging for several years now. I am very interested in integrating high resolution HDR with my PIMP techniques to solve some of the more complex graphics problems, making virtual environment like this much more real and cinematic.

As far as the timing goes I can't be sure. It is difficult to do the R&D for these projects independently. Unfortunately, paying gigs usually take precedence over developing new ideas for future customers.

More of Greg Downing's work:
Downing’s Garfield Trailer

Petra Exhibit at the American Museum of Natural History in New York, featuring a Surround movie generated by Downing’s panoramas

Email: greg@gregdowning.com