Overview of Image Based Modeling Techniques

Introduction

The purpose of this text is to review some of the more recent techniques used in image based modeling. Usually this subject is talked about in the context of IBMR (Image Based Modeling and Rendering). This is quite a wide field that covers various topics such as inverse lighting, light fields, layered depth images, 3d reconstruction etc… This text will focus more precisely on Modeling Geometric objects from images and will not discuss image based rendering or modeling other entities such as lighting unless directly related to retrieving geometry. Image based modeling techniques all rely on some type of visual cue, the following is a list of example visual cues that some of these techniques rely on.

Visual Cues

Shading

Texture

Motion

Focus

Highlights

Shadows

Silhouette

Inter-reflections

Symmetry

Light polarization

Features such as corners and edges

Taxonomy

Image Based Modeling techniques can be categorised in various ways. The following are Taxonomies employed in the literature.

Active vs. Passive

All image based modeling techniques can fall into one of two groups, active and passive. Active techniques change the environment in some way (i.e. Illuminate the environment) while passive techniques capture the environment without changing it. Active techniques are usually more accurate but expensive and not always viable. Passive techniques on the other hand are cheap and viable but at the cost of accuracy.

Autonomous vs. Semi Autonomous

Again all the following techniques can fall into one of these two groups. Autonomous systems require no user interaction while Semi Autonomous systems require varying degrees of user interaction. This field has been researched with different goals in mind. One example is robotics researchers have pursued this area with robot navigation and exploration as a goal and have therefore focused on trying to produce autonomous systems of very high accuracy. 3d graphics and specialFX communities have prioritised aesthetically pleasing results at the cost of autonomy in order to rapidly produce compelling virtual environments.

The following is a list of taxonomies commonly used in this area of research.

Shape from Single View

These are techniques that rely solely on one image for its input. Because these techniques require such little input data they usually necessitate user interaction. One possible technique presented by A. Criminisi, I. Reid and A. Zisserman (ICCV 99) and makes use of the following assumptions;

1 3 orthogonal sets of parallel lines

2 4 known points on ground plane

3 1 height in the scene

An alternative technique for single view that also scales to multiple views is presented by Paul Debevec and used in his Façade application. Another interesting approach is presented in Image-Based Modeling and Photo Editing. In Potential and limitation for the 3D documentation of cultural heritage from a single image certain commercially available software capable of 3d reconstruction from a single view are examined.

Refs:

Potential and limitation for the 3D documentation of cultural heritage from a single image André Streilein and Frank A. van den Heuvel

Single View Metrology A. Criminisi, I. Reid and A. Zisserman

Modeling and Rendering Architecture from Photographs Paul Ernest Debevec

Modeling and Rendering Architecture fromPhotographs: A hybrid geometry- and image-based approach Paul E. Debevec Camillo J. Taylor Jitendra Malik

Recovering Arches in Façade using Ray - Plane intersections in 3-D G. D. Borshukov and P. Debevec

Image-Based Modeling and Photo Editing Byong Mok Oh Max Chen Julie Dorsey Fr´edo Durand

Shape from Stereo

Once a point in 3d space has been projected onto an image plane it looses its depth information. That point can lie anywhere along the ray passing through the centre of projection and the pixel the point was projected onto. However one can recover its depth information of that pixel if it is projected onto the image plane of another camera by using triangulation. Shape from stereo can be decomposed into two problems. The first, is one of correspondence, given a pixel in image A representing the projection of a 3d point what is the projection of that point in image B. This can be reduced to a 1d search by using the epipolar constraint. The second problem is one of triangulation.

See http://cat.middlebury.edu/stereo/ for a good survey. In Spacetime Stereo: Shape Recovery for Dynamic Scenes an active system that uses structured lighting is presented.

Refs:

Motion – Stereo Integration for Depth Estimation, Christoph Strecha and Luc Van Gool

Complete Stereovision using level set methods, Olivier Faugeras and Renaud Keriven

Omnivergent Stereo, Heung-Yeung Shum, Adam Kalai, Steven M. Seitz

ADVANCES IN COMPUTATIONAL STEREO, M. Z. Brown, D. Burschka and G. D. Hager

Fast Stereo Matching Using Rectangular Subregioning and 3D Maximum-Surface Techniques, Changming Sun

Shape from n-View

Space Carving

A Theory of Shape by Space Carving (1998), Kiriakos N. Kutulakos, Steven M. Seitz

Voxel Colouring

Photorealistic Scene Reconstruction by Voxel Coloring, Steven M. Seitz Charles R. Dyer