max planck institut
informatik
informatik
Due to the progress in rendering techniques and graphics hardware in recent years, both the potential for handling complex virtual scenes and the demand for realism in graphics have increased tremendously. The traditional technique that is based on manual design of shapes, textures, reflectance properties and 3D motion by artists is challenged by the number and detail of objects to be modeled, and by the development of sophisticated reflectance models that involve parameters that are difficult to control manually. The goal of learning-based modeling is to replace much of the manual design by automated procedures for measurement and data analysis, and to provide artists with new methods for high-level control of visual appearance. In our group, this is achieved by physical measurements and by a statistical data analysis. The measurements involve 3D scans of faces, and sample images recorded at different illuminations. For data analysis, we use probability density estimation, such as Principal Component Analysis, and regression techniques based on fitting parametric models of shapes or reflectance functions. Moreover, algorithms from Computer Vision, such as Optical Flow, are applied to 3D data processing.
Current projects of the group involve measurements of the reflectance properties of skin, image-based relighting of scenes, robust methods for face recognition, measurements of the anatomical changes in children's faces during growth, and model-based representation of facial movements. The measurements of facial movements are performed with a newly acquired dynamic 3D scanner that records textured depth images at rate of 40 frames per second, which is sufficient for capturing even the fast mouth movements during speech. The data analysis, which is currently going on, is based on our previous work that won the Eurographics 2003 Günter Enderle Award (Best Paper Award, 1st Prize) [BBVP03].
Investigators: Martin Fuchs, Hendrik Lensch, and Volker Blanz
Visual effects in movie productions have involved more and more synthetic scenes with human faces in recent years. For photo-realistic results, it is crucial to capture the complex reflectance properties of skin. Exhaustive methods, as pioneered by Debevec et al. [DHT+00], record the complete light-dependent appearance of objects. However, even when restricted to a fixed viewpoint, they require thousands of input images in controlled conditions.
Lensch et al. [LKG+01] have presented a method which reconstructs a spatially variant Bidirectional Reflectance Function (BRDF) on an object surface of known geometry, yielding a relightable free-viewpoint object representation. By clustering the data to similar regions, different material types in the input can be distinguished. As the BRDF is expressed by few parameters of a BRDF model, the number of required input images remains low.
Measurements of faces, however, pose additional challenges since the face cannot be kept rigid throughout the measurement period. Precise registration of shape and reflectance, and of the separate reflectance samples at different illuminations are required. In 1999, Blanz and Vetter[BV99] introduced a morphable face model that was learned from a dataset of 3D scans and provides a reconstruction of face geometry from even a single image, thus solving the registration problem.
In the project ``Reflectance Properties of Human Faces'', we merge the technologies by Lensch and Blanz-Vetter. Our acquisition setup builds on work previously done in our working group: the measurements take place in a dark room with controllable near-point light conditions [GLHS00]. A test subject is lit by a movable HMI light source, the position of which is determined by calibration images as in the method by Lensch et al. We take about 20 pictures in varying orientations towards the viewer (typically 3) and different light conditions (about 7).
We fit the morphable face model by Blanz-Vetter on each input image to obtain a geometry estimate, a pose estimate and a texture of the face, compensating for movements over the measurement process (see Figure 0.1).
|
|
|
Due to the general nature of the morphable model, the texture is provided in a parameterization of the face surface which is consistent for all human faces. Applying the information about incident light and camera, this allows us to combine the different illumination samples and estimate BRDF parameters at each point using non-linear optimization. Clusters of different materials in the face can be defined a-priori. Within each of these clusters, most BRDF parameters are spatially constant, while a few are optimized locally, capturing fine details such as pores and freckles (Figure 0.2). As an additional benefit of the common face surface parameterization, face surface properties can easily be transferred from one human to another.
|
|
|
This work has emerged from a diploma thesis [Fuc04] and resulted in a publication at the IEEE Transactions for Visualization and Computer Graphics, to appear in 2005 [FBLS05].
Investigators: Kristina Scherbaum and Volker Blanz
A well-known procedure in traditional photo-retouche is to replace a face in an image by another individual's face, based on simple 2D cut-and-paste operations. However, this is only applicable if both faces are shown from the same viewing direction. We have developed a 3D model-based technique that allows the user to exchange any two faces in images. It automatically compensates for changes in viewpoint, illumination and color tone and contrast, reducing the manual work to clicking 7 feature points in both images [BSVS04].
The algorithm introduces a new paradigm for high-level image manipulation by entirely separating face identity from scene and image parameters. After fitting a morphable model of 3D faces [BV03] both to the face that is to be replaced, and to the new individual to be pasted, the new individual's 3D face is drawn with the scene parameters of the original face. The original face, thus, can be considered a light probe with unknown shape and texture for measuring the scene illumination. Practical applications of the software can be found in mixed reality, image processing and consumer software. The paper presenting this work won the Eurographics 2004 Günter Enderle Award (Best Paper Award, 1st Prize) [BSVS04].
|
An additional, very relevant application of our system is in face recognition. We have presented a new paradigm for face recognition across changes in viewpoint that computes a 3D reconstruction from a side view, renders a synthetic front view, and transfers this image to an commercial, image-based recognition system [BGPV05]. This approach combines the robustness of 3D model-based recognition with the efficiency of image-based systems. In a comparison with our previous, model-based recognition [BV03], we have shown that the intermediate synthetic image does not involve a significant loss of diagnostic information about identity [BGPV05].
