Yebin Liu 刘烨斌

 

 

 

Max-Planck-Institut for Informatik

Graphics, Vision and Video Group

Campus E 1 4, Room 223

66123 Saarbrucken Germany

Telephone:  0681-9325-423(o)

E-Mail:   yliu@mpi-inf.mpg.de



 

 

 

 

 

 

 

 

I was a resercher working with Christian Theobalt in MPI.

I am now working in Tsinghua University and this webpage will not be updated.

Research Interests

Research Projects

Video-based Characters - Creating New Human Performances from a Multi-view Video Database Siggraph 2011（Conditionally accepted ）
We present a method to synthesize plausible video sequences of humans according to user-defined body motions and viewpoints. We first capture a small database of multi-view video sequences of an actor performing various basic motions. This database needs to be captured only once and serves as the input to our synthesis algorithm. We then apply a marker-less model-based performance capture approach to the entire database to obtain pose and geometry of the actor in each database frame. To create novel video sequences of the actor from the database, a user animates a 3D human skeleton with novel motion and viewpoints. Our technique then synthesizes a realistic video sequence of the actor performing the specified motion based only on the initial database. The first key component of our approach is a new efficient retrieval strategy to find appropriate spatio-temporally coherent database frames from which to synthesize target video frames. The second key component is a warping-based texture synthesis approach that uses the retrieved most-similar database frames to synthesize spatio-temporally coherent target video frames. For instance, this enables us to easily create video sequences of actors performing dangerous stunts without them being placed in harm’s way. We show through a variety of result videos and a user study that we can synthesize realistic videos of people, even if the target motions and camera views are different from the database content.

Markerless Motion Capture of Interacting Characters Using Multi-view Image Segmentation CVPR 2011（Oral）
We present a markerless motion capture approach that reconstructs the skeletal motion and detailed time-varying surface geometry of two closely interacting people from multi-view video. Due to ambiguities in feature-to-person assignments and frequent occlusions, it is not feasible to directly apply single-person capture approaches to the multiperson case. We therefore propose a combined image segmentation and tracking approach to overcome these difficulties. A new probabilistic shape and appearance model is employed to segment the input images and to assign each pixel uniquely to one person. Thereafter, a single-person markerless motion and surface capture approach can be applied to each individual, either one-by-one or in parallel, even under strong occlusions. We demonstrate the performance of our approach on several challenging multi-person motions, including dance and martial arts, and also provide a reference dataset for multi-person motion capture with ground truth.

Multi-view Photometric Stereo Using Multiple Unknown Lights TVCG 2011
This works presents a 3D object relighting technique for multi-view-multi-lighting (MVML) image sets. Our relighting technique is a fusion of multi-view photometric stereo (MVPS) technique and image based lighting (IBL) technique. The MVML dataset consists of multiple camera view with each view filmed under multiple time-multiplex illumination modes. A multi-view 3D reconstruction algorithm is applied using our MVPS algorithm. It conposes of Firstly, initial normal maps are obtained to enhance the correspondence mapping. Then, the depth for every pixel is estimated by combining photometric constraint with occlusion robust photo-consistency. Finally, after filtering the point cloud, a Poisson surface reconstruction is applied to obtain a watertight mesh.When MVPS is finised, the reconstructed model is relighted through an image based relighting scheme for each camera view, followed with view-independent texture mapping procedure. Interactive relighting results demonstrate our high quality reconstruction accuracy, realistic relighting effects and real-time relighting performance. Moreover, our relighting technique is suitable for dynamic 3D object relighting.
Continuous Depth Estimation for Multi-view Stereo CVPR09
Depth-map merging approaches have become more and more popular in multi-view stereo (MVS) because of their flexibility and superior performance. The quality of depth map used for merging is vital for accurate 3D reconstruction. While traditional depth map estimation has been performed in a discrete manner, we suggest the use of a continuous counterpart. In this paper, we first integrate silhouette information and epipolar constraint into the variational method for continuous depth map estimation. Then, several depth candidates are generated based on a multiple starting scales (MSS) framework. From these candidates, refined depth maps for each view are synthesized according to path-based NCC (normalized cross correlation) metric and global optimization technique. Finally, the multiview depth maps are merged to produce 3D models. Our algorithm excels at detail capture and produces one of the most accurate results among the current algorithms for sparse MVS datasets according to the Middlebury benchmark. Additionally, our approach shows its outstanding robustness and accuracy in free-viewpoint video scenario.
Point Cloud Based MVS for Free-viewpoint Video TVCG09
Free-viewpoint video data set faces problems as inadequate texture information, noise or blur due to inadequate lighting and object motion, occluded region or unrecorded region due to unpredictable shape and motion and incomplete camera array setting. In this work, we propose a point cloud based multi-view stereo algorithm for free-viewpoint video. Our reconstruction scheme is totally point cloud based, and composes of three sequential stages: point cloud extraction, point cloud merging and point cloud meshing. To improve reconstruction accuracy, point clouds are first extracted according to an occlusion robust color consistency metric, and then, noise and conflicting points are adaptively removed in 3D space based on surface prior and all point properties including position, normal, fidelity, extracted view. To counteract the quality defect of free-viewpoint video datasets and maintain reconstruction completeness and robustness, multiple shape cues such as silhouette, frontier points and implicit points are fused in all the three point cloud reconstruction modules.Reconstruction result can be downloaded from our free-viewpoint video webpage. See the multi-view stereo evaluation project by Steve Seitz et al. for quantitative evaluations of our results.
Multi-camera and Multi-Lighting Dome ICME09a ICME09b ICME09c
In our lab, we construct a dome to record the geometry, texture and motion of human actors in a dedicated multiple-camera studio with controlled lighting and a chromakey background. The diameter of the dome is 6 meters which provides enough space for character perform. 40 PointGrey flea2 cameras are ring-shape arranged on the dome and 320 LEDs are evenly spaced on the hemisphere of the dome. This system is cooperated with Bennett Wilburn in MSRA.
Light Field Camera Array and Dynamic Light Field Streaming EUROSIP08 VCIP07 ICME06
We present a flexible 3DTV system in which multi-view video streams are captured, compressed, transmitted, and finally converted to high-quality 3D video in real time. Our system consists of an 8× 8 camera array, 16 producer PCs, a streaming server, multiple clients, and several auto-stereoscopic displays. The whole system is implemented over IP network to provide multiple users with interactive 2D/3D switching, viewpoint control, and synthesis for dynamic scenes.

Talks

"Geometry, Motion and Appearance Modeling in Multi-camera and Multi-lighting (MVML) Dome".This talk has been presented in the following research groups: (Slides) 

Data, Tool and Code

Code: Camera display software for calibration parameter. Updated by July.23, 2009.

Data: 4x8 dynamic light filed dataset. Filmed in 2005.

Data: Free-viewpoint video on a ring. Updated by Aug.6, 2009.

Data: Multi-view multi-lighting datasets. Updated by July.23, 2009.

Publications

Honors and Awards