Publications

Human performance capture is a highly important computer vision problem with
many applications in movie production and virtual/augmented reality. Many
previous performance capture approaches either required expensive multi-view
setups or did not recover dense space-time coherent geometry with
frame-to-frame correspondences. We propose a novel deep learning approach for
monocular dense human performance capture. Our method is trained in a weakly
supervised manner based on multi-view supervision completely removing the need
for training data with 3D ground truth annotations. The network architecture is
based on two separate networks that disentangle the task into a pose estimation
and a non-rigid surface deformation step. Extensive qualitative and
quantitative evaluations show that our approach outperforms the state of the
art in terms of quality and robustness. This work is an extended version of
DeepCap where we provide more detailed explanations, comparisons and results as
well as applications.

Image deblurring aims to recover the latent sharp image from its blurry
counterpart and has a wide range of applications in computer vision. The
Convolution Neural Networks (CNNs) have performed well in this domain for many
years, and until recently an alternative network architecture, namely
Transformer, has demonstrated even stronger performance. One can attribute its
superiority to the multi-head self-attention (MHSA) mechanism, which offers a
larger receptive field and better input content adaptability than CNNs.
However, as MHSA demands high computational costs that grow quadratically with
respect to the input resolution, it becomes impractical for high-resolution
image deblurring tasks. In this work, we propose a unified lightweight CNN
network that features a large effective receptive field (ERF) and demonstrates
comparable or even better performance than Transformers while bearing less
computational costs. Our key design is an efficient CNN block dubbed LaKD,
equipped with a large kernel depth-wise convolution and spatial-channel mixing
structure, attaining comparable or larger ERF than Transformers but with a
smaller parameter scale. Specifically, we achieve +0.17dB / +0.43dB PSNR over
the state-of-the-art Restormer on defocus / motion deblurring benchmark
datasets with 32% fewer parameters and 39% fewer MACs. Extensive experiments
demonstrate the superior performance of our network and the effectiveness of
each module. Furthermore, we propose a compact and intuitive ERFMeter metric
that quantitatively characterizes ERF, and shows a high correlation to the
network performance. We hope this work can inspire the research community to
further explore the pros and cons of CNN and Transformer architectures beyond
image deblurring tasks.

Eye-tracking technology is an integral component of new display devices such
as virtual and augmented reality headsets. Applications of gaze information
range from new interaction techniques exploiting eye patterns to
gaze-contingent digital content creation. However, system latency is still a
significant issue in many of these applications because it breaks the
synchronization between the current and measured gaze positions. Consequently,
it may lead to unwanted visual artifacts and degradation of user experience. In
this work, we focus on foveated rendering applications where the quality of an
image is reduced towards the periphery for computational savings. In foveated
rendering, the presence of latency leads to delayed updates to the rendered
frame, making the quality degradation visible to the user. To address this
issue and to combat system latency, recent work proposes to use saccade landing
position prediction to extrapolate the gaze information from delayed
eye-tracking samples. While the benefits of such a strategy have already been
demonstrated, the solutions range from simple and efficient ones, which make
several assumptions about the saccadic eye movements, to more complex and
costly ones, which use machine learning techniques. Yet, it is unclear to what
extent the prediction can benefit from accounting for additional factors. This
paper presents a series of experiments investigating the importance of
different factors for saccades prediction in common virtual and augmented
reality applications. In particular, we investigate the effects of saccade
orientation in 3D space and smooth pursuit eye-motion (SPEM) and how their
influence compares to the variability across users. We also present a simple
yet efficient correction method that adapts the existing saccade prediction
methods to handle these factors without performing extensive data collection.

Video frame interpolation (VFI) enables many important applications that
might involve the temporal domain, such as slow motion playback, or the spatial
domain, such as stop motion sequences. We are focusing on the former task,
where one of the key challenges is handling high dynamic range (HDR) scenes in
the presence of complex motion. To this end, we explore possible advantages of
dual-exposure sensors that readily provide sharp short and blurry long
exposures that are spatially registered and whose ends are temporally aligned.
This way, motion blur registers temporally continuous information on the scene
motion that, combined with the sharp reference, enables more precise motion
sampling within a single camera shot. We demonstrate that this facilitates a
more complex motion reconstruction in the VFI task, as well as HDR frame
reconstruction that so far has been considered only for the originally captured
frames, not in-between interpolated frames. We design a neural network trained
in these tasks that clearly outperforms existing solutions. We also propose a
metric for scene motion complexity that provides important insights into the
performance of VFI methods at the test time.

Resolving morphological chemical phase transformations at the nanoscale is of
vital importance to many scientific and industrial applications across various
disciplines. The TXM-XANES imaging technique, by combining full field
transmission X-ray microscopy (TXM) and X-ray absorption near edge structure
(XANES), has been an emerging tool which operates by acquiring a series of
microscopy images with multi-energy X-rays and fitting to obtain the chemical
map. Its capability, however, is limited by the poor signal-to-noise ratios due
to the system errors and low exposure illuminations for fast acquisition. In
this work, by exploiting the intrinsic properties and subspace modeling of the
TXM-XANES imaging data, we introduce a simple and robust denoising approach to
improve the image quality, which enables fast and high-sensitivity chemical
imaging. Extensive experiments on both synthetic and real datasets demonstrate
the superior performance of the proposed method.

Although considerable progress has been made in semantic scene understanding
under clear weather, it is still a tough problem under adverse weather
conditions, such as dense fog, due to the uncertainty caused by imperfect
observations. Besides, difficulties in collecting and labeling foggy images
hinder the progress of this field. Considering the success in semantic scene
understanding under clear weather, we think it is reasonable to transfer
knowledge learned from clear images to the foggy domain. As such, the problem
becomes to bridge the domain gap between clear images and foggy images. Unlike
previous methods that mainly focus on closing the domain gap caused by fog --
defogging the foggy images or fogging the clear images, we propose to alleviate
the domain gap by considering fog influence and style variation simultaneously.
The motivation is based on our finding that the style-related gap and the
fog-related gap can be divided and closed respectively, by adding an
intermediate domain. Thus, we propose a new pipeline to cumulatively adapt
style, fog and the dual-factor (style and fog). Specifically, we devise a
unified framework to disentangle the style factor and the fog factor
separately, and then the dual-factor from images in different domains.
Furthermore, we collaborate the disentanglement of three factors with a novel
cumulative loss to thoroughly disentangle these three factors. Our method
achieves the state-of-the-art performance on three benchmarks and shows
generalization ability in rainy and snowy scenes.

Most in-the-wild images are stored in Low Dynamic Range (LDR) form, serving
as a partial observation of the High Dynamic Range (HDR) visual world. Despite
limited dynamic range, these LDR images are often captured with different
exposures, implicitly containing information about the underlying HDR image
distribution. Inspired by this intuition, in this work we present, to the best
of our knowledge, the first method for learning a generative model of HDR
images from in-the-wild LDR image collections in a fully unsupervised manner.
The key idea is to train a generative adversarial network (GAN) to generate HDR
images which, when projected to LDR under various exposures, are
indistinguishable from real LDR images. The projection from HDR to LDR is
achieved via a camera model that captures the stochasticity in exposure and
camera response function. Experiments show that our method GlowGAN can
synthesize photorealistic HDR images in many challenging cases such as
landscapes, lightning, or windows, where previous supervised generative models
produce overexposed images. We further demonstrate the new application of
unsupervised inverse tone mapping (ITM) enabled by GlowGAN. Our ITM method does
not need HDR images or paired multi-exposure images for training, yet it
reconstructs more plausible information for overexposed regions than
state-of-the-art supervised learning models trained on such data.

In computational design and fabrication, neural networks are becoming
important surrogates for bulky forward simulations. A long-standing,
intertwined question is that of inverse design: how to compute a design that
satisfies a desired target performance? Here, we show that the piecewise linear
property, very common in everyday neural networks, allows for an inverse design
formulation based on mixed-integer linear programming. Our mixed-integer
inverse design uncovers globally optimal or near optimal solutions in a
principled manner. Furthermore, our method significantly facilitates emerging,
but challenging, combinatorial inverse design tasks, such as material
selection. For problems where finding the optimal solution is not desirable or
tractable, we develop an efficient yet near-optimal hybrid optimization.
Eventually, our method is able to find solutions provably robust to possible
fabrication perturbations among multiple designs with similar performances.

Most of the images one finds in the media, such as on the Internet or in textbooks and magazines, contain humans as the main point of attention. Thus, there is an inherent necessity for industry, society, and private persons to be able to thoroughly analyze and synthesize the human-related content in these images. One aspect of this analysis and subject of this thesis is to infer the 3D pose and surface deformation, using only visual information, which is also known as human performance capture. Human performance capture enables the tracking of virtual characters from real-world observations, and this is key for visual effects, games, VR, and AR, to name just a few application areas. However, traditional capture methods usually rely on expensive multi-view (marker-based) systems that are prohibitively expensive for the vast majority of people, or they use depth sensors, which are still not as common as single color cameras. Recently, some approaches have attempted to solve the task by assuming only a single RGB image is given. Nonetheless, they can either not track the dense deforming geometry of the human, such as the clothing layers, or they are far from real time, which is indispensable for many applications. To overcome these shortcomings, this thesis proposes two monocular human performance capture methods, which for the first time allow the real-time capture of the dense deforming geometry as well as an unseen 3D accuracy for pose and surface deformations. At the technical core, this work introduces novel GPU-based and data-parallel optimization strategies in conjunction with other algorithmic design choices that are all geared towards real-time performance at high accuracy. Moreover, this thesis presents a new weakly supervised multiview training strategy combined with a fully differentiable character representation that shows superior 3D accuracy. However, there is more to human-related Computer Vision than only the analysis of people in images. It is equally important to synthesize new images of humans in unseen poses and also from camera viewpoints that have not been observed in the real world. Such tools are essential for the movie industry because they, for example, allow the synthesis of photo-realistic virtual worlds with real-looking humans or of contents that are too dangerous for actors to perform on set. But also video conferencing and telepresence applications can benefit from photo-real 3D characters, as they can enhance the immersive experience of these applications. Here, the traditional Computer Graphics pipeline for rendering photo-realistic images involves many tedious and time-consuming steps that require expert knowledge and are far from real time. Traditional rendering involves character rigging and skinning, the modeling of the surface appearance properties, and physically based ray tracing. Recent learning-based methods attempt to simplify the traditional rendering pipeline and instead learn the rendering function from data resulting in methods that are easier accessible to non-experts. However, most of them model the synthesis task entirely in image space such that 3D consistency cannot be achieved, and/or they fail to model motion- and view-dependent appearance effects. To this end, this thesis presents a method and ongoing work on character synthesis, which allow the synthesis of controllable photoreal characters that achieve motion- and view-dependent appearance effects as well as 3D consistency and which run in real time. This is technically achieved by a novel coarse-to-fine geometric character representation for efficient synthesis, which can be solely supervised on multi-view imagery. Furthermore, this work shows how such a geometric representation can be combined with an implicit surface representation to boost synthesis and geometric quality.

We propose the first approach to automatically and jointly synthesize both
the synchronous 3D conversational body and hand gestures, as well as 3D face
and head animations, of a virtual character from speech input. Our algorithm
uses a CNN architecture that leverages the inherent correlation between facial
expression and hand gestures. Synthesis of conversational body gestures is a
multi-modal problem since many similar gestures can plausibly accompany the
same input speech. To synthesize plausible body gestures in this setting, we
train a Generative Adversarial Network (GAN) based model that measures the
plausibility of the generated sequences of 3D body motion when paired with the
input audio features. We also contribute a new way to create a large corpus of
more than 33 hours of annotated body, hand, and face data from in-the-wild
videos of talking people. To this end, we apply state-of-the-art monocular
approaches for 3D body and hand pose estimation as well as dense 3D face
performance capture to the video corpus. In this way, we can train on orders of
magnitude more data than previous algorithms that resort to complex in-studio
motion capture solutions, and thereby train more expressive synthesis
algorithms. Our experiments and user study show the state-of-the-art quality of
our speech-synthesized full 3D character animations.

Virtual reality has the potential to change the way we create and consume
content in our everyday life. Entertainment, training, design and
manufacturing, communication, or advertising are all applications that already
benefit from this new medium reaching consumer level. VR is inherently
different from traditional media: it offers a more immersive experience, and
has the ability to elicit a sense of presence through the place and
plausibility illusions. It also gives the user unprecedented capabilities to
explore their environment, in contrast with traditional media. In VR, like in
the real world, users integrate the multimodal sensory information they receive
to create a unified perception of the virtual world. Therefore, the sensory
cues that are available in a virtual environment can be leveraged to enhance
the final experience. This may include increasing realism, or the sense of
presence; predicting or guiding the attention of the user through the
experience; or increasing their performance if the experience involves the
completion of certain tasks. In this state-of-the-art report, we survey the
body of work addressing multimodality in virtual reality, its role and benefits
in the final user experience. The works here reviewed thus encompass several
fields of research, including computer graphics, human computer interaction, or
psychology and perception. Additionally, we give an overview of different
applications that leverage multimodal input in areas such as medicine, training
and education, or entertainment; we include works in which the integration of
multiple sensory information yields significant improvements, demonstrating how
multimodality can play a fundamental role in the way VR systems are designed,
and VR experiences created and consumed.

Human re-rendering from a single image is a starkly under-constrained
problem, and state-of-the-art algorithms often exhibit undesired artefacts,
such as over-smoothing, unrealistic distortions of the body parts and garments,
or implausible changes of the texture. To address these challenges, we propose
a new method for neural re-rendering of a human under a novel user-defined pose
and viewpoint, given one input image. Our algorithm represents body pose and
shape as a parametric mesh which can be reconstructed from a single image and
easily reposed. Instead of a colour-based UV texture map, our approach further
employs a learned high-dimensional UV feature map to encode appearance. This
rich implicit representation captures detailed appearance variation across
poses, viewpoints, person identities and clothing styles better than learned
colour texture maps. The body model with the rendered feature maps is fed
through a neural image-translation network that creates the final rendered
colour image. The above components are combined in an end-to-end-trained neural
network architecture that takes as input a source person image, and images of
the parametric body model in the source pose and desired target pose.
Experimental evaluation demonstrates that our approach produces higher quality
single image re-rendering results than existing methods.

Foveated image reconstruction recovers full image from a sparse set of
samples distributed according to the human visual system's retinal sensitivity
that rapidly drops with eccentricity. Recently, the use of Generative
Adversarial Networks was shown to be a promising solution for such a task as
they can successfully hallucinate missing image information. Like for other
supervised learning approaches, also for this one, the definition of the loss
function and training strategy heavily influences the output quality. In this
work, we pose the question of how to efficiently guide the training of foveated
reconstruction techniques such that they are fully aware of the human visual
system's capabilities and limitations, and therefore, reconstruct visually
important image features. Due to the nature of GAN-based solutions, we
concentrate on the human's sensitivity to hallucination for different input
sample densities. We present new psychophysical experiments, a dataset, and a
procedure for training foveated image reconstruction. The strategy provides
flexibility to the generator network by penalizing only perceptually important
deviations in the output. As a result, the method aims to preserve perceived
image statistics rather than natural image statistics. We evaluate our strategy
and compare it to alternative solutions using a newly trained objective metric
and user experiments.

We propose Blue Noise Plots, two-dimensional dot plots that depict data
points of univariate data sets. While often one-dimensional strip plots are
used to depict such data, one of their main problems is visual clutter which
results from overlap. To reduce this overlap, jitter plots were introduced,
whereby an additional, non-encoding plot dimension is introduced, along which
the data point representing dots are randomly perturbed. Unfortunately, this
randomness can suggest non-existent clusters, and often leads to visually
unappealing plots, in which overlap might still occur. To overcome these
shortcomings, we introduce BlueNoise Plots where random jitter along the
non-encoding plot dimension is replaced by optimizing all dots to keep a
minimum distance in 2D i. e., Blue Noise. We evaluate the effectiveness as well
as the aesthetics of Blue Noise Plots through both, a quantitative and a
qualitative user study.

High Dynamic Range (HDR) content is becoming ubiquitous due to the rapid
development of capture technologies. Nevertheless, the dynamic range of common
display devices is still limited, therefore tone mapping (TM) remains a key
challenge for image visualization. Recent work has demonstrated that neural
networks can achieve remarkable performance in this task when compared to
traditional methods, however, the quality of the results of these
learning-based methods is limited by the training data. Most existing works use
as training set a curated selection of best-performing results from existing
traditional tone mapping operators (often guided by a quality metric),
therefore, the quality of newly generated results is fundamentally limited by
the performance of such operators. This quality might be even further limited
by the pool of HDR content that is used for training. In this work we propose a
learning-based self-supervised tone mapping operator that is trained at test
time specifically for each HDR image and does not need any data labeling. The
key novelty of our approach is a carefully designed loss function built upon
fundamental knowledge on contrast perception that allows for directly comparing
the content in the HDR and tone mapped images. We achieve this goal by
reformulating classic VGG feature maps into feature contrast maps that
normalize local feature differences by their average magnitude in a local
neighborhood, allowing our loss to account for contrast masking effects. We
perform extensive ablation studies and exploration of parameters and
demonstrate that our solution outperforms existing approaches with a single set
of fixed parameters, as confirmed by both objective and subjective metrics.

The humble loop shrinking property played a central role in the inception of
modern topology but it has been eclipsed by more abstract algebraic formalism.
This is particularly true in the context of detecting relevant non-contractible
loops on surfaces where elaborate homological and/or graph theoretical
constructs are favored in algorithmic solutions. In this work, we devise a
variational analogy to the loop shrinking property and show that it yields a
simple, intuitive, yet powerful solution allowing a streamlined treatment of
the problem of handle and tunnel loop detection. Our formalization tracks the
evolution of a diffusion front randomly initiated on a single location on the
surface. Capitalizing on a diffuse interface representation combined with a set
of rules for concurrent front interactions, we develop a dynamic data structure
for tracking the evolution on the surface encoded as a sparse matrix which
serves for performing both diffusion numerics and loop detection and acts as
the workhorse of our fully parallel implementation. The substantiated results
suggest our approach outperforms state of the art and robustly copes with
highly detailed geometric models. As a byproduct, our approach can be used to
construct Reeb graphs by diffusion thus avoiding commonly encountered issues
when using Morse functions.

This article introduces a new physics-based method for rigid point set
alignment called Fast Gravitational Approach (FGA). In FGA, the source and
target point sets are interpreted as rigid particle swarms with masses
interacting in a globally multiply-linked manner while moving in a simulated
gravitational force field. The optimal alignment is obtained by explicit
modeling of forces acting on the particles as well as their velocities and
displacements with second-order ordinary differential equations of motion.
Additional alignment cues (point-based or geometric features, and other
boundary conditions) can be integrated into FGA through particle masses. We
propose a smooth-particle mass function for point mass initialization, which
improves robustness to noise and structural discontinuities. To avoid
prohibitive quadratic complexity of all-to-all point interactions, we adapt a
Barnes-Hut tree for accelerated force computation and achieve quasilinear
computational complexity. We show that the new method class has characteristics
not found in previous alignment methods such as efficient handling of partial
overlaps, inhomogeneous point sampling densities, and coping with large point
clouds with reduced runtime compared to the state of the art. Experiments show
that our method performs on par with or outperforms all compared competing
non-deep-learning-based and general-purpose techniques (which do not assume the
availability of training data and a scene prior) in resolving transformations
for LiDAR data and gains state-of-the-art accuracy and speed when coping with
different types of data disturbances.

We suggest to represent an X-Field -a set of 2D images taken across different
view, time or illumination conditions, i.e., video, light field, reflectance
fields or combinations thereof-by learning a neural network (NN) to map their
view, time or light coordinates to 2D images. Executing this NN at new
coordinates results in joint view, time or light interpolation. The key idea to
make this workable is a NN that already knows the "basic tricks" of graphics
(lighting, 3D projection, occlusion) in a hard-coded and differentiable form.
The NN represents the input to that rendering as an implicit map, that for any
view, time, or light coordinate and for any pixel can quantify how it will move
if view, time or light coordinates change (Jacobian of pixel position with
respect to view, time, illumination, etc.). Our X-Field representation is
trained for one scene within minutes, leading to a compact set of trainable
parameters and hence real-time navigation in view, time and illumination.

We present a convex mixed-integer programming formulation for non-rigid shape
matching. To this end, we propose a novel shape deformation model based on an
efficient low-dimensional discrete model, so that finding a globally optimal
solution is tractable in (most) practical cases. Our approach combines several
favourable properties: it is independent of the initialisation, it is much more
efficient to solve to global optimality compared to analogous quadratic
assignment problem formulations, and it is highly flexible in terms of the
variants of matching problems it can handle. Experimentally we demonstrate that
our approach outperforms existing methods for sparse shape matching, that it
can be used for initialising dense shape matching methods, and we showcase its
flexibility on several examples.

We address the problem of fitting 3D human models to 3D scans of dressed
humans. Classical methods optimize both the data-to-model correspondences and
the human model parameters (pose and shape), but are reliable only when
initialized close to the solution. Some methods initialize the optimization
based on fully supervised correspondence predictors, which is not
differentiable end-to-end, and can only process a single scan at a time. Our
main contribution is LoopReg, an end-to-end learning framework to register a
corpus of scans to a common 3D human model. The key idea is to create a
self-supervised loop. A backward map, parameterized by a Neural Network,
predicts the correspondence from every scan point to the surface of the human
model. A forward map, parameterized by a human model, transforms the
corresponding points back to the scan based on the model parameters (pose and
shape), thus closing the loop. Formulating this closed loop is not
straightforward because it is not trivial to force the output of the NN to be
on the surface of the human model - outside this surface the human model is not
even defined. To this end, we propose two key innovations. First, we define the
canonical surface implicitly as the zero level set of a distance field in R3,
which in contrast to morecommon UV parameterizations, does not require cutting
the surface, does not have discontinuities, and does not induce distortion.
Second, we diffuse the human model to the 3D domain R3. This allows to map the
NN predictions forward,even when they slightly deviate from the zero level set.
Results demonstrate that we can train LoopRegmainly self-supervised - following
a supervised warm-start, the model becomes increasingly more accurate as
additional unlabelled raw scans are processed. Our code and pre-trained models
can be downloaded for research.

Implicit functions represented as deep learning approximations are powerful
for reconstructing 3D surfaces. However, they can only produce static surfaces
that are not controllable, which provides limited ability to modify the
resulting model by editing its pose or shape parameters. Nevertheless, such
features are essential in building flexible models for both computer graphics
and computer vision. In this work, we present methodology that combines
detail-rich implicit functions and parametric representations in order to
reconstruct 3D models of people that remain controllable and accurate even in
the presence of clothing. Given sparse 3D point clouds sampled on the surface
of a dressed person, we use an Implicit Part Network (IP-Net)to jointly predict
the outer 3D surface of the dressed person, the and inner body surface, and the
semantic correspondences to a parametric body model. We subsequently use
correspondences to fit the body model to our inner surface and then non-rigidly
deform it (under a parametric body + displacement model) to the outer surface
in order to capture garment, face and hair detail. In quantitative and
qualitative experiments with both full body data and hand scans we show that
the proposed methodology generalizes, and is effective even given incomplete
point clouds collected from single-view depth images. Our models and code can
be downloaded from virtualhumans.mpi-inf.mpg.de/ipnet.

Realistic image synthesis involves computing high-dimensional light transport
integrals which in practice are numerically estimated using Monte Carlo
integration. The error of this estimation manifests itself in the image as
visually displeasing aliasing or noise. To ameliorate this, we develop a
theoretical framework for optimizing screen-space error distribution. Our model
is flexible and works for arbitrary target error power spectra. We focus on
perceptual error optimization by leveraging models of the human visual system's
(HVS) point spread function (PSF) from halftoning literature. This results in a
specific optimization problem whose solution distributes the error as visually
pleasing blue noise in image space. We develop a set of algorithms that provide
a trade-off between quality and speed, showing substantial improvements over
prior state of the art. We perform evaluations using both quantitative and
perceptual error metrics to support our analysis, and provide extensive
supplemental material to help evaluate the perceptual improvements achieved by
our methods.

We seek to reconstruct sharp and noise-free high-dynamic range (HDR) video
from a dual-exposure sensor that records different low-dynamic range (LDR)
information in different pixel columns: Odd columns provide low-exposure,
sharp, but noisy information; even columns complement this with less noisy,
high-exposure, but motion-blurred data. Previous LDR work learns to deblur and
denoise (DISTORTED->CLEAN) supervised by pairs of CLEAN and DISTORTED images.
Regrettably, capturing DISTORTED sensor readings is time-consuming; as well,
there is a lack of CLEAN HDR videos. We suggest a method to overcome those two
limitations. First, we learn a different function instead: CLEAN->DISTORTED,
which generates samples containing correlated pixel noise, and row and column
noise, as well as motion blur from a low number of CLEAN sensor readings.
Second, as there is not enough CLEAN HDR video available, we devise a method to
learn from LDR video in-stead. Our approach compares favorably to several
strong baselines, and can boost existing methods when they are re-trained on
our data. Combined with spatial and temporal super-resolution, it enables
applications such as re-lighting with low noise or blur.

New approaches to synthesize and manipulate face videos at very high quality
have paved the way for new applications in computer animation, virtual and
augmented reality, or face video analysis. However, there are concerns that
they may be used in a malicious way, e.g. to manipulate videos of public
figures, politicians or reporters, to spread false information. The research
community therefore developed techniques for automated detection of modified
imagery, and assembled benchmark datasets showing manipulatons by
state-of-the-art techniques. In this paper, we contribute to this initiative in
two ways: First, we present a new audio-visual benchmark dataset. It shows some
of the highest quality visual manipulations available today. Human observers
find them significantly harder to identify as forged than videos from other
benchmarks. Furthermore we propose new family of deep-learning-based fake
detectors, demonstrating that existing detectors are not well-suited for
detecting fakes of a quality as high as presented in our dataset. Our detectors
examine spatial and temporal features. This allows them to outperform existing
approaches both in terms of high detection accuracy and generalization to
unseen fake generation methods and unseen identities.

Polyhedral surfaces are fundamental objects in architectural geometry and industrial design. Whereas closeness of a given mesh to a smooth reference surface and its suitability for numerical simulations were already studied extensively, the aim of our work is to find and to discuss suitable assessments of smoothness of polyhedral surfaces that only take the geometry of the polyhedral surface itself into account. Motivated by analogies to classical differential geometry, we propose a theory of smoothness of polyhedral surfaces including suitable notions of normal vectors, tangent planes, asymptotic directions, and parabolic curves that are invariant under projective transformations. It is remarkable that seemingly mild conditions significantly limit the shapes of faces of a smooth polyhedral surface. Besides being of theoretical interest, we believe that smoothness of polyhedral surfaces is of interest in the architectural context, where vertices and edges of polyhedral surfaces are highly visible.

Human performance capture is a highly important computer vision problem with
many applications in movie production and virtual/augmented reality. Many
previous performance capture approaches either required expensive multi-view
setups or did not recover dense space-time coherent geometry with
frame-to-frame correspondences. We propose a novel deep learning approach for
monocular dense human performance capture. Our method is trained in a weakly
supervised manner based on multi-view supervision completely removing the need
for training data with 3D ground truth annotations. The network architecture is
based on two separate networks that disentangle the task into a pose estimation
and a non-rigid surface deformation step. Extensive qualitative and
quantitative evaluations show that our approach outperforms the state of the
art in terms of quality and robustness.

Video-based human motion transfer creates video animations of humans
following a source motion. Current methods show remarkable results for
tightly-clad subjects. However, the lack of temporally consistent handling of
plausible clothing dynamics, including fine and high-frequency details,
significantly limits the attainable visual quality. We address these
limitations for the first time in the literature and present a new framework
which performs high-fidelity and temporally-consistent human motion transfer
with natural pose-dependent non-rigid deformations, for several types of loose
garments. In contrast to the previous techniques, we perform image generation
in three subsequent stages, synthesizing human shape, structure, and
appearance. Given a monocular RGB video of an actor, we train a stack of
recurrent deep neural networks that generate these intermediate representations
from 2D poses and their temporal derivatives. Splitting the difficult motion
transfer problem into subtasks that are aware of the temporal motion context
helps us to synthesize results with plausible dynamics and pose-dependent
detail. It also allows artistic control of results by manipulation of
individual framework stages. In the experimental results, we significantly
outperform the state-of-the-art in terms of video realism. Our code and data
will be made publicly available.

Photo-realistic free-viewpoint rendering of real-world scenes using classical
computer graphics techniques is challenging, because it requires the difficult
step of capturing detailed appearance and geometry models. Recent studies have
demonstrated promising results by learning scene representations that
implicitly encode both geometry and appearance without 3D supervision. However,
existing approaches in practice often show blurry renderings caused by the
limited network capacity or the difficulty in finding accurate intersections of
camera rays with the scene geometry. Synthesizing high-resolution imagery from
these representations often requires time-consuming optical ray marching. In
this work, we introduce Neural Sparse Voxel Fields (NSVF), a new neural scene
representation for fast and high-quality free-viewpoint rendering. NSVF defines
a set of voxel-bounded implicit fields organized in a sparse voxel octree to
model local properties in each cell. We progressively learn the underlying
voxel structures with a differentiable ray-marching operation from only a set
of posed RGB images. With the sparse voxel octree structure, rendering novel
views can be accelerated by skipping the voxels containing no relevant scene
content. Our method is typically over 10 times faster than the state-of-the-art
(namely, NeRF(Mildenhall et al., 2020)) at inference time while achieving
higher quality results. Furthermore, by utilizing an explicit sparse voxel
representation, our method can easily be applied to scene editing and scene
composition. We also demonstrate several challenging tasks, including
multi-scene learning, free-viewpoint rendering of a moving human, and
large-scale scene rendering. Code and data are available at our website:
github.com/facebookresearch/NSVF.

We present a novel method for multi-view depth estimation from a single
video, which is a critical task in various applications, such as perception,
reconstruction and robot navigation. Although previous learning-based methods
have demonstrated compelling results, most works estimate depth maps of
individual video frames independently, without taking into consideration the
strong geometric and temporal coherence among the frames. Moreover, current
state-of-the-art (SOTA) models mostly adopt a fully 3D convolution network for
cost regularization and therefore require high computational cost, thus
limiting their deployment in real-world applications. Our method achieves
temporally coherent depth estimation results by using a novel Epipolar
Spatio-Temporal (EST) transformer to explicitly associate geometric and
temporal correlation with multiple estimated depth maps. Furthermore, to reduce
the computational cost, inspired by recent Mixture-of-Experts models, we design
a compact hybrid network consisting of a 2D context-aware network and a 3D
matching network which learn 2D context information and 3D disparity cues
separately. Extensive experiments demonstrate that our method achieves higher
accuracy in depth estimation and significant speedup than the SOTA methods.

We present a new learning-based method for multi-frame depth estimation from
a color video, which is a fundamental problem in scene understanding, robot
navigation or handheld 3D reconstruction. While recent learning-based methods
estimate depth at high accuracy, 3D point clouds exported from their depth maps
often fail to preserve important geometric feature (e.g., corners, edges,
planes) of man-made scenes. Widely-used pixel-wise depth errors do not
specifically penalize inconsistency on these features. These inaccuracies are
particularly severe when subsequent depth reconstructions are accumulated in an
attempt to scan a full environment with man-made objects with this kind of
features. Our depth estimation algorithm therefore introduces a Combined Normal
Map (CNM) constraint, which is designed to better preserve high-curvature
features and global planar regions. In order to further improve the depth
estimation accuracy, we introduce a new occlusion-aware strategy that
aggregates initial depth predictions from multiple adjacent views into one
final depth map and one occlusion probability map for the current reference
view. Our method outperforms the state-of-the-art in terms of depth estimation
accuracy, and preserves essential geometric features of man-made indoor scenes
much better than other algorithms.

3D hand shape and pose estimation from a single depth map is a new and
challenging computer vision problem with many applications. The
state-of-the-art methods directly regress 3D hand meshes from 2D depth images
via 2D convolutional neural networks, which leads to artefacts in the
estimations due to perspective distortions in the images. In contrast, we
propose a novel architecture with 3D convolutions trained in a
weakly-supervised manner. The input to our method is a 3D voxelized depth map,
and we rely on two hand shape representations. The first one is the 3D
voxelized grid of the shape which is accurate but does not preserve the mesh
topology and the number of mesh vertices. The second representation is the 3D
hand surface which is less accurate but does not suffer from the limitations of
the first representation. We combine the advantages of these two
representations by registering the hand surface to the voxelized hand shape. In
the extensive experiments, the proposed approach improves over the state of the
art by 47.8% on the SynHand5M dataset. Moreover, our augmentation policy for
voxelized depth maps further enhances the accuracy of 3D hand pose estimation
on real data. Our method produces visually more reasonable and realistic hand
shapes on NYU and BigHand2.2M datasets compared to the existing approaches.

Most 3D face reconstruction methods rely on 3D morphable models, which
disentangle the space of facial deformations into identity geometry,
expressions and skin reflectance. These models are typically learned from a
limited number of 3D scans and thus do not generalize well across different
identities and expressions. We present the first approach to learn complete 3D
models of face identity geometry, albedo and expression just from images and
videos. The virtually endless collection of such data, in combination with our
self-supervised learning-based approach allows for learning face models that
generalize beyond the span of existing approaches. Our network design and loss
functions ensure a disentangled parameterization of not only identity and
albedo, but also, for the first time, an expression basis. Our method also
allows for in-the-wild monocular reconstruction at test time. We show that our
learned models better generalize and lead to higher quality image-based
reconstructions than existing approaches.

The reflectance field of a face describes the reflectance properties
responsible for complex lighting effects including diffuse, specular,
inter-reflection and self shadowing. Most existing methods for estimating the
face reflectance from a monocular image assume faces to be diffuse with very
few approaches adding a specular component. This still leaves out important
perceptual aspects of reflectance as higher-order global illumination effects
and self-shadowing are not modeled. We present a new neural representation for
face reflectance where we can estimate all components of the reflectance
responsible for the final appearance from a single monocular image. Instead of
modeling each component of the reflectance separately using parametric models,
our neural representation allows us to generate a basis set of faces in a
geometric deformation-invariant space, parameterized by the input light
direction, viewpoint and face geometry. We learn to reconstruct this
reflectance field of a face just from a monocular image, which can be used to
render the face from any viewpoint in any light condition. Our method is
trained on a light-stage training dataset, which captures 300 people
illuminated with 150 light conditions from 8 viewpoints. We show that our
method outperforms existing monocular reflectance reconstruction methods, in
terms of photorealism due to better capturing of physical premitives, such as
sub-surface scattering, specularities, self-shadows and other higher-order
effects.

3D hand reconstruction from image data is a widely-studied problem in com-
puter vision and graphics, and has a particularly high relevance for virtual
and augmented reality. Although several 3D hand reconstruction approaches
leverage hand models as a strong prior to resolve ambiguities and achieve a
more robust reconstruction, most existing models account only for the hand
shape and poses and do not model the texture. To fill this gap, in this work
we present the first parametric texture model of human hands. Our model
spans several dimensions of hand appearance variability (e.g., related to gen-
der, ethnicity, or age) and only requires a commodity camera for data acqui-
sition. Experimentally, we demonstrate that our appearance model can be
used to tackle a range of challenging problems such as 3D hand reconstruc-
tion from a single monocular image. Furthermore, our appearance model
can be used to define a neural rendering layer that enables training with a
self-supervised photometric loss. We make our model publicly available.

Deep neural networks have been shown to be susceptible to adversarial
examples -- small, imperceptible changes constructed to cause
mis-classification in otherwise highly accurate image classifiers. As a
practical alternative, recent work proposed so-called adversarial patches:
clearly visible, but adversarially crafted rectangular patches in images. These
patches can easily be printed and applied in the physical world. While defenses
against imperceptible adversarial examples have been studied extensively,
robustness against adversarial patches is poorly understood. In this work, we
first devise a practical approach to obtain adversarial patches while actively
optimizing their location within the image. Then, we apply adversarial training
on these location-optimized adversarial patches and demonstrate significantly
improved robustness on CIFAR10 and GTSRB. Additionally, in contrast to
adversarial training on imperceptible adversarial examples, our adversarial
patch training does not reduce accuracy.

Marker-less 3D human motion capture from a single colour camera has seen
significant progress. However, it is a very challenging and severely ill-posed
problem. In consequence, even the most accurate state-of-the-art approaches
have significant limitations. Purely kinematic formulations on the basis of
individual joints or skeletons, and the frequent frame-wise reconstruction in
state-of-the-art methods greatly limit 3D accuracy and temporal stability
compared to multi-view or marker-based motion capture. Further, captured 3D
poses are often physically incorrect and biomechanically implausible, or
exhibit implausible environment interactions (floor penetration, foot skating,
unnatural body leaning and strong shifting in depth), which is problematic for
any use case in computer graphics. We, therefore, present PhysCap, the first
algorithm for physically plausible, real-time and marker-less human 3D motion
capture with a single colour camera at 25 fps. Our algorithm first captures 3D
human poses purely kinematically. To this end, a CNN infers 2D and 3D joint
positions, and subsequently, an inverse kinematics step finds space-time
coherent joint angles and global 3D pose. Next, these kinematic reconstructions
are used as constraints in a real-time physics-based pose optimiser that
accounts for environment constraints (e.g., collision handling and floor
placement), gravity, and biophysical plausibility of human postures. Our
approach employs a combination of ground reaction force and residual force for
plausible root control, and uses a trained neural network to detect foot
contact events in images. Our method captures physically plausible and
temporally stable global 3D human motion, without physically implausible
postures, floor penetrations or foot skating, from video in real time and in
general scenes. The video is available at
gvv.mpi-inf.mpg.de/projects/PhysCap

Efficient rendering of photo-realistic virtual worlds is a long standing
effort of computer graphics. Modern graphics techniques have succeeded in
synthesizing photo-realistic images from hand-crafted scene representations.
However, the automatic generation of shape, materials, lighting, and other
aspects of scenes remains a challenging problem that, if solved, would make
photo-realistic computer graphics more widely accessible. Concurrently,
progress in computer vision and machine learning have given rise to a new
approach to image synthesis and editing, namely deep generative models. Neural
rendering is a new and rapidly emerging field that combines generative machine
learning techniques with physical knowledge from computer graphics, e.g., by
the integration of differentiable rendering into network training. With a
plethora of applications in computer graphics and vision, neural rendering is
poised to become a new area in the graphics community, yet no survey of this
emerging field exists. This state-of-the-art report summarizes the recent
trends and applications of neural rendering. We focus on approaches that
combine classic computer graphics techniques with deep generative models to
obtain controllable and photo-realistic outputs. Starting with an overview of
the underlying computer graphics and machine learning concepts, we discuss
critical aspects of neural rendering approaches. This state-of-the-art report
is focused on the many important use cases for the described algorithms such as
novel view synthesis, semantic photo manipulation, facial and body reenactment,
relighting, free-viewpoint video, and the creation of photo-realistic avatars
for virtual and augmented reality telepresence. Finally, we conclude with a
discussion of the social implications of such technology and investigate open
research problems.

StyleGAN generates photorealistic portrait images of faces with eyes, teeth,
hair and context (neck, shoulders, background), but lacks a rig-like control
over semantic face parameters that are interpretable in 3D, such as face pose,
expressions, and scene illumination. Three-dimensional morphable face models
(3DMMs) on the other hand offer control over the semantic parameters, but lack
photorealism when rendered and only model the face interior, not other parts of
a portrait image (hair, mouth interior, background). We present the first
method to provide a face rig-like control over a pretrained and fixed StyleGAN
via a 3DMM. A new rigging network, RigNet is trained between the 3DMM's
semantic parameters and StyleGAN's input. The network is trained in a
self-supervised manner, without the need for manual annotations. At test time,
our method generates portrait images with the photorealism of StyleGAN and
provides explicit control over the 3D semantic parameters of the face.

Editing of portrait images is a very popular and important research topic
with a large variety of applications. For ease of use, control should be
provided via a semantically meaningful parameterization that is akin to
computer animation controls. The vast majority of existing techniques do not
provide such intuitive and fine-grained control, or only enable coarse editing
of a single isolated control parameter. Very recently, high-quality
semantically controlled editing has been demonstrated, however only on
synthetically created StyleGAN images. We present the first approach for
embedding real portrait images in the latent space of StyleGAN, which allows
for intuitive editing of the head pose, facial expression, and scene
illumination in the image. Semantic editing in parameter space is achieved
based on StyleRig, a pretrained neural network that maps the control space of a
3D morphable face model to the latent space of the GAN. We design a novel
hierarchical non-linear optimization problem to obtain the embedding. An
identity preservation energy term allows spatially coherent edits while
maintaining facial integrity. Our approach runs at interactive frame rates and
thus allows the user to explore the space of possible edits. We evaluate our
approach on a wide set of portrait photos, compare it to the current state of
the art, and validate the effectiveness of its components in an ablation study.

We present Face2Face, a novel approach for real-time facial reenactment of a
monocular target video sequence (e.g., Youtube video). The source sequence is
also a monocular video stream, captured live with a commodity webcam. Our goal
is to animate the facial expressions of the target video by a source actor and
re-render the manipulated output video in a photo-realistic fashion. To this
end, we first address the under-constrained problem of facial identity recovery
from monocular video by non-rigid model-based bundling. At run time, we track
facial expressions of both source and target video using a dense photometric
consistency measure. Reenactment is then achieved by fast and efficient
deformation transfer between source and target. The mouth interior that best
matches the re-targeted expression is retrieved from the target sequence and
warped to produce an accurate fit. Finally, we convincingly re-render the
synthesized target face on top of the corresponding video stream such that it
seamlessly blends with the real-world illumination. We demonstrate our method
in a live setup, where Youtube videos are reenacted in real time.

Implicit surface representations, such as signed-distance functions, combined
with deep learning have led to impressive models which can represent detailed
shapes of objects with arbitrary topology. Since a continuous function is
learned, the reconstructions can also be extracted at any arbitrary resolution.
However, large datasets such as ShapeNet are required to train such models. In
this paper, we present a new mid-level patch-based surface representation. At
the level of patches, objects across different categories share similarities,
which leads to more generalizable models. We then introduce a novel method to
learn this patch-based representation in a canonical space, such that it is as
object-agnostic as possible. We show that our representation trained on one
category of objects from ShapeNet can also well represent detailed shapes from
any other category. In addition, it can be trained using much fewer shapes,
compared to existing approaches. We show several applications of our new
representation, including shape interpolation and partial point cloud
completion. Due to explicit control over positions, orientations and scales of
patches, our representation is also more controllable compared to object-level
representations, which enables us to deform encoded shapes non-rigidly.

In this tech report, we present the current state of our ongoing work on
reconstructing Neural Radiance Fields (NERF) of general non-rigid scenes via
ray bending. Non-rigid NeRF (NR-NeRF) takes RGB images of a deforming object
(e.g., from a monocular video) as input and then learns a geometry and
appearance representation that not only allows to reconstruct the input
sequence but also to re-render any time step into novel camera views with high
fidelity. In particular, we show that a consumer-grade camera is sufficient to
synthesize convincing bullet-time videos of short and simple scenes. In
addition, the resulting representation enables correspondence estimation across
views and time, and provides rigidity scores for each point in the scene. We
urge the reader to watch the supplemental videos for qualitative results. We
will release our code.

We propose to use a model-based generative loss for training hand pose
estimators on depth images based on a volumetric hand model. This additional
loss allows training of a hand pose estimator that accurately infers the entire
set of 21 hand keypoints while only using supervision for 6 easy-to-annotate
keypoints (fingertips and wrist). We show that our partially-supervised method
achieves results that are comparable to those of fully-supervised methods which
enforce articulation consistency. Moreover, for the first time we demonstrate
that such an approach can be used to train on datasets that have erroneous
annotations, i.e. "ground truth" with notable measurement errors, while
obtaining predictions that explain the depth images better than the given
"ground truth".

Thin structures, such as wire-frame sculptures, fences, cables, power lines,
and tree branches, are common in the real world. It is extremely challenging to
acquire their 3D digital models using traditional image-based or depth-based
reconstruction methods because thin structures often lack distinct point
features and have severe self-occlusion. We propose the first approach that
simultaneously estimates camera motion and reconstructs the geometry of complex
3D thin structures in high quality from a color video captured by a handheld
camera. Specifically, we present a new curve-based approach to estimate
accurate camera poses by establishing correspondences between featureless thin
objects in the foreground in consecutive video frames, without requiring visual
texture in the background scene to lock on. Enabled by this effective
curve-based camera pose estimation strategy, we develop an iterative
optimization method with tailored measures on geometry, topology as well as
self-occlusion handling for reconstructing 3D thin structures. Extensive
validations on a variety of thin structures show that our method achieves
accurate camera pose estimation and faithful reconstruction of 3D thin
structures with complex shape and topology at a level that has not been
attained by other existing reconstruction methods.

Deep implicit field regression methods are effective for 3D reconstruction
from single-view images. However, the impact of different sampling patterns on
the reconstruction quality is not well-understood. In this work, we first study
the effect of point set discrepancy on the network training. Based on Farthest
Point Sampling algorithm, we propose a sampling scheme that theoretically
encourages better generalization performance, and results in fast convergence
for SGD-based optimization algorithms. Secondly, based on the reflective
symmetry of an object, we propose a feature fusion method that alleviates
issues due to self-occlusions which makes it difficult to utilize local image
features. Our proposed system Ladybird is able to create high quality 3D object
reconstructions from a single input image. We evaluate Ladybird on a large
scale 3D dataset (ShapeNet) demonstrating highly competitive results in terms
of Chamfer distance, Earth Mover's distance and Intersection Over Union (IoU).

We present the first deep implicit 3D morphable model (i3DMM) of full heads.
Unlike earlier morphable face models it not only captures identity-specific
geometry, texture, and expressions of the frontal face, but also models the
entire head, including hair. We collect a new dataset consisting of 64 people
with different expressions and hairstyles to train i3DMM. Our approach has the
following favorable properties: (i) It is the first full head morphable model
that includes hair. (ii) In contrast to mesh-based models it can be trained on
merely rigidly aligned scans, without requiring difficult non-rigid
registration. (iii) We design a novel architecture to decouple the shape model
into an implicit reference shape and a deformation of this reference shape.
With that, dense correspondences between shapes can be learned implicitly. (iv)
This architecture allows us to semantically disentangle the geometry and color
components, as color is learned in the reference space. Geometry is further
disentangled as identity, expressions, and hairstyle, while color is
disentangled as identity and hairstyle components. We show the merits of i3DMM
using ablation studies, comparisons to state-of-the-art models, and
applications such as semantic head editing and texture transfer. We will make
our model publicly available.

We present a new pose transfer method for synthesizing a human animation from
a single image of a person controlled by a sequence of body poses. Existing
pose transfer methods exhibit significant visual artifacts when applying to a
novel scene, resulting in temporal inconsistency and failures in preserving the
identity and textures of the person. To address these limitations, we design a
compositional neural network that predicts the silhouette, garment labels, and
textures. Each modular network is explicitly dedicated to a subtask that can be
learned from the synthetic data. At the inference time, we utilize the trained
network to produce a unified representation of appearance and its labels in UV
coordinates, which remains constant across poses. The unified representation
provides an incomplete yet strong guidance to generating the appearance in
response to the pose change. We use the trained network to complete the
appearance and render it with the background. With these strategies, we are
able to synthesize human animations that can preserve the identity and
appearance of the person in a temporally coherent way without any fine-tuning
of the network on the testing scene. Experiments show that our method
outperforms the state-of-the-arts in terms of synthesis quality, temporal
coherence, and generalization ability.

We present the first method for real-time full body capture that estimates
shape and motion of body and hands together with a dynamic 3D face model from a
single color image. Our approach uses a new neural network architecture that
exploits correlations between body and hands at high computational efficiency.
Unlike previous works, our approach is jointly trained on multiple datasets
focusing on hand, body or face separately, without requiring data where all the
parts are annotated at the same time, which is much more difficult to create at
sufficient variety. The possibility of such multi-dataset training enables
superior generalization ability. In contrast to earlier monocular full body
methods, our approach captures more expressive 3D face geometry and color by
estimating the shape, expression, albedo and illumination parameters of a
statistical face model. Our method achieves competitive accuracy on public
benchmarks, while being significantly faster and providing more complete face
reconstructions.

We present a novel method for monocular hand shape and pose estimation at
unprecedented runtime performance of 100fps and at state-of-the-art accuracy.
This is enabled by a new learning based architecture designed such that it can
make use of all the sources of available hand training data: image data with
either 2D or 3D annotations, as well as stand-alone 3D animations without
corresponding image data. It features a 3D hand joint detection module and an
inverse kinematics module which regresses not only 3D joint positions but also
maps them to joint rotations in a single feed-forward pass. This output makes
the method more directly usable for applications in computer vision and
graphics compared to only regressing 3D joint positions. We demonstrate that
our architectural design leads to a significant quantitative and qualitative
improvement over the state of the art on several challenging benchmarks. Our
model is publicly available for future research.

We present a simple yet effective method to infer detailed full human body
shape from only a single photograph. Our model can infer full-body shape
including face, hair, and clothing including wrinkles at interactive
frame-rates. Results feature details even on parts that are occluded in the
input image. Our main idea is to turn shape regression into an aligned
image-to-image translation problem. The input to our method is a partial
texture map of the visible region obtained from off-the-shelf methods. From a
partial texture, we estimate detailed normal and vector displacement maps,
which can be applied to a low-resolution smooth body model to add detail and
clothing. Despite being trained purely with synthetic data, our model
generalizes well to real-world photographs. Numerous results demonstrate the
versatility and robustness of our method.

We present a simple yet effective method to infer detailed full human body
shape from only a single photograph. Our model can infer full-body shape
including face, hair, and clothing including wrinkles at interactive
frame-rates. Results feature details even on parts that are occluded in the
input image. Our main idea is to turn shape regression into an aligned
image-to-image translation problem. The input to our method is a partial
texture map of the visible region obtained from off-the-shelf methods. From a
partial texture, we estimate detailed normal and vector displacement maps,
which can be applied to a low-resolution smooth body model to add detail and
clothing. Despite being trained purely with synthetic data, our model
generalizes well to real-world photographs. Numerous results demonstrate the
versatility and robustness of our method.

We present a learning-based model to infer the personalized 3D shape of
people from a few frames (1-8) of a monocular video in which the person is
moving, in less than 10 seconds with a reconstruction accuracy of 5mm. Our
model learns to predict the parameters of a statistical body model and instance
displacements that add clothing and hair to the shape. The model achieves fast
and accurate predictions based on two key design choices. First, by predicting
shape in a canonical T-pose space, the network learns to encode the images of
the person into pose-invariant latent codes, where the information is fused.
Second, based on the observation that feed-forward predictions are fast but do
not always align with the input images, we predict using both, bottom-up and
top-down streams (one per view) allowing information to flow in both
directions. Learning relies only on synthetic 3D data. Once learned, the model
can take a variable number of frames as input, and is able to reconstruct
shapes even from a single image with an accuracy of 6mm. Results on 3 different
datasets demonstrate the efficacy and accuracy of our approach.

We suggest representing light field (LF) videos as "one-off" neural networks
(NN), i.e., a learned mapping from view-plus-time coordinates to
high-resolution color values, trained on sparse views. Initially, this sounds
like a bad idea for three main reasons: First, a NN LF will likely have less
quality than a same-sized pixel basis representation. Second, only few training
data, e.g., 9 exemplars per frame are available for sparse LF videos. Third,
there is no generalization across LFs, but across view and time instead.
Consequently, a network needs to be trained for each LF video. Surprisingly,
these problems can turn into substantial advantages: Other than the linear
pixel basis, a NN has to come up with a compact, non-linear i.e., more
intelligent, explanation of color, conditioned on the sparse view and time
coordinates. As observed for many NN however, this representation now is
interpolatable: if the image output for sparse view coordinates is plausible,
it is for all intermediate, continuous coordinates as well. Our specific
network architecture involves a differentiable occlusion-aware warping step,
which leads to a compact set of trainable parameters and consequently fast
learning and fast execution.

We present Multi-Garment Network (MGN), a method to predict body shape and
clothing, layered on top of the SMPL model from a few frames (1-8) of a video.
Several experiments demonstrate that this representation allows higher level of
control when compared to single mesh or voxel representations of shape. Our
model allows to predict garment geometry, relate it to the body shape, and
transfer it to new body shapes and poses. To train MGN, we leverage a digital
wardrobe containing 712 digital garments in correspondence, obtained with a
novel method to register a set of clothing templates to a dataset of real 3D
scans of people in different clothing and poses. Garments from the digital
wardrobe, or predicted by MGN, can be used to dress any body shape in arbitrary
poses. We will make publicly available the digital wardrobe, the MGN model, and
code to dress SMPL with the garments.

Applying data-driven approaches to non-rigid 3D reconstruction has been
difficult, which we believe can be attributed to the lack of a large-scale
training corpus. One recent approach proposes self-supervision based on
non-rigid reconstruction. Unfortunately, this method fails for important cases
such as highly non-rigid deformations. We first address this problem of lack of
data by introducing a novel semi-supervised strategy to obtain dense
inter-frame correspondences from a sparse set of annotations. This way, we
obtain a large dataset of 400 scenes, over 390,000 RGB-D frames, and 2,537
densely aligned frame pairs; in addition, we provide a test set along with
several metrics for evaluation. Based on this corpus, we introduce a
data-driven non-rigid feature matching approach, which we integrate into an
optimization-based reconstruction pipeline. Here, we propose a new neural
network that operates on RGB-D frames, while maintaining robustness under large
non-rigid deformations and producing accurate predictions. Our approach
significantly outperforms both existing non-rigid reconstruction methods that
do not use learned data terms, as well as learning-based approaches that only
use self-supervision.

In this paper, we provide a detailed survey of 3D Morphable Face Models over
the 20 years since they were first proposed. The challenges in building and
applying these models, namely capture, modeling, image formation, and image
analysis, are still active research topics, and we review the state-of-the-art
in each of these areas. We also look ahead, identifying unsolved challenges,
proposing directions for future research and highlighting the broad range of
current and future applications.

Face performance capture and reenactment techniques use multiple cameras and
sensors, positioned at a distance from the face or mounted on heavy wearable
devices. This limits their applications in mobile and outdoor environments. We
present EgoFace, a radically new lightweight setup for face performance capture
and front-view videorealistic reenactment using a single egocentric RGB camera.
Our lightweight setup allows operations in uncontrolled environments, and lends
itself to telepresence applications such as video-conferencing from dynamic
environments. The input image is projected into a low dimensional latent space
of the facial expression parameters. Through careful adversarial training of
the parameter-space synthetic rendering, a videorealistic animation is
produced. Our problem is challenging as the human visual system is sensitive to
the smallest face irregularities that could occur in the final results. This
sensitivity is even stronger for video results. Our solution is trained in a
pre-processing stage, through a supervised manner without manual annotations.
EgoFace captures a wide variety of facial expressions, including mouth
movements and asymmetrical expressions. It works under varying illuminations,
background, movements, handles people from different ethnicities and can
operate in real time.

Editing talking-head video to change the speech content or to remove filler
words is challenging. We propose a novel method to edit talking-head video
based on its transcript to produce a realistic output video in which the
dialogue of the speaker has been modified, while maintaining a seamless
audio-visual flow (i.e. no jump cuts). Our method automatically annotates an
input talking-head video with phonemes, visemes, 3D face pose and geometry,
reflectance, expression and scene illumination per frame. To edit a video, the
user has to only edit the transcript, and an optimization strategy then chooses
segments of the input corpus as base material. The annotated parameters
corresponding to the selected segments are seamlessly stitched together and
used to produce an intermediate video representation in which the lower half of
the face is rendered with a parametric face model. Finally, a recurrent video
generation network transforms this representation to a photorealistic video
that matches the edited transcript. We demonstrate a large variety of edits,
such as the addition, removal, and alteration of words, as well as convincing
language translation and full sentence synthesis.

Modern adiabatic quantum computers (AQC) are already used to solve difficult
combinatorial optimisation problems in various domains of science. Currently,
only a few applications of AQC in computer vision have been demonstrated. We
review modern AQC and derive the first algorithm for transformation estimation
and point set alignment suitable for AQC. Our algorithm has a subquadratic
computational complexity of state preparation. We perform a systematic
experimental analysis of the proposed approach and show several examples of
successful point set alignment by simulated sampling. With this paper, we hope
to boost the research on AQC for computer vision.

While dense non-rigid structure from motion (NRSfM) has been extensively
studied from the perspective of the reconstructability problem over the recent
years, almost no attempts have been undertaken to bring it into the practical
realm. The reasons for the slow dissemination are the severe ill-posedness,
high sensitivity to motion and deformation cues and the difficulty to obtain
reliable point tracks in the vast majority of practical scenarios. To fill this
gap, we propose a hybrid approach that extracts prior shape knowledge from an
input sequence with NRSfM and uses it as a dynamic shape prior for sequential
surface recovery in scenarios with recurrence. Our Dynamic Shape Prior
Reconstruction (DSPR) method can be combined with existing dense NRSfM
techniques while its energy functional is optimised with stochastic gradient
descent at real-time rates for new incoming point tracks. The proposed
versatile framework with a new core NRSfM approach outperforms several other
methods in the ability to handle inaccurate and noisy point tracks, provided we
have access to a representative (in terms of the deformation variety) image
sequence. Comprehensive experiments highlight convergence properties and the
accuracy of DSPR under different disturbing effects. We also perform a joint
study of tracking and reconstruction and show applications to shape compression
and heart reconstruction under occlusions. We achieve state-of-the-art metrics
(accuracy and compression ratios) in different scenarios.

Convolutional Neural Network based approaches for monocular 3D human pose
estimation usually require a large amount of training images with 3D pose
annotations. While it is feasible to provide 2D joint annotations for large
corpora of in-the-wild images with humans, providing accurate 3D annotations to
such in-the-wild corpora is hardly feasible in practice. Most existing 3D
labelled data sets are either synthetically created or feature in-studio
images. 3D pose estimation algorithms trained on such data often have limited
ability to generalize to real world scene diversity. We therefore propose a new
deep learning based method for monocular 3D human pose estimation that shows
high accuracy and generalizes better to in-the-wild scenes. It has a network
architecture that comprises a new disentangled hidden space encoding of
explicit 2D and 3D features, and uses supervision by a new learned projection
model from predicted 3D pose. Our algorithm can be jointly trained on image
data with 3D labels and image data with only 2D labels. It achieves
state-of-the-art accuracy on challenging in-the-wild data.

Trusses are load-carrying light-weight structures consisting of bars
connected at joints ubiquitously applied in a variety of engineering scenarios.
Designing optimal trusses that satisfy functional specifications with a minimal
amount of material has interested both theoreticians and practitioners for more
than a century. In this paper, we introduce two main ideas to improve upon the
state of the art. First, we formulate an alternating linear programming problem
for geometry optimization. Second, we introduce two sets of complementary
topological operations, including a novel subdivision scheme for global
topology refinement inspired by Michell's famed theoretical study. Based on
these two ideas, we build an efficient computational framework for the design
of lightweight trusses. \AD{We illustrate our framework with a variety of
functional specifications and extensions. We show that our method achieves
trusses with smaller volumes and is over two orders of magnitude faster
compared with recent state-of-the-art approaches.