Publications - The Year Before Last

In computational design and fabrication, neural networks are becoming
important surrogates for bulky forward simulations. A long-standing,
intertwined question is that of inverse design: how to compute a design that
satisfies a desired target performance? Here, we show that the piecewise linear
property, very common in everyday neural networks, allows for an inverse design
formulation based on mixed-integer linear programming. Our mixed-integer
inverse design uncovers globally optimal or near optimal solutions in a
principled manner. Furthermore, our method significantly facilitates emerging,
but challenging, combinatorial inverse design tasks, such as material
selection. For problems where finding the optimal solution is not desirable or
tractable, we develop an efficient yet near-optimal hybrid optimization.
Eventually, our method is able to find solutions provably robust to possible
fabrication perturbations among multiple designs with similar performances.

We propose the first approach to automatically and jointly synthesize both
the synchronous 3D conversational body and hand gestures, as well as 3D face
and head animations, of a virtual character from speech input. Our algorithm
uses a CNN architecture that leverages the inherent correlation between facial
expression and hand gestures. Synthesis of conversational body gestures is a
multi-modal problem since many similar gestures can plausibly accompany the
same input speech. To synthesize plausible body gestures in this setting, we
train a Generative Adversarial Network (GAN) based model that measures the
plausibility of the generated sequences of 3D body motion when paired with the
input audio features. We also contribute a new way to create a large corpus of
more than 33 hours of annotated body, hand, and face data from in-the-wild
videos of talking people. To this end, we apply state-of-the-art monocular
approaches for 3D body and hand pose estimation as well as dense 3D face
performance capture to the video corpus. In this way, we can train on orders of
magnitude more data than previous algorithms that resort to complex in-studio
motion capture solutions, and thereby train more expressive synthesis
algorithms. Our experiments and user study show the state-of-the-art quality of
our speech-synthesized full 3D character animations.

Virtual reality has the potential to change the way we create and consume
content in our everyday life. Entertainment, training, design and
manufacturing, communication, or advertising are all applications that already
benefit from this new medium reaching consumer level. VR is inherently
different from traditional media: it offers a more immersive experience, and
has the ability to elicit a sense of presence through the place and
plausibility illusions. It also gives the user unprecedented capabilities to
explore their environment, in contrast with traditional media. In VR, like in
the real world, users integrate the multimodal sensory information they receive
to create a unified perception of the virtual world. Therefore, the sensory
cues that are available in a virtual environment can be leveraged to enhance
the final experience. This may include increasing realism, or the sense of
presence; predicting or guiding the attention of the user through the
experience; or increasing their performance if the experience involves the
completion of certain tasks. In this state-of-the-art report, we survey the
body of work addressing multimodality in virtual reality, its role and benefits
in the final user experience. The works here reviewed thus encompass several
fields of research, including computer graphics, human computer interaction, or
psychology and perception. Additionally, we give an overview of different
applications that leverage multimodal input in areas such as medicine, training
and education, or entertainment; we include works in which the integration of
multiple sensory information yields significant improvements, demonstrating how
multimodality can play a fundamental role in the way VR systems are designed,
and VR experiences created and consumed.

Human re-rendering from a single image is a starkly under-constrained
problem, and state-of-the-art algorithms often exhibit undesired artefacts,
such as over-smoothing, unrealistic distortions of the body parts and garments,
or implausible changes of the texture. To address these challenges, we propose
a new method for neural re-rendering of a human under a novel user-defined pose
and viewpoint, given one input image. Our algorithm represents body pose and
shape as a parametric mesh which can be reconstructed from a single image and
easily reposed. Instead of a colour-based UV texture map, our approach further
employs a learned high-dimensional UV feature map to encode appearance. This
rich implicit representation captures detailed appearance variation across
poses, viewpoints, person identities and clothing styles better than learned
colour texture maps. The body model with the rendered feature maps is fed
through a neural image-translation network that creates the final rendered
colour image. The above components are combined in an end-to-end-trained neural
network architecture that takes as input a source person image, and images of
the parametric body model in the source pose and desired target pose.
Experimental evaluation demonstrates that our approach produces higher quality
single image re-rendering results than existing methods.

Foveated image reconstruction recovers full image from a sparse set of
samples distributed according to the human visual system's retinal sensitivity
that rapidly drops with eccentricity. Recently, the use of Generative
Adversarial Networks was shown to be a promising solution for such a task as
they can successfully hallucinate missing image information. Like for other
supervised learning approaches, also for this one, the definition of the loss
function and training strategy heavily influences the output quality. In this
work, we pose the question of how to efficiently guide the training of foveated
reconstruction techniques such that they are fully aware of the human visual
system's capabilities and limitations, and therefore, reconstruct visually
important image features. Due to the nature of GAN-based solutions, we
concentrate on the human's sensitivity to hallucination for different input
sample densities. We present new psychophysical experiments, a dataset, and a
procedure for training foveated image reconstruction. The strategy provides
flexibility to the generator network by penalizing only perceptually important
deviations in the output. As a result, the method aims to preserve perceived
image statistics rather than natural image statistics. We evaluate our strategy
and compare it to alternative solutions using a newly trained objective metric
and user experiments.

We propose Blue Noise Plots, two-dimensional dot plots that depict data
points of univariate data sets. While often one-dimensional strip plots are
used to depict such data, one of their main problems is visual clutter which
results from overlap. To reduce this overlap, jitter plots were introduced,
whereby an additional, non-encoding plot dimension is introduced, along which
the data point representing dots are randomly perturbed. Unfortunately, this
randomness can suggest non-existent clusters, and often leads to visually
unappealing plots, in which overlap might still occur. To overcome these
shortcomings, we introduce BlueNoise Plots where random jitter along the
non-encoding plot dimension is replaced by optimizing all dots to keep a
minimum distance in 2D i. e., Blue Noise. We evaluate the effectiveness as well
as the aesthetics of Blue Noise Plots through both, a quantitative and a
qualitative user study.

High Dynamic Range (HDR) content is becoming ubiquitous due to the rapid
development of capture technologies. Nevertheless, the dynamic range of common
display devices is still limited, therefore tone mapping (TM) remains a key
challenge for image visualization. Recent work has demonstrated that neural
networks can achieve remarkable performance in this task when compared to
traditional methods, however, the quality of the results of these
learning-based methods is limited by the training data. Most existing works use
as training set a curated selection of best-performing results from existing
traditional tone mapping operators (often guided by a quality metric),
therefore, the quality of newly generated results is fundamentally limited by
the performance of such operators. This quality might be even further limited
by the pool of HDR content that is used for training. In this work we propose a
learning-based self-supervised tone mapping operator that is trained at test
time specifically for each HDR image and does not need any data labeling. The
key novelty of our approach is a carefully designed loss function built upon
fundamental knowledge on contrast perception that allows for directly comparing
the content in the HDR and tone mapped images. We achieve this goal by
reformulating classic VGG feature maps into feature contrast maps that
normalize local feature differences by their average magnitude in a local
neighborhood, allowing our loss to account for contrast masking effects. We
perform extensive ablation studies and exploration of parameters and
demonstrate that our solution outperforms existing approaches with a single set
of fixed parameters, as confirmed by both objective and subjective metrics.

The humble loop shrinking property played a central role in the inception of
modern topology but it has been eclipsed by more abstract algebraic formalism.
This is particularly true in the context of detecting relevant non-contractible
loops on surfaces where elaborate homological and/or graph theoretical
constructs are favored in algorithmic solutions. In this work, we devise a
variational analogy to the loop shrinking property and show that it yields a
simple, intuitive, yet powerful solution allowing a streamlined treatment of
the problem of handle and tunnel loop detection. Our formalization tracks the
evolution of a diffusion front randomly initiated on a single location on the
surface. Capitalizing on a diffuse interface representation combined with a set
of rules for concurrent front interactions, we develop a dynamic data structure
for tracking the evolution on the surface encoded as a sparse matrix which
serves for performing both diffusion numerics and loop detection and acts as
the workhorse of our fully parallel implementation. The substantiated results
suggest our approach outperforms state of the art and robustly copes with
highly detailed geometric models. As a byproduct, our approach can be used to
construct Reeb graphs by diffusion thus avoiding commonly encountered issues
when using Morse functions.

Most of the images one finds in the media, such as on the Internet or in textbooks and magazines, contain humans as the main point of attention. Thus, there is an inherent necessity for industry, society, and private persons to be able to thoroughly analyze and synthesize the human-related content in these images. One aspect of this analysis and subject of this thesis is to infer the 3D pose and surface deformation, using only visual information, which is also known as human performance capture. Human performance capture enables the tracking of virtual characters from real-world observations, and this is key for visual effects, games, VR, and AR, to name just a few application areas. However, traditional capture methods usually rely on expensive multi-view (marker-based) systems that are prohibitively expensive for the vast majority of people, or they use depth sensors, which are still not as common as single color cameras. Recently, some approaches have attempted to solve the task by assuming only a single RGB image is given. Nonetheless, they can either not track the dense deforming geometry of the human, such as the clothing layers, or they are far from real time, which is indispensable for many applications. To overcome these shortcomings, this thesis proposes two monocular human performance capture methods, which for the first time allow the real-time capture of the dense deforming geometry as well as an unseen 3D accuracy for pose and surface deformations. At the technical core, this work introduces novel GPU-based and data-parallel optimization strategies in conjunction with other algorithmic design choices that are all geared towards real-time performance at high accuracy. Moreover, this thesis presents a new weakly supervised multiview training strategy combined with a fully differentiable character representation that shows superior 3D accuracy. However, there is more to human-related Computer Vision than only the analysis of people in images. It is equally important to synthesize new images of humans in unseen poses and also from camera viewpoints that have not been observed in the real world. Such tools are essential for the movie industry because they, for example, allow the synthesis of photo-realistic virtual worlds with real-looking humans or of contents that are too dangerous for actors to perform on set. But also video conferencing and telepresence applications can benefit from photo-real 3D characters, as they can enhance the immersive experience of these applications. Here, the traditional Computer Graphics pipeline for rendering photo-realistic images involves many tedious and time-consuming steps that require expert knowledge and are far from real time. Traditional rendering involves character rigging and skinning, the modeling of the surface appearance properties, and physically based ray tracing. Recent learning-based methods attempt to simplify the traditional rendering pipeline and instead learn the rendering function from data resulting in methods that are easier accessible to non-experts. However, most of them model the synthesis task entirely in image space such that 3D consistency cannot be achieved, and/or they fail to model motion- and view-dependent appearance effects. To this end, this thesis presents a method and ongoing work on character synthesis, which allow the synthesis of controllable photoreal characters that achieve motion- and view-dependent appearance effects as well as 3D consistency and which run in real time. This is technically achieved by a novel coarse-to-fine geometric character representation for efficient synthesis, which can be solely supervised on multi-view imagery. Furthermore, this work shows how such a geometric representation can be combined with an implicit surface representation to boost synthesis and geometric quality.