Human performance capture is a highly important computer vision problem with<br>many applications in movie production and virtual/augmented reality. Many<br>previous performance capture approaches either required expensive multi-view<br>setups or did not recover dense space-time coherent geometry with<br>frame-to-frame correspondences. We propose a novel deep learning approach for<br>monocular dense human performance capture. Our method is trained in a weakly<br>supervised manner based on multi-view supervision completely removing the need<br>for training data with 3D ground truth annotations. The network architecture is<br>based on two separate networks that disentangle the task into a pose estimation<br>and a non-rigid surface deformation step. Extensive qualitative and<br>quantitative evaluations show that our approach outperforms the state of the<br>art in terms of quality and robustness. This work is an extended version of<br>DeepCap where we provide more detailed explanations, comparisons and results as<br>well as applications.<br>

Deep neural network (DNN) accelerators received considerable attention in<br>past years due to saved energy compared to mainstream hardware. Low-voltage<br>operation of DNN accelerators allows to further reduce energy consumption<br>significantly, however, causes bit-level failures in the memory storing the<br>quantized DNN weights. In this paper, we show that a combination of robust<br>fixed-point quantization, weight clipping, and random bit error training<br>(RandBET) improves robustness against random bit errors in (quantized) DNN<br>weights significantly. This leads to high energy savings from both low-voltage<br>operation as well as low-precision quantization. Our approach generalizes<br>across operating voltages and accelerators, as demonstrated on bit errors from<br>profiled SRAM arrays. We also discuss why weight clipping alone is already a<br>quite effective way to achieve robustness against bit errors. Moreover, we<br>specifically discuss the involved trade-offs regarding accuracy, robustness and<br>precision: Without losing more than 1% in accuracy compared to a normally<br>trained 8-bit DNN, we can reduce energy consumption on CIFAR-10 by 20%. Higher<br>energy savings of, e.g., 30%, are possible at the cost of 2.5% accuracy, even<br>for 4-bit DNNs.<br>

We present a massively parallel Lagrange decomposition method for solving 0-1<br>integer linear programs occurring in structured prediction. We propose a new<br>iterative update scheme for solving the Lagrangean dual and a perturbation<br>technique for decoding primal solutions. For representing subproblems we follow<br>Lange et al. (2021) and use binary decision diagrams (BDDs). Our primal and<br>dual algorithms require little synchronization between subproblems and<br>optimization over BDDs needs only elementary operations without complicated<br>control flow. This allows us to exploit the parallelism offered by GPUs for all<br>components of our method. We present experimental results on combinatorial<br>problems from MAP inference for Markov Random Fields, quadratic assignment and<br>cell tracking for developmental biology. Our highly parallel GPU implementation<br>improves upon the running times of the algorithms from Lange et al. (2021) by<br>up to an order of magnitude. In particular, we come close to or outperform some<br>state-of-the-art specialized heuristics while being problem agnostic.<br>

We introduce a new family of neural network models called Convolutional<br>Dynamic Alignment Networks (CoDA Nets), which are performant classifiers with a<br>high degree of inherent interpretability. Their core building blocks are<br>Dynamic Alignment Units (DAUs), which are optimised to transform their inputs<br>with dynamically computed weight vectors that align with task-relevant<br>patterns. As a result, CoDA Nets model the classification prediction through a<br>series of input-dependent linear transformations, allowing for linear<br>decomposition of the output into individual input contributions. Given the<br>alignment of the DAUs, the resulting contribution maps align with<br>discriminative input patterns. These model-inherent decompositions are of high<br>visual quality and outperform existing attribution methods under quantitative<br>metrics. Further, CoDA Nets constitute performant classifiers, achieving on par<br>results to ResNet and VGG models on e.g. CIFAR-10 and TinyImagenet. Lastly,<br>CoDA Nets can be combined with conventional neural network models to yield<br>powerful classifiers that more easily scale to complex datasets such as<br>Imagenet whilst exhibiting an increased interpretable depth, i.e., the output<br>can be explained well in terms of contributions from intermediate layers within<br>the network.<br>

Explaining the decision of a multi-modal decision-maker requires to determine<br>the evidence from both modalities. Recent advances in XAI provide explanations<br>for models trained on still images. However, when it comes to modeling multiple<br>sensory modalities in a dynamic world, it remains underexplored how to<br>demystify the mysterious dynamics of a complex multi-modal model. In this work,<br>we take a crucial step forward and explore learnable explanations for<br>audio-visual recognition. Specifically, we propose a novel space-time attention<br>network that uncovers the synergistic dynamics of audio and visual data over<br>both space and time. Our model is capable of predicting the audio-visual video<br>events, while justifying its decision by localizing where the relevant visual<br>cues appear, and when the predicted sounds occur in videos. We benchmark our<br>model on three audio-visual video event datasets, comparing extensively to<br>multiple recent multi-modal representation learners and intrinsic explanation<br>models. Experimental results demonstrate the clear superior performance of our<br>model over the existing methods on audio-visual video event recognition.<br>Moreover, we conduct an in-depth study to analyze the explainability of our<br>model based on robustness analysis via perturbation tests and pointing games<br>using human annotations.<br>

Text and images are the two most common data modalities found on the Internet. Understanding the synergy between text and images, that is, seamlessly analyzing information from these modalities may be trivial for humans, but is challenging for software systems. In this dissertation we study problems where deciphering text-image synergy is crucial for finding solutions. We propose methods and ideas that establish semantic connections between text and images in multimodal contents, and empirically show their effectiveness in four interconnected problems: Image Retrieval, Image Tag Refinement, Image-Text Alignment, and Image Captioning. Our promising results and observations open up interesting scopes for future research involving text-image data understanding.Text and images are the two most common data modalities found on the Internet. Understanding the synergy between text and images, that is, seamlessly analyzing information from these modalities may be trivial for humans, but is challenging for software systems. In this dissertation we study problems where deciphering text-image synergy is crucial for finding solutions. We propose methods and ideas that establish semantic connections between text and images in multimodal contents, and empirically show their effectiveness in four interconnected problems: Image Retrieval, Image Tag Refinement, Image-Text Alignment, and Image Captioning. Our promising results and observations open up interesting scopes for future research involving text-image data understanding.

Traditional domain adaptation addresses the task of adapting a model to a<br>novel target domain under limited or no additional supervision. While tackling<br>the input domain gap, the standard domain adaptation settings assume no domain<br>change in the output space. In semantic prediction tasks, different datasets<br>are often labeled according to different semantic taxonomies. In many<br>real-world settings, the target domain task requires a different taxonomy than<br>the one imposed by the source domain. We therefore introduce the more general<br>taxonomy adaptive domain adaptation (TADA) problem, allowing for inconsistent<br>taxonomies between the two domains. We further propose an approach that jointly<br>addresses the image-level and label-level domain adaptation. On the<br>label-level, we employ a bilateral mixed sampling strategy to augment the<br>target domain, and a relabelling method to unify and align the label spaces. We<br>address the image-level domain gap by proposing an uncertainty-rectified<br>contrastive learning method, leading to more domain-invariant and class<br>discriminative features. We extensively evaluate the effectiveness of our<br>framework under different TADA settings: open taxonomy, coarse-to-fine<br>taxonomy, and partially-overlapping taxonomy. Our framework outperforms<br>previous state-of-the-art by a large margin, while capable of adapting to<br>target taxonomies.<br>

Training deep networks for semantic segmentation requires large amounts of<br>labeled training data, which presents a major challenge in practice, as<br>labeling segmentation masks is a highly labor-intensive process. To address<br>this issue, we present a framework for semi-supervised and domain-adaptive<br>semantic segmentation, which is enhanced by self-supervised monocular depth<br>estimation (SDE) trained only on unlabeled image sequences.<br> In particular, we utilize SDE as an auxiliary task comprehensively across the<br>entire learning framework: First, we automatically select the most useful<br>samples to be annotated for semantic segmentation based on the correlation of<br>sample diversity and difficulty between SDE and semantic segmentation. Second,<br>we implement a strong data augmentation by mixing images and labels using the<br>geometry of the scene. Third, we transfer knowledge from features learned<br>during SDE to semantic segmentation by means of transfer and multi-task<br>learning. And fourth, we exploit additional labeled synthetic data with<br>Cross-Domain DepthMix and Matching Geometry Sampling to align synthetic and<br>real data.<br> We validate the proposed model on the Cityscapes dataset, where all four<br>contributions demonstrate significant performance gains, and achieve<br>state-of-the-art results for semi-supervised semantic segmentation as well as<br>for semi-supervised domain adaptation. In particular, with only 1/30 of the<br>Cityscapes labels, our method achieves 92% of the fully-supervised baseline<br>performance and even 97% when exploiting additional data from GTA. The source<br>code is available at<br>https://github.com/lhoyer/improving_segmentation_with_selfsupervised_depth.<br>

Compositional Zero-Shot learning (CZSL) aims to recognize unseen compositions<br>of state and object visual primitives seen during training. A problem with<br>standard CZSL is the assumption of knowing which unseen compositions will be<br>available at test time. In this work, we overcome this assumption operating on<br>the open world setting, where no limit is imposed on the compositional space at<br>test time, and the search space contains a large number of unseen compositions.<br>To address this problem, we propose a new approach, Compositional Cosine Graph<br>Embeddings (Co-CGE), based on two principles. First, Co-CGE models the<br>dependency between states, objects and their compositions through a graph<br>convolutional neural network. The graph propagates information from seen to<br>unseen concepts, improving their representations. Second, since not all unseen<br>compositions are equally feasible, and less feasible ones may damage the<br>learned representations, Co-CGE estimates a feasibility score for each unseen<br>composition, using the scores as margins in a cosine similarity-based loss and<br>as weights in the adjacency matrix of the graphs. Experiments show that our<br>approach achieves state-of-the-art performances in standard CZSL while<br>outperforming previous methods in the open world scenario.<br>

Abstract<br>Humans are at the centre of a significant amount of research in computer vision.<br>Endowing machines with the ability to perceive people from visual data is an immense<br>scientific challenge with a high degree of direct practical relevance. Success in automatic<br>perception can be measured at different levels of abstraction, and this will depend on<br>which intelligent behaviour we are trying to replicate: the ability to localise persons in<br>an image or in the environment, understanding how persons are moving at the skeleton<br>and at the surface level, interpreting their interactions with the environment including<br>with other people, and perhaps even anticipating future actions. In this thesis we tackle<br>different sub-problems of the broad research area referred to as "looking at people",<br>aiming to perceive humans in images at different levels of granularity.<br>We start with bounding box-level pedestrian detection: We present a retrospective<br>analysis of methods published in the decade preceding our work, identifying various<br>strands of research that have advanced the state of the art. With quantitative exper-<br>iments, we demonstrate the critical role of developing better feature representations<br>and having the right training distribution. We then contribute two methods based<br>on the insights derived from our analysis: one that combines the strongest aspects of<br>past detectors and another that focuses purely on learning representations. The latter<br>method outperforms more complicated approaches, especially those based on hand-<br>crafted features. We conclude our work on pedestrian detection with a forward-looking<br>analysis that maps out potential avenues for future research.<br>We then turn to pixel-level methods: Perceiving humans requires us to both separate<br>them precisely from the background and identify their surroundings. To this end, we<br>introduce Cityscapes, a large-scale dataset for street scene understanding. This has since<br>established itself as a go-to benchmark for segmentation and detection. We additionally<br>develop methods that relax the requirement for expensive pixel-level annotations, focusing<br>on the task of boundary detection, i.e. identifying the outlines of relevant objects and<br>surfaces. Next, we make the jump from pixels to 3D surfaces, from localising and<br>labelling to fine-grained spatial understanding. We contribute a method for recovering<br>3D human shape and pose, which marries the advantages of learning-based and model-<br>based approaches.<br>We conclude the thesis with a detailed discussion of benchmarking practices in<br>computer vision. Among other things, we argue that the design of future datasets<br>should be driven by the general goal of combinatorial robustness besides task-specific<br>considerations.

Deep neural network (DNN) accelerators received considerable attention in<br>recent years due to the potential to save energy compared to mainstream<br>hardware. Low-voltage operation of DNN accelerators allows to further reduce<br>energy consumption significantly, however, causes bit-level failures in the<br>memory storing the quantized DNN weights. Furthermore, DNN accelerators have<br>been shown to be vulnerable to adversarial attacks on voltage controllers or<br>individual bits. In this paper, we show that a combination of robust<br>fixed-point quantization, weight clipping, as well as random bit error training<br>(RandBET) or adversarial bit error training (AdvBET) improves robustness<br>against random or adversarial bit errors in quantized DNN weights<br>significantly. This leads not only to high energy savings for low-voltage<br>operation as well as low-precision quantization, but also improves security of<br>DNN accelerators. Our approach generalizes across operating voltages and<br>accelerators, as demonstrated on bit errors from profiled SRAM arrays, and<br>achieves robustness against both targeted and untargeted bit-level attacks.<br>Without losing more than 0.8%/2% in test accuracy, we can reduce energy<br>consumption on CIFAR10 by 20%/30% for 8/4-bit quantization using RandBET.<br>Allowing up to 320 adversarial bit errors, AdvBET reduces test error from above<br>90% (chance level) to 26.22% on CIFAR10.<br>

Most learning methods for 3D data (point clouds, meshes) suffer significant<br>performance drops when the data is not carefully aligned to a canonical<br>orientation. Aligning real world 3D data collected from different sources is<br>non-trivial and requires manual intervention. In this paper, we propose the<br>Adjoint Rigid Transform (ART) Network, a neural module which can be integrated<br>with a variety of 3D networks to significantly boost their performance. ART<br>learns to rotate input shapes to a learned canonical orientation, which is<br>crucial for a lot of tasks such as shape reconstruction, interpolation,<br>non-rigid registration, and latent disentanglement. ART achieves this with<br>self-supervision and a rotation equivariance constraint on predicted rotations.<br>The remarkable result is that with only self-supervision, ART facilitates<br>learning a unique canonical orientation for both rigid and nonrigid shapes,<br>which leads to a notable boost in performance of aforementioned tasks. We will<br>release our code and pre-trained models for further research.<br>

Modern approaches to pose and body shape estimation have recently achieved<br>strong performance even under challenging real-world conditions. Even from a<br>single image of a clothed person, a realistic looking body shape can be<br>inferred that captures a users' weight group and body shape type well. This<br>opens up a whole spectrum of applications -- in particular in fashion -- where<br>virtual try-on and recommendation systems can make use of these new and<br>automatized cues. However, a realistic depiction of the undressed body is<br>regarded highly private and therefore might not be consented by most people.<br>Hence, we ask if the automatic extraction of such information can be<br>effectively evaded. While adversarial perturbations have been shown to be<br>effective for manipulating the output of machine learning models -- in<br>particular, end-to-end deep learning approaches -- state of the art shape<br>estimation methods are composed of multiple stages. We perform the first<br>investigation of different strategies that can be used to effectively<br>manipulate the automatic shape estimation while preserving the overall<br>appearance of the original image.<br>