Deep learning models have proven to be successful in a wide <br>range of machine learning tasks. Yet, they are often highly sensitive to <br>perturbations on the input data which can lead to incorrect decisions <br>with high confidence, hampering their deployment for practical <br>use-cases. Thus, finding architectures that are (more) robust against <br>perturbations has received much attention in recent years. Just like the <br>search for well-performing architectures in terms of clean accuracy, <br>this usually involves a tedious trial-and-error process with one <br>additional challenge: the evaluation of a network's robustness is <br>significantly more expensive than its evaluation for clean accuracy. <br>Thus, the aim of this paper is to facilitate better streamlined research <br>on architectural design choices with respect to their impact on <br>robustness as well as, for example, the evaluation of surrogate measures <br>for robustness. We therefore borrow one of the most commonly considered <br>search spaces for neural architecture search for image classification, <br>NAS-Bench-201, which contains a manageable size of 6466 non-isomorphic <br>network designs. We evaluate all these networks on a range of common <br>adversarial attacks and corruption types and introduce a database on <br>neural architecture design and robustness evaluations. We further <br>present three exemplary use cases of this dataset, in which we (i) <br>benchmark robustness measurements based on Jacobian and Hessian matrices <br>for their robustness predictability, (ii) perform neural architecture <br>search on robust accuracies, and (iii) provide an initial analysis of <br>how architectural design choices affect robustness. We find that <br>carefully crafting the topology of a network can have substantial impact <br>on its robustness, where networks with the same parameter count range in <br>mean adversarial robust accuracy from 20%-41%.

Minimum cost lifted multicut problem is a generalization of the multicut problem and is a means to optimizing a decomposition of a graph w.r.t. both positive and negative edge costs. Its main advantage is that multicut-based formulations do not require the number of components given a priori; instead, it is deduced from the solution. However, the standard multicut cost function is limited to pairwise relationships between nodes, while several important applications either require or can benefit from a higher-order cost function, i.e. hyper-edges. In this paper, we propose a pseudo-boolean formulation for a multiple model fitting problem. It is based on a formulation of any-order minimum cost lifted multicuts, which allows to partition an undirected graph with pairwise connectivity such as to minimize costs defined over any set of hyper-edges. As the proposed formulation is NP-hard and the branch-and-bound algorithm is too slow in practice, we propose an efficient local search algorithm for inference into resulting problems. We demonstrate versatility and effectiveness of our approach in several applications: geometric multiple model fitting, homography and motion estimation, motion segmentation.

Zero-shot semantic segmentation (ZS3) aims to segment the novel categories<br>that have not been seen in the training. Existing works formulate ZS3 as a<br>pixel-level zero-shot classification problem, and transfer semantic knowledge<br>from seen classes to unseen ones with the help of language models pre-trained<br>only with texts. While simple, the pixel-level ZS3 formulation shows the<br>limited capability to integrate vision-language models that are often<br>pre-trained with image-text pairs and currently demonstrate great potential for<br>vision tasks. Inspired by the observation that humans often perform<br>segment-level semantic labeling, we propose to decouple the ZS3 into two<br>sub-tasks: 1) a class-agnostic grouping task to group the pixels into segments.<br>2) a zero-shot classification task on segments. The former sub-task does not<br>involve category information and can be directly transferred to group pixels<br>for unseen classes. The latter subtask performs at segment-level and provides a<br>natural way to leverage large-scale vision-language models pre-trained with<br>image-text pairs (e.g. CLIP) for ZS3. Based on the decoupling formulation, we<br>propose a simple and effective zero-shot semantic segmentation model, called<br>ZegFormer, which outperforms the previous methods on ZS3 standard benchmarks by<br>large margins, e.g., 35 points on the PASCAL VOC and 3 points on the COCO-Stuff<br>in terms of mIoU for unseen classes. Code will be released at<br>https://github.com/dingjiansw101/ZegFormer.<br>

In this paper, we propose a novel co-learning framework (CoSSL) with<br>decoupled representation learning and classifier learning for imbalanced SSL.<br>To handle the data imbalance, we devise Tail-class Feature Enhancement (TFE)<br>for classifier learning. Furthermore, the current evaluation protocol for<br>imbalanced SSL focuses only on balanced test sets, which has limited<br>practicality in real-world scenarios. Therefore, we further conduct a<br>comprehensive evaluation under various shifted test distributions. In<br>experiments, we show that our approach outperforms other methods over a large<br>range of shifted distributions, achieving state-of-the-art performance on<br>benchmark datasets ranging from CIFAR-10, CIFAR-100, ImageNet, to Food-101. Our<br>code will be made publicly available.<br>

As acquiring pixel-wise annotations of real-world images for semantic<br>segmentation is a costly process, a model can instead be trained with more<br>accessible synthetic data and adapted to real images without requiring their<br>annotations. This process is studied in unsupervised domain adaptation (UDA).<br>Even though a large number of methods propose new adaptation strategies, they<br>are mostly based on outdated network architectures. As the influence of recent<br>network architectures has not been systematically studied, we first benchmark<br>different network architectures for UDA and then propose a novel UDA method,<br>DAFormer, based on the benchmark results. The DAFormer network consists of a<br>Transformer encoder and a multi-level context-aware feature fusion decoder. It<br>is enabled by three simple but crucial training strategies to stabilize the<br>training and to avoid overfitting DAFormer to the source domain: While the Rare<br>Class Sampling on the source domain improves the quality of pseudo-labels by<br>mitigating the confirmation bias of self-training towards common classes, the<br>Thing-Class ImageNet Feature Distance and a learning rate warmup promote<br>feature transfer from ImageNet pretraining. DAFormer significantly improves the<br>state-of-the-art performance by 10.8 mIoU for GTA->Cityscapes and 5.4 mIoU for<br>Synthia->Cityscapes and enables learning even difficult classes such as train,<br>bus, and truck well. The implementation is available at<br>https://github.com/lhoyer/DAFormer.<br>

Although considerable progress has been made in semantic scene understanding<br>under clear weather, it is still a tough problem under adverse weather<br>conditions, such as dense fog, due to the uncertainty caused by imperfect<br>observations. Besides, difficulties in collecting and labeling foggy images<br>hinder the progress of this field. Considering the success in semantic scene<br>understanding under clear weather, we think it is reasonable to transfer<br>knowledge learned from clear images to the foggy domain. As such, the problem<br>becomes to bridge the domain gap between clear images and foggy images. Unlike<br>previous methods that mainly focus on closing the domain gap caused by fog --<br>defogging the foggy images or fogging the clear images, we propose to alleviate<br>the domain gap by considering fog influence and style variation simultaneously.<br>The motivation is based on our finding that the style-related gap and the<br>fog-related gap can be divided and closed respectively, by adding an<br>intermediate domain. Thus, we propose a new pipeline to cumulatively adapt<br>style, fog and the dual-factor (style and fog). Specifically, we devise a<br>unified framework to disentangle the style factor and the fog factor<br>separately, and then the dual-factor from images in different domains.<br>Furthermore, we collaborate the disentanglement of three factors with a novel<br>cumulative loss to thoroughly disentangle these three factors. Our method<br>achieves the state-of-the-art performance on three benchmarks and shows<br>generalization ability in rainy and snowy scenes.<br>

Multi-Camera Multi-Object Tracking is currently drawing attention in the<br>computer vision field due to its superior performance in real-world<br>applications such as video surveillance with crowded scenes or in vast space.<br>In this work, we propose a mathematically elegant multi-camera multiple object<br>tracking approach based on a spatial-temporal lifted multicut formulation. Our<br>model utilizes state-of-the-art tracklets produced by single-camera trackers as<br>proposals. As these tracklets may contain ID-Switch errors, we refine them<br>through a novel pre-clustering obtained from 3D geometry projections. As a<br>result, we derive a better tracking graph without ID switches and more precise<br>affinity costs for the data association phase. Tracklets are then matched to<br>multi-camera trajectories by solving a global lifted multicut formulation that<br>incorporates short and long-range temporal interactions on tracklets located in<br>the same camera as well as inter-camera ones. Experimental results on the<br>WildTrack dataset yield near-perfect result, outperforming state-of-the-art<br>trackers on Campus while being on par on the PETS-09 dataset. We will make our<br>implementations available upon acceptance of the paper.<br>

3D human motion capture from monocular RGB images respecting interactions of<br>a subject with complex and possibly deformable environments is a very<br>challenging, ill-posed and under-explored problem. Existing methods address it<br>only weakly and do not model possible surface deformations often occurring when<br>humans interact with scene surfaces. In contrast, this paper proposes<br>MoCapDeform, i.e., a new framework for monocular 3D human motion capture that<br>is the first to explicitly model non-rigid deformations of a 3D scene for<br>improved 3D human pose estimation and deformable environment reconstruction.<br>MoCapDeform accepts a monocular RGB video and a 3D scene mesh aligned in the<br>camera space. It first localises a subject in the input monocular video along<br>with dense contact labels using a new raycasting based strategy. Next, our<br>human-environment interaction constraints are leveraged to jointly optimise<br>global 3D human poses and non-rigid surface deformations. MoCapDeform achieves<br>superior accuracy than competing methods on several datasets, including our<br>newly recorded one with deforming background scenes.<br>

The minimum cost multicut problem is the NP-hard/APX-hard combinatorial<br>optimization problem of partitioning a real-valued edge-weighted graph such as<br>to minimize the total cost of the partition. While graph convolutional neural<br>networks (GNN) have proven to be promising in the context of combinatorial<br>optimization, most of them are only tailored to or tested on positive-valued<br>edge weights, i.e. they do not comply to the nature of the multicut problem. We<br>therefore adapt various GNN architectures including Graph Convolutional<br>Networks, Signed Graph Convolutional Networks and Graph Isomorphic Networks to<br>facilitate the efficient encoding of real-valued edge costs. Moreover, we<br>employ a reformulation of the multicut ILP constraints to a polynomial program<br>as loss function that allows to learn feasible multicut solutions in a scalable<br>way. Thus, we provide the first approach towards end-to-end trainable<br>multicuts. Our findings support that GNN approaches can produce good solutions<br>in practice while providing lower computation times and largely improved<br>scalability compared to LP solvers and optimized heuristics, especially when<br>considering large instances.<br>

The Minimum Cost Multicut Problem (MP) is a popular way for obtaining a graph<br>decomposition by optimizing binary edge labels over edge costs. While the<br>formulation of a MP from independently estimated costs per edge is highly<br>flexible and intuitive, solving the MP is NP-hard and time-expensive. As a<br>remedy, recent work proposed to predict edge probabilities with awareness to<br>potential conflicts by incorporating cycle constraints in the prediction<br>process. We argue that such formulation, while providing a first step towards<br>end-to-end learnable edge weights, is suboptimal, since it is built upon a<br>loose relaxation of the MP. We therefore propose an adaptive CRF that allows to<br>progressively consider more violated constraints and, in consequence, to issue<br>solutions with higher validity. Experiments on the BSDS500 benchmark for<br>natural image segmentation as well as on electron microscopic recordings show<br>that our approach yields more precise edge detection and image segmentation.<br>

Besides standard cameras, autonomous vehicles typically include multiple<br>additional sensors, such as lidars and radars, which help acquire richer<br>information for perceiving the content of the driving scene. While several<br>recent works focus on fusing certain pairs of sensors - such as camera and<br>lidar or camera and radar - by using architectural components specific to the<br>examined setting, a generic and modular sensor fusion architecture is missing<br>from the literature. In this work, we focus on 2D object detection, a<br>fundamental high-level task which is defined on the 2D image domain, and<br>propose HRFuser, a multi-resolution sensor fusion architecture that scales<br>straightforwardly to an arbitrary number of input modalities. The design of<br>HRFuser is based on state-of-the-art high-resolution networks for image-only<br>dense prediction and incorporates a novel multi-window cross-attention block as<br>the means to perform fusion of multiple modalities at multiple resolutions.<br>Even though cameras alone provide very informative features for 2D detection,<br>we demonstrate via extensive experiments on the nuScenes and Seeing Through Fog<br>datasets that our model effectively leverages complementary features from<br>additional modalities, substantially improving upon camera-only performance and<br>consistently outperforming state-of-the-art fusion methods for 2D detection<br>both in normal and adverse conditions. The source code will be made publicly<br>available.<br>

Semi-supervised learning (SSL) has shown great promise in leveraging<br>unlabeled data to improve model performance. While standard SSL assumes uniform<br>data distribution, we consider a more realistic and challenging setting called<br>imbalanced SSL, where imbalanced class distributions occur in both labeled and<br>unlabeled data. Although there are existing endeavors to tackle this challenge,<br>their performance degenerates when facing severe imbalance since they can not<br>reduce the class imbalance sufficiently and effectively. In this paper, we<br>study a simple yet overlooked baseline -- SimiS -- which tackles data imbalance<br>by simply supplementing labeled data with pseudo-labels, according to the<br>difference in class distribution from the most frequent class. Such a simple<br>baseline turns out to be highly effective in reducing class imbalance. It<br>outperforms existing methods by a significant margin, e.g., 12.8%, 13.6%, and<br>16.7% over previous SOTA on CIFAR100-LT, FOOD101-LT, and ImageNet127<br>respectively. The reduced imbalance results in faster convergence and better<br>pseudo-label accuracy of SimiS. The simplicity of our method also makes it<br>possible to be combined with other re-balancing techniques to improve the<br>performance further. Moreover, our method shows great robustness to a wide<br>range of data distributions, which holds enormous potential in practice. Code<br>will be publicly available.<br>

State-of-the-art deep learning models are often trained with a large amount<br>of costly labeled training data. However, requiring exhaustive manual<br>annotations may degrade the model's generalizability in the limited-label<br>regime. Semi-supervised learning and unsupervised learning offer promising<br>paradigms to learn from an abundance of unlabeled visual data. Recent progress<br>in these paradigms has indicated the strong benefits of leveraging unlabeled<br>data to improve model generalization and provide better model initialization.<br>In this survey, we review the recent advanced deep learning algorithms on<br>semi-supervised learning (SSL) and unsupervised learning (UL) for visual<br>recognition from a unified perspective. To offer a holistic understanding of<br>the state-of-the-art in these areas, we propose a unified taxonomy. We<br>categorize existing representative SSL and UL with comprehensive and insightful<br>analysis to highlight their design rationales in different learning scenarios<br>and applications in different computer vision tasks. Lastly, we discuss the<br>emerging trends and open challenges in SSL and UL to shed light on future<br>critical research directions.<br>

Unintentional actions are rare occurrences that are difficult to define<br>precisely and that are highly dependent on the temporal context of the action.<br>In this work, we explore such actions and seek to identify the points in videos<br>where the actions transition from intentional to unintentional. We propose a<br>multi-stage framework that exploits inherent biases such as motion speed,<br>motion direction, and order to recognize unintentional actions. To enhance<br>representations via self-supervised training for the task of unintentional<br>action recognition we propose temporal transformations, called Temporal<br>Transformations of Inherent Biases of Unintentional Actions (T2IBUA). The<br>multi-stage approach models the temporal information on both the level of<br>individual frames and full clips. These enhanced representations show strong<br>performance for unintentional action recognition tasks. We provide an extensive<br>ablation study of our framework and report results that significantly improve<br>over the state-of-the-art.<br>

Improving model's generalizability against domain shifts is crucial,<br>especially for safety-critical applications such as autonomous driving.<br>Real-world domain styles can vary substantially due to environment changes and<br>sensor noises, but deep models only know the training domain style. Such domain<br>style gap impedes model generalization on diverse real-world domains. Our<br>proposed Normalization Perturbation (NP) can effectively overcome this domain<br>style overfitting problem. We observe that this problem is mainly caused by the<br>biased distribution of low-level features learned in shallow CNN layers. Thus,<br>we propose to perturb the channel statistics of source domain features to<br>synthesize various latent styles, so that the trained deep model can perceive<br>diverse potential domains and generalizes well even without observations of<br>target domain data in training. We further explore the style-sensitive channels<br>for effective style synthesis. Normalization Perturbation only relies on a<br>single source domain and is surprisingly effective and extremely easy to<br>implement. Extensive experiments verify the effectiveness of our method for<br>generalizing models under real-world domain shifts.<br>

In everyday lives, humans naturally modify the surrounding environment<br>through interactions, e.g., moving a chair to sit on it. To reproduce such<br>interactions in virtual spaces (e.g., metaverse), we need to be able to capture<br>and model them, including changes in the scene geometry, ideally from<br>ego-centric input alone (head camera and body-worn inertial sensors). This is<br>an extremely hard problem, especially since the object/scene might not be<br>visible from the head camera (e.g., a human not looking at a chair while<br>sitting down, or not looking at the door handle while opening a door). In this<br>paper, we present HOPS, the first method to capture interactions such as<br>dragging objects and opening doors from ego-centric data alone. Central to our<br>method is reasoning about human-object interactions, allowing to track objects<br>even when they are not visible from the head camera. HOPS localizes and<br>registers both the human and the dynamic object in a pre-scanned static scene.<br>HOPS is an important first step towards advanced AR/VR applications based on<br>immersive virtual universes, and can provide human-centric training data to<br>teach machines to interact with their surroundings. The supplementary video,<br>data, and code will be available on our project page at<br>http://virtualhumans.mpi-inf.mpg.de/hops/<br>

In unsupervised domain adaptation (UDA), a model trained on source data (e.g.<br>synthetic) is adapted to target data (e.g. real-world) without access to target<br>annotation. Most previous UDA methods struggle with classes that have a similar<br>visual appearance on the target domain as no ground truth is available to learn<br>the slight appearance differences. To address this problem, we propose a Masked<br>Image Consistency (MIC) module to enhance UDA by learning spatial context<br>relations of the target domain as additional clues for robust visual<br>recognition. MIC enforces the consistency between predictions of masked target<br>images, where random patches are withheld, and pseudo-labels that are generated<br>based on the complete image by an exponential moving average teacher. To<br>minimize the consistency loss, the network has to learn to infer the<br>predictions of the masked regions from their context. Due to its simple and<br>universal concept, MIC can be integrated into various UDA methods across<br>different visual recognition tasks such as image classification, semantic<br>segmentation, and object detection. MIC significantly improves the<br>state-of-the-art performance across the different recognition tasks for<br>synthetic-to-real, day-to-nighttime, and clear-to-adverse-weather UDA. For<br>instance, MIC achieves an unprecedented UDA performance of 75.9 mIoU and 92.8%<br>on GTA-to-Cityscapes and VisDA-2017, respectively, which corresponds to an<br>improvement of +2.1 and +3.0 percent points over the previous state of the art.<br>The implementation is available at https://github.com/lhoyer/MIC.<br>

This paper introduces DGNet, a novel deep framework that exploits object<br>gradient supervision for camouflaged object detection (COD). It decouples the<br>task into two connected branches, i.e., a context and a texture encoder. The<br>essential connection is the gradient-induced transition, representing a soft<br>grouping between context and texture features. Benefiting from the simple but<br>efficient framework, DGNet outperforms existing state-of-the-art COD models by<br>a large margin. Notably, our efficient version, DGNet-S, runs in real-time (80<br>fps) and achieves comparable results to the cutting-edge model<br>JCSOD-CVPR$_{21}$ with only 6.82% parameters. Application results also show<br>that the proposed DGNet performs well in polyp segmentation, defect detection,<br>and transparent object segmentation tasks. Codes will be made available at<br>https://github.com/GewelsJI/DGNet.<br>

We present a novel method for performing flexible, 3D-aware image content<br>manipulation while enabling high-quality novel view synthesis. While NeRF-based<br>approaches are effective for novel view synthesis, such models memorize the<br>radiance for every point in a scene within a neural network. Since these models<br>are scene-specific and lack a 3D scene representation, classical editing such<br>as shape manipulation, or combining scenes is not possible. Hence, editing and<br>combining NeRF-based scenes has not been demonstrated. With the aim of<br>obtaining interpretable and controllable scene representations, our model<br>couples learnt scene-specific feature volumes with a scene agnostic neural<br>rendering network. With this hybrid representation, we decouple neural<br>rendering from scene-specific geometry and appearance. We can generalize to<br>novel scenes by optimizing only the scene-specific 3D feature representation,<br>while keeping the parameters of the rendering network fixed. The rendering<br>function learnt during the initial training stage can thus be easily applied to<br>new scenes, making our approach more flexible. More importantly, since the<br>feature volumes are independent of the rendering model, we can manipulate and<br>combine scenes by editing their corresponding feature volumes. The edited<br>volume can then be plugged into the rendering model to synthesize high-quality<br>novel views. We demonstrate various scene manipulations, including mixing<br>scenes, deforming objects and inserting objects into scenes, while still<br>producing photo-realistic results.<br>

Planning an optimal route in a complex environment requires efficient<br>reasoning about the surrounding scene. While human drivers prioritize important<br>objects and ignore details not relevant to the decision, learning-based<br>planners typically extract features from dense, high-dimensional grid<br>representations containing all vehicle and road context information. In this<br>paper, we propose PlanT, a novel approach for planning in the context of<br>self-driving that uses a standard transformer architecture. PlanT is based on<br>imitation learning with a compact object-level input representation. On the<br>Longest6 benchmark for CARLA, PlanT outperforms all prior methods (matching the<br>driving score of the expert) while being 5.3x faster than equivalent<br>pixel-based planning baselines during inference. Combining PlanT with an<br>off-the-shelf perception module provides a sensor-based driving system that is<br>more than 10 points better in terms of driving score than the existing state of<br>the art. Furthermore, we propose an evaluation protocol to quantify the ability<br>of planners to identify relevant objects, providing insights regarding their<br>decision-making. Our results indicate that PlanT can focus on the most relevant<br>object in the scene, even when this object is geometrically distant.<br>

Prior work has extensively studied the latent space structure of GANs for<br>unconditional image synthesis, enabling global editing of generated images by<br>the unsupervised discovery of interpretable latent directions. However, the<br>discovery of latent directions for conditional GANs for semantic image<br>synthesis (SIS) has remained unexplored. In this work, we specifically focus on<br>addressing this gap. We propose a novel optimization method for finding<br>spatially disentangled class-specific directions in the latent space of<br>pretrained SIS models. We show that the latent directions found by our method<br>can effectively control the local appearance of semantic classes, e.g.,<br>changing their internal structure, texture or color independently from each<br>other. Visual inspection and quantitative evaluation of the discovered GAN<br>controls on various datasets demonstrate that our method discovers a diverse<br>set of unique and semantically meaningful latent directions for class-specific<br>edits.<br>

In this report, we present the 1st place solution for motion prediction track<br>in 2022 Waymo Open Dataset Challenges. We propose a novel Motion Transformer<br>framework for multimodal motion prediction, which introduces a small set of<br>novel motion query pairs for generating better multimodal future trajectories<br>by jointly performing the intention localization and iterative motion<br>refinement. A simple model ensemble strategy with non-maximum-suppression is<br>adopted to further boost the final performance. Our approach achieves the 1st<br>place on the motion prediction leaderboard of 2022 Waymo Open Dataset<br>Challenges, outperforming other methods with remarkable margins. Code will be<br>available at https://github.com/sshaoshuai/MTR.<br>

Deep learning is becoming increasingly relevant for many high-stakes applications such as autonomous driving or medical diagnosis where wrong decisions can have massive impact on human lives. Unfortunately, deep neural networks are typically assessed solely based on generalization, e.g., accuracy on a fixed test set. However, this is clearly insufficient for safe deployment as potential malicious actors and distribution shifts or the effects of quantization and unreliable hardware are disregarded. Thus, recent work additionally evaluates performance on potentially manipulated or corrupted inputs as well as after quantization and deployment on specialized hardware. In such settings, it is also important to obtain reasonable estimates of the model's confidence alongside its predictions. This thesis studies robustness and uncertainty estimation in deep learning along three main directions: First, we consider so-called adversarial examples, slightly perturbed inputs causing severe drops in accuracy. Second, we study weight perturbations, focusing particularly on bit errors in quantized weights. This is relevant for deploying models on special-purpose hardware for efficient inference, so-called accelerators. Finally, we address uncertainty estimation to improve robustness and provide meaningful statistical performance guarantees for safe deployment. In detail, we study the existence of adversarial examples with respect to the underlying data manifold. In this context, we also investigate adversarial training which improves robustness by augmenting training with adversarial examples at the cost of reduced accuracy. We show that regular adversarial examples leave the data manifold in an almost orthogonal direction. While we find no inherent trade-off between robustness and accuracy, this contributes to a higher sample complexity as well as severe overfitting of adversarial training. Using a novel measure of flatness in the robust loss landscape with respect to weight changes, we also show that robust overfitting is caused by converging to particularly sharp minima. In fact, we find a clear correlation between flatness and good robust generalization. Further, we study random and adversarial bit errors in quantized weights. In accelerators, random bit errors occur in the memory when reducing voltage with the goal of improving energy-efficiency. Here, we consider a robust quantization scheme, use weight clipping as regularization and perform random bit error training to improve bit error robustness, allowing considerable energy savings without requiring hardware changes. In contrast, adversarial bit errors are maliciously introduced through hardware- or software-based attacks on the memory, with severe consequences on performance. We propose a novel adversarial bit error attack to study this threat and use adversarial bit error training to improve robustness and thereby also the accelerator's security. Finally, we view robustness in the context of uncertainty estimation. By encouraging low-confidence predictions on adversarial examples, our confidence-calibrated adversarial training successfully rejects adversarial, corrupted as well as out-of-distribution examples at test time. Thereby, we are also able to improve the robustness-accuracy trade-off compared to regular adversarial training. However, even robust models do not provide any guarantee for safe deployment. To address this problem, conformal prediction allows the model to predict confidence sets with user-specified guarantee of including the true label. Unfortunately, as conformal prediction is usually applied after training, the model is trained without taking this calibration step into account. To address this limitation, we propose conformal training which allows training conformal predictors end-to-end with the underlying model. This not only improves the obtained uncertainty estimates but also enables optimizing application-specific objectives without losing the provided guarantee. Besides our work on robustness or uncertainty, we also address the problem of 3D shape completion of partially observed point clouds. Specifically, we consider an autonomous driving or robotics setting where vehicles are commonly equipped with LiDAR or depth sensors and obtaining a complete 3D representation of the environment is crucial. However, ground truth shapes that are essential for applying deep learning techniques are extremely difficult to obtain. Thus, we propose a weakly-supervised approach that can be trained on the incomplete point clouds while offering efficient inference. In summary, this thesis contributes to our understanding of robustness against both input and weight perturbations. To this end, we also develop methods to improve robustness alongside uncertainty estimation for safe deployment of deep learning methods in high-stakes applications. In the particular context of autonomous driving, we also address 3D shape completion of sparse point clouds.

Structured prediction problems are one of the fundamental tools in machine<br>learning. In order to facilitate algorithm development for their numerical<br>solution, we collect in one place a large number of datasets in easy to read<br>formats for a diverse set of problem classes. We provide archival links to<br>datasets, description of the considered problems and problem formats, and a<br>short summary of problem characteristics including size, number of instances<br>etc. For reference we also give a non-exhaustive selection of algorithms<br>proposed in the literature for their solution. We hope that this central<br>repository will make benchmarking and comparison to established works easier.<br>We welcome submission of interesting new datasets and algorithms for inclusion<br>in our archive.<br>

Modern deep learning architecture utilize batch normalization (BN) to<br>stabilize training and improve accuracy. It has been shown that the BN layers<br>alone are surprisingly expressive. In the context of robustness against<br>adversarial examples, however, BN is argued to increase vulnerability. That is,<br>BN helps to learn fragile features. Nevertheless, BN is still used in<br>adversarial training, which is the de-facto standard to learn robust features.<br>In order to shed light on the role of BN in adversarial training, we<br>investigate to what extent the expressiveness of BN can be used to robustify<br>fragile features in comparison to random features. On CIFAR10, we find that<br>adversarially fine-tuning just the BN layers can result in non-trivial<br>adversarial robustness. Adversarially training only the BN layers from scratch,<br>in contrast, is not able to convey meaningful adversarial robustness. Our<br>results indicate that fragile features can be used to learn models with<br>moderate adversarial robustness, while random features cannot<br>

Current efficient LiDAR-based detection frameworks are lacking in exploiting<br>object relations, which naturally present in both spatial and temporal manners.<br>To this end, we introduce a simple, efficient, and effective two-stage<br>detector, termed as Ret3D. At the core of Ret3D is the utilization of novel<br>intra-frame and inter-frame relation modules to capture the spatial and<br>temporal relations accordingly. More Specifically, intra-frame relation module<br>(IntraRM) encapsulates the intra-frame objects into a sparse graph and thus<br>allows us to refine the object features through efficient message passing. On<br>the other hand, inter-frame relation module (InterRM) densely connects each<br>object in its corresponding tracked sequences dynamically, and leverages such<br>temporal information to further enhance its representations efficiently through<br>a lightweight transformer network. We instantiate our novel designs of IntraRM<br>and InterRM with general center-based or anchor-based detectors and evaluate<br>them on Waymo Open Dataset (WOD). With negligible extra overhead, Ret3D<br>achieves the state-of-the-art performance, being 5.5% and 3.2% higher than the<br>recent competitor in terms of the LEVEL 1 and LEVEL 2 mAPH metrics on vehicle<br>detection, respectively.<br>

We present TOCH, a method for refining incorrect 3D hand-object interaction<br>sequences using a data prior. Existing hand trackers, especially those that<br>rely on very few cameras, often produce visually unrealistic results with<br>hand-object intersection or missing contacts. Although correcting such errors<br>requires reasoning about temporal aspects of interaction, most previous work<br>focus on static grasps and contacts. The core of our method are TOCH fields, a<br>novel spatio-temporal representation for modeling correspondences between hands<br>and objects during interaction. The key component is a point-wise<br>object-centric representation which encodes the hand position relative to the<br>object. Leveraging this novel representation, we learn a latent manifold of<br>plausible TOCH fields with a temporal denoising auto-encoder. Experiments<br>demonstrate that TOCH outperforms state-of-the-art (SOTA) 3D hand-object<br>interaction models, which are limited to static grasps and contacts. More<br>importantly, our method produces smooth interactions even before and after<br>contact. Using a single trained TOCH model, we quantitatively and qualitatively<br>demonstrate its usefulness for 1) correcting erroneous reconstruction results<br>from off-the-shelf RGB/RGB-D hand-object reconstruction methods, 2) de-noising,<br>and 3) grasp transfer across objects. We will release our code and trained<br>model on our project page at http://virtualhumans.mpi-inf.mpg.de/toch/<br>

Skeleton-based action recognition aims to predict human actions given human<br>joint coordinates with skeletal interconnections. To model such off-grid data<br>points and their co-occurrences, Transformer-based formulations would be a<br>natural choice. However, Transformers still lag behind state-of-the-art methods<br>using graph convolutional networks (GCNs). Transformers assume that the input<br>is permutation-invariant and homogeneous (partially alleviated by positional<br>encoding), which ignores an important characteristic of skeleton data, i.e.,<br>bone connectivity. Furthermore, each type of body joint has a clear physical<br>meaning in human motion, i.e., motion retains an intrinsic relationship<br>regardless of the joint coordinates, which is not explored in Transformers. In<br>fact, certain re-occurring groups of body joints are often involved in specific<br>actions, such as the subconscious hand movement for keeping balance. Vanilla<br>attention is incapable of describing such underlying relations that are<br>persistent and beyond pair-wise. In this work, we aim to exploit these unique<br>aspects of skeleton data to close the performance gap between Transformers and<br>GCNs. Specifically, we propose a new self-attention (SA) extension, named<br>Hypergraph Self-Attention (HyperSA), to incorporate inherently higher-order<br>relations into the model. The K-hop relative positional embeddings are also<br>employed to take bone connectivity into account. We name the resulting model<br>Hyperformer, and it achieves comparable or better performance w.r.t. accuracy<br>and efficiency than state-of-the-art GCN architectures on NTU RGB+D, NTU RGB+D<br>120, and Northwestern-UCLA datasets. On the largest NTU RGB+D 120 dataset, the<br>significantly improved performance reached by our Hyperformer demonstrates the<br>underestimated potential of Transformer models in this field.<br>

Human performance capture is a highly important computer vision problem with<br>many applications in movie production and virtual/augmented reality. Many<br>previous performance capture approaches either required expensive multi-view<br>setups or did not recover dense space-time coherent geometry with<br>frame-to-frame correspondences. We propose a novel deep learning approach for<br>monocular dense human performance capture. Our method is trained in a weakly<br>supervised manner based on multi-view supervision completely removing the need<br>for training data with 3D ground truth annotations. The network architecture is<br>based on two separate networks that disentangle the task into a pose estimation<br>and a non-rigid surface deformation step. Extensive qualitative and<br>quantitative evaluations show that our approach outperforms the state of the<br>art in terms of quality and robustness. This work is an extended version of<br>DeepCap where we provide more detailed explanations, comparisons and results as<br>well as applications.<br>

Deep neural network (DNN) accelerators received considerable attention in<br>past years due to saved energy compared to mainstream hardware. Low-voltage<br>operation of DNN accelerators allows to further reduce energy consumption<br>significantly, however, causes bit-level failures in the memory storing the<br>quantized DNN weights. In this paper, we show that a combination of robust<br>fixed-point quantization, weight clipping, and random bit error training<br>(RandBET) improves robustness against random bit errors in (quantized) DNN<br>weights significantly. This leads to high energy savings from both low-voltage<br>operation as well as low-precision quantization. Our approach generalizes<br>across operating voltages and accelerators, as demonstrated on bit errors from<br>profiled SRAM arrays. We also discuss why weight clipping alone is already a<br>quite effective way to achieve robustness against bit errors. Moreover, we<br>specifically discuss the involved trade-offs regarding accuracy, robustness and<br>precision: Without losing more than 1% in accuracy compared to a normally<br>trained 8-bit DNN, we can reduce energy consumption on CIFAR-10 by 20%. Higher<br>energy savings of, e.g., 30%, are possible at the cost of 2.5% accuracy, even<br>for 4-bit DNNs.<br>

We present a massively parallel Lagrange decomposition method for solving 0-1<br>integer linear programs occurring in structured prediction. We propose a new<br>iterative update scheme for solving the Lagrangean dual and a perturbation<br>technique for decoding primal solutions. For representing subproblems we follow<br>Lange et al. (2021) and use binary decision diagrams (BDDs). Our primal and<br>dual algorithms require little synchronization between subproblems and<br>optimization over BDDs needs only elementary operations without complicated<br>control flow. This allows us to exploit the parallelism offered by GPUs for all<br>components of our method. We present experimental results on combinatorial<br>problems from MAP inference for Markov Random Fields, quadratic assignment and<br>cell tracking for developmental biology. Our highly parallel GPU implementation<br>improves upon the running times of the algorithms from Lange et al. (2021) by<br>up to an order of magnitude. In particular, we come close to or outperform some<br>state-of-the-art specialized heuristics while being problem agnostic.<br>

We introduce a new family of neural network models called Convolutional<br>Dynamic Alignment Networks (CoDA Nets), which are performant classifiers with a<br>high degree of inherent interpretability. Their core building blocks are<br>Dynamic Alignment Units (DAUs), which are optimised to transform their inputs<br>with dynamically computed weight vectors that align with task-relevant<br>patterns. As a result, CoDA Nets model the classification prediction through a<br>series of input-dependent linear transformations, allowing for linear<br>decomposition of the output into individual input contributions. Given the<br>alignment of the DAUs, the resulting contribution maps align with<br>discriminative input patterns. These model-inherent decompositions are of high<br>visual quality and outperform existing attribution methods under quantitative<br>metrics. Further, CoDA Nets constitute performant classifiers, achieving on par<br>results to ResNet and VGG models on e.g. CIFAR-10 and TinyImagenet. Lastly,<br>CoDA Nets can be combined with conventional neural network models to yield<br>powerful classifiers that more easily scale to complex datasets such as<br>Imagenet whilst exhibiting an increased interpretable depth, i.e., the output<br>can be explained well in terms of contributions from intermediate layers within<br>the network.<br>

Explaining the decision of a multi-modal decision-maker requires to determine<br>the evidence from both modalities. Recent advances in XAI provide explanations<br>for models trained on still images. However, when it comes to modeling multiple<br>sensory modalities in a dynamic world, it remains underexplored how to<br>demystify the mysterious dynamics of a complex multi-modal model. In this work,<br>we take a crucial step forward and explore learnable explanations for<br>audio-visual recognition. Specifically, we propose a novel space-time attention<br>network that uncovers the synergistic dynamics of audio and visual data over<br>both space and time. Our model is capable of predicting the audio-visual video<br>events, while justifying its decision by localizing where the relevant visual<br>cues appear, and when the predicted sounds occur in videos. We benchmark our<br>model on three audio-visual video event datasets, comparing extensively to<br>multiple recent multi-modal representation learners and intrinsic explanation<br>models. Experimental results demonstrate the clear superior performance of our<br>model over the existing methods on audio-visual video event recognition.<br>Moreover, we conduct an in-depth study to analyze the explainability of our<br>model based on robustness analysis via perturbation tests and pointing games<br>using human annotations.<br>

Text and images are the two most common data modalities found on the Internet. Understanding the synergy between text and images, that is, seamlessly analyzing information from these modalities may be trivial for humans, but is challenging for software systems. In this dissertation we study problems where deciphering text-image synergy is crucial for finding solutions. We propose methods and ideas that establish semantic connections between text and images in multimodal contents, and empirically show their effectiveness in four interconnected problems: Image Retrieval, Image Tag Refinement, Image-Text Alignment, and Image Captioning. Our promising results and observations open up interesting scopes for future research involving text-image data understanding.Text and images are the two most common data modalities found on the Internet. Understanding the synergy between text and images, that is, seamlessly analyzing information from these modalities may be trivial for humans, but is challenging for software systems. In this dissertation we study problems where deciphering text-image synergy is crucial for finding solutions. We propose methods and ideas that establish semantic connections between text and images in multimodal contents, and empirically show their effectiveness in four interconnected problems: Image Retrieval, Image Tag Refinement, Image-Text Alignment, and Image Captioning. Our promising results and observations open up interesting scopes for future research involving text-image data understanding.

Traditional domain adaptation addresses the task of adapting a model to a<br>novel target domain under limited or no additional supervision. While tackling<br>the input domain gap, the standard domain adaptation settings assume no domain<br>change in the output space. In semantic prediction tasks, different datasets<br>are often labeled according to different semantic taxonomies. In many<br>real-world settings, the target domain task requires a different taxonomy than<br>the one imposed by the source domain. We therefore introduce the more general<br>taxonomy adaptive domain adaptation (TADA) problem, allowing for inconsistent<br>taxonomies between the two domains. We further propose an approach that jointly<br>addresses the image-level and label-level domain adaptation. On the<br>label-level, we employ a bilateral mixed sampling strategy to augment the<br>target domain, and a relabelling method to unify and align the label spaces. We<br>address the image-level domain gap by proposing an uncertainty-rectified<br>contrastive learning method, leading to more domain-invariant and class<br>discriminative features. We extensively evaluate the effectiveness of our<br>framework under different TADA settings: open taxonomy, coarse-to-fine<br>taxonomy, and partially-overlapping taxonomy. Our framework outperforms<br>previous state-of-the-art by a large margin, while capable of adapting to<br>target taxonomies.<br>

Training deep networks for semantic segmentation requires large amounts of<br>labeled training data, which presents a major challenge in practice, as<br>labeling segmentation masks is a highly labor-intensive process. To address<br>this issue, we present a framework for semi-supervised and domain-adaptive<br>semantic segmentation, which is enhanced by self-supervised monocular depth<br>estimation (SDE) trained only on unlabeled image sequences.<br> In particular, we utilize SDE as an auxiliary task comprehensively across the<br>entire learning framework: First, we automatically select the most useful<br>samples to be annotated for semantic segmentation based on the correlation of<br>sample diversity and difficulty between SDE and semantic segmentation. Second,<br>we implement a strong data augmentation by mixing images and labels using the<br>geometry of the scene. Third, we transfer knowledge from features learned<br>during SDE to semantic segmentation by means of transfer and multi-task<br>learning. And fourth, we exploit additional labeled synthetic data with<br>Cross-Domain DepthMix and Matching Geometry Sampling to align synthetic and<br>real data.<br> We validate the proposed model on the Cityscapes dataset, where all four<br>contributions demonstrate significant performance gains, and achieve<br>state-of-the-art results for semi-supervised semantic segmentation as well as<br>for semi-supervised domain adaptation. In particular, with only 1/30 of the<br>Cityscapes labels, our method achieves 92% of the fully-supervised baseline<br>performance and even 97% when exploiting additional data from GTA. The source<br>code is available at<br>https://github.com/lhoyer/improving_segmentation_with_selfsupervised_depth.<br>

Compositional Zero-Shot learning (CZSL) aims to recognize unseen compositions<br>of state and object visual primitives seen during training. A problem with<br>standard CZSL is the assumption of knowing which unseen compositions will be<br>available at test time. In this work, we overcome this assumption operating on<br>the open world setting, where no limit is imposed on the compositional space at<br>test time, and the search space contains a large number of unseen compositions.<br>To address this problem, we propose a new approach, Compositional Cosine Graph<br>Embeddings (Co-CGE), based on two principles. First, Co-CGE models the<br>dependency between states, objects and their compositions through a graph<br>convolutional neural network. The graph propagates information from seen to<br>unseen concepts, improving their representations. Second, since not all unseen<br>compositions are equally feasible, and less feasible ones may damage the<br>learned representations, Co-CGE estimates a feasibility score for each unseen<br>composition, using the scores as margins in a cosine similarity-based loss and<br>as weights in the adjacency matrix of the graphs. Experiments show that our<br>approach achieves state-of-the-art performances in standard CZSL while<br>outperforming previous methods in the open world scenario.<br>

Abstract<br>Humans are at the centre of a significant amount of research in computer vision.<br>Endowing machines with the ability to perceive people from visual data is an immense<br>scientific challenge with a high degree of direct practical relevance. Success in automatic<br>perception can be measured at different levels of abstraction, and this will depend on<br>which intelligent behaviour we are trying to replicate: the ability to localise persons in<br>an image or in the environment, understanding how persons are moving at the skeleton<br>and at the surface level, interpreting their interactions with the environment including<br>with other people, and perhaps even anticipating future actions. In this thesis we tackle<br>different sub-problems of the broad research area referred to as "looking at people",<br>aiming to perceive humans in images at different levels of granularity.<br>We start with bounding box-level pedestrian detection: We present a retrospective<br>analysis of methods published in the decade preceding our work, identifying various<br>strands of research that have advanced the state of the art. With quantitative exper-<br>iments, we demonstrate the critical role of developing better feature representations<br>and having the right training distribution. We then contribute two methods based<br>on the insights derived from our analysis: one that combines the strongest aspects of<br>past detectors and another that focuses purely on learning representations. The latter<br>method outperforms more complicated approaches, especially those based on hand-<br>crafted features. We conclude our work on pedestrian detection with a forward-looking<br>analysis that maps out potential avenues for future research.<br>We then turn to pixel-level methods: Perceiving humans requires us to both separate<br>them precisely from the background and identify their surroundings. To this end, we<br>introduce Cityscapes, a large-scale dataset for street scene understanding. This has since<br>established itself as a go-to benchmark for segmentation and detection. We additionally<br>develop methods that relax the requirement for expensive pixel-level annotations, focusing<br>on the task of boundary detection, i.e. identifying the outlines of relevant objects and<br>surfaces. Next, we make the jump from pixels to 3D surfaces, from localising and<br>labelling to fine-grained spatial understanding. We contribute a method for recovering<br>3D human shape and pose, which marries the advantages of learning-based and model-<br>based approaches.<br>We conclude the thesis with a detailed discussion of benchmarking practices in<br>computer vision. Among other things, we argue that the design of future datasets<br>should be driven by the general goal of combinatorial robustness besides task-specific<br>considerations.

Deep neural network (DNN) accelerators received considerable attention in<br>recent years due to the potential to save energy compared to mainstream<br>hardware. Low-voltage operation of DNN accelerators allows to further reduce<br>energy consumption significantly, however, causes bit-level failures in the<br>memory storing the quantized DNN weights. Furthermore, DNN accelerators have<br>been shown to be vulnerable to adversarial attacks on voltage controllers or<br>individual bits. In this paper, we show that a combination of robust<br>fixed-point quantization, weight clipping, as well as random bit error training<br>(RandBET) or adversarial bit error training (AdvBET) improves robustness<br>against random or adversarial bit errors in quantized DNN weights<br>significantly. This leads not only to high energy savings for low-voltage<br>operation as well as low-precision quantization, but also improves security of<br>DNN accelerators. Our approach generalizes across operating voltages and<br>accelerators, as demonstrated on bit errors from profiled SRAM arrays, and<br>achieves robustness against both targeted and untargeted bit-level attacks.<br>Without losing more than 0.8%/2% in test accuracy, we can reduce energy<br>consumption on CIFAR10 by 20%/30% for 8/4-bit quantization using RandBET.<br>Allowing up to 320 adversarial bit errors, AdvBET reduces test error from above<br>90% (chance level) to 26.22% on CIFAR10.<br>

Most learning methods for 3D data (point clouds, meshes) suffer significant<br>performance drops when the data is not carefully aligned to a canonical<br>orientation. Aligning real world 3D data collected from different sources is<br>non-trivial and requires manual intervention. In this paper, we propose the<br>Adjoint Rigid Transform (ART) Network, a neural module which can be integrated<br>with a variety of 3D networks to significantly boost their performance. ART<br>learns to rotate input shapes to a learned canonical orientation, which is<br>crucial for a lot of tasks such as shape reconstruction, interpolation,<br>non-rigid registration, and latent disentanglement. ART achieves this with<br>self-supervision and a rotation equivariance constraint on predicted rotations.<br>The remarkable result is that with only self-supervision, ART facilitates<br>learning a unique canonical orientation for both rigid and nonrigid shapes,<br>which leads to a notable boost in performance of aforementioned tasks. We will<br>release our code and pre-trained models for further research.<br>

Modern approaches to pose and body shape estimation have recently achieved<br>strong performance even under challenging real-world conditions. Even from a<br>single image of a clothed person, a realistic looking body shape can be<br>inferred that captures a users' weight group and body shape type well. This<br>opens up a whole spectrum of applications -- in particular in fashion -- where<br>virtual try-on and recommendation systems can make use of these new and<br>automatized cues. However, a realistic depiction of the undressed body is<br>regarded highly private and therefore might not be consented by most people.<br>Hence, we ask if the automatic extraction of such information can be<br>effectively evaded. While adversarial perturbations have been shown to be<br>effective for manipulating the output of machine learning models -- in<br>particular, end-to-end deep learning approaches -- state of the art shape<br>estimation methods are composed of multiple stages. We perform the first<br>investigation of different strategies that can be used to effectively<br>manipulate the automatic shape estimation while preserving the overall<br>appearance of the original image.<br>

Current datasets for video-based person re-identification (re-ID) do not<br>include structural knowledge in form of human pose annotations for the persons<br>of interest. Nonetheless, pose information is very helpful to disentangle<br>useful feature information from background or occlusion noise. Especially<br>real-world scenarios, such as surveillance, contain a lot of occlusions in<br>human crowds or by obstacles. On the other hand, video-based person re-ID can<br>benefit other tasks such as multi-person pose tracking in terms of robust<br>feature matching. For that reason, we present PoseTrackReID, a large-scale<br>dataset for multi-person pose tracking and video-based person re-ID. With<br>PoseTrackReID, we want to bridge the gap between person re-ID and multi-person<br>pose tracking. Additionally, this dataset provides a good benchmark for current<br>state-of-the-art methods on multi-frame person re-ID.<br>

Intuitively, image classification should profit from using spatial<br>information. Recent work, however, suggests that this might be overrated in<br>standard CNNs. In this paper, we are pushing the envelope and aim to further<br>investigate the reliance on spatial information. We propose spatial shuffling<br>and GAP+FC to destroy spatial information during both training and testing<br>phases. Interestingly, we observe that spatial information can be deleted from<br>later layers with small performance drops, which indicates spatial information<br>at later layers is not necessary for good performance. For example, test<br>accuracy of VGG-16 only drops by 0.03% and 2.66% with spatial information<br>completely removed from the last 30% and 53% layers on CIFAR100, respectively.<br>Evaluation on several object recognition datasets (CIFAR100, Small-ImageNet,<br>ImageNet) with a wide range of CNN architectures (VGG16, ResNet50, ResNet152)<br>shows an overall consistent pattern.<br>

Modern deep learning has enabled amazing developments of computer vision in recent years (Hinton and Salakhutdinov, 2006; Krizhevsky et al., 2012). As a fundamental task, semantic segmentation aims to predict class labels for each pixel of images, which empowers machines perception of the visual world. In spite of recent successes of fully convolutional networks (Long etal., 2015), several challenges remain to be addressed. In this thesis, we focus on this topic, under different kinds of input formats and various types of scenes. Specifically, our study contains two aspects: (1) Data-driven neural modules for improved performance. (2) Leverage of datasets w.r.t.training systems with higher performances and better data privacy guarantees. In the first part of this thesis, we improve semantic segmentation by designing new modules which are compatible with existing architectures. First, we develop a spatio-temporal data-driven pooling, which brings additional information of data (i.e. superpixels) into neural networks, benefiting the training of neural networks as well as the inference on novel data. We investigate our approach in RGB-D videos for segmenting indoor scenes, where depth provides complementary cues to colors and our model performs particularly well. Second, we design learnable dilated convolutions, which are the extension of standard dilated convolutions, whose dilation factors (Yu and Koltun, 2016) need to be carefully determined by hand to obtain decent performance. We present a method to learn dilation factors together with filter weights of convolutions to avoid a complicated search of dilation factors. We explore extensive studies on challenging street scenes, across various baselines with different complexity as well as several datasets at varying image resolutions. In the second part, we investigate how to utilize expensive training data. First, we start from the generative modelling and study the network architectures and the learning pipeline for generating multiple examples. We aim to improve the diversity of generated examples but also to preserve the comparable quality of the examples. Second, we develop a generative model for synthesizing features of a network. With a mixture of real images and synthetic features, we are able to train a segmentation model with better generalization capability. Our approach is evaluated on different scene parsing tasks to demonstrate the effectiveness of the proposed method. Finally, we study membership inference on the semantic segmentation task. We propose the first membership inference attack system against black-box semantic segmentation models, that tries to infer if a data pair is used as training data or not. From our observations, information on training data is indeed leaking. To mitigate the leakage, we leverage our synthetic features to perform prediction obfuscations, reducing the posterior distribution gaps between a training and a testing set. Consequently, our study provides not only an approach for detecting illegal use of data, but also the foundations for a safer use of semantic segmentation models.

Few-shot learning methods operate in low data regimes. The aim is to learn<br>with few training examples per class. Although significant progress has been<br>made in few-shot image classification, few-shot video recognition is relatively<br>unexplored and methods based on 2D CNNs are unable to learn temporal<br>information. In this work we thus develop a simple 3D CNN baseline, surpassing<br>existing methods by a large margin. To circumvent the need of labeled examples,<br>we propose to leverage weakly-labeled videos from a large dataset using tag<br>retrieval followed by selecting the best clips with visual similarities,<br>yielding further improvement. Our results saturate current 5-way benchmarks for<br>few-shot video classification and therefore we propose a new challenging<br>benchmark involving more classes and a mixture of classes with varying<br>supervision.<br>

Automatic detection of emergent leaders in small groups from nonverbal<br>behaviour is a growing research topic in social signal processing but existing<br>methods were evaluated on single datasets -- an unrealistic assumption for<br>real-world applications in which systems are required to also work in settings<br>unseen at training time. It therefore remains unclear whether current methods<br>for emergent leadership detection generalise to similar but new settings and to<br>which extent. To overcome this limitation, we are the first to study a<br>cross-dataset evaluation setting for the emergent leadership detection task. We<br>provide evaluations for within- and cross-dataset prediction using two current<br>datasets (PAVIS and MPIIGroupInteraction), as well as an investigation on the<br>robustness of commonly used feature channels (visual focus of attention, body<br>pose, facial action units, speaking activity) and online prediction in the<br>cross-dataset setting. Our evaluations show that using pose and eye contact<br>based features, cross-dataset prediction is possible with an accuracy of 0.68,<br>as such providing another important piece of the puzzle towards emergent<br>leadership detection in the real world.<br>

When labeled training data is scarce, a promising data augmentation approach is to generate visual features of unknown classes using their attributes. To learn the class conditional distribution of CNN features, these models rely on pairs of image features and class attributes. Hence, they can not make use of the abundance of unlabeled data samples. In this paper, we tackle any-shot learning problems i.e. zero-shot and few-shot, in a unified feature generating framework that operates in both inductive and transductive learning settings. We develop a conditional generative model that combines the strength of VAE and GANs and in addition, via an unconditional discriminator, learns the marginal feature distribution of unlabeled images. We empirically show that our model learns highly discriminative CNN features for five datasets, i.e. CUB, SUN, AWA and ImageNet, and establish a new state-of-the-art in any-shot learning, i.e. inductive and transductive (generalized) zero- and few-shot learning settings. We also demonstrate that our learned features are interpretable: we visualize them by inverting them back to the pixel space and we explain them by generating textual arguments of why they are associated with a certain label.

When labeled training data is scarce, a promising data augmentation approach is to generate visual features of unknown classes using their attributes. To learn the class conditional distribution of CNN features, these models rely on pairs of image features and class attributes. Hence, they can not make use of the abundance of unlabeled data samples. In this paper, we tackle any-shot learning problems i.e. zero-shot and few-shot, in a unified feature generating framework that operates in both inductive and transductive learning settings. We develop a conditional generative model that combines the strength of VAE and GANs and in addition, via an unconditional discriminator, learns the marginal feature distribution of unlabeled images. We empirically show that our model learns highly discriminative CNN features for five datasets, i.e. CUB, SUN, AWA and ImageNet, and establish a new state-of-the-art in any-shot learning, i.e. inductive and transductive (generalized) zero- and few-shot learning settings. We also demonstrate that our learned features are interpretable: we visualize them by inverting them back to the pixel space and we explain them by generating textual arguments of why they are associated with a certain label.

Due to the importance of zero-shot learning, i.e. classifying images where<br>there is a lack of labeled training data, the number of proposed approaches has<br>recently increased steadily. We argue that it is time to take a step back and<br>to analyze the status quo of the area. The purpose of this paper is three-fold.<br>First, given the fact that there is no agreed upon zero-shot learning<br>benchmark, we first define a new benchmark by unifying both the evaluation<br>protocols and data splits of publicly available datasets used for this task.<br>This is an important contribution as published results are often not comparable<br>and sometimes even flawed due to, e.g. pre-training on zero-shot test classes.<br>Moreover, we propose a new zero-shot learning dataset, the Animals with<br>Attributes 2 (AWA2) dataset which we make publicly available both in terms of<br>image features and the images themselves. Second, we compare and analyze a<br>significant number of the state-of-the-art methods in depth, both in the<br>classic zero-shot setting but also in the more realistic generalized zero-shot<br>setting. Finally, we discuss in detail the limitations of the current status of<br>the area which can be taken as a basis for advancing it.<br>

We present a simple yet effective method to infer detailed full human body<br>shape from only a single photograph. Our model can infer full-body shape<br>including face, hair, and clothing including wrinkles at interactive<br>frame-rates. Results feature details even on parts that are occluded in the<br>input image. Our main idea is to turn shape regression into an aligned<br>image-to-image translation problem. The input to our method is a partial<br>texture map of the visible region obtained from off-the-shelf methods. From a<br>partial texture, we estimate detailed normal and vector displacement maps,<br>which can be applied to a low-resolution smooth body model to add detail and<br>clothing. Despite being trained purely with synthetic data, our model<br>generalizes well to real-world photographs. Numerous results demonstrate the<br>versatility and robustness of our method.<br>

Internal thought refers to the process of directing attention away from a<br>primary visual task to internal cognitive processing. Internal thought is a<br>pervasive mental activity and closely related to primary task performance. As<br>such, automatic detection of internal thought has significant potential for<br>user modelling in intelligent interfaces, particularly for e-learning<br>applications. Despite the close link between the eyes and the human mind, only<br>a few studies have investigated vergence behaviour during internal thought and<br>none has studied moment-to-moment detection of internal thought from gaze.<br>While prior studies relied on long-term data analysis and required a large<br>number of gaze characteristics, we describe a novel method that is<br>computationally light-weight and that only requires eye vergence information<br>that is readily available from binocular eye trackers. We further propose a<br>novel paradigm to obtain ground truth internal thought annotations that<br>exploits human blur perception. We evaluate our method for three increasingly<br>challenging detection tasks: (1) during a controlled math-solving task, (2)<br>during natural viewing of lecture videos, and (3) during daily activities, such<br>as coding, browsing, and reading. Results from these evaluations demonstrate<br>the performance and robustness of vergence-based detection of internal thought<br>and, as such, open up new directions for research on interfaces that adapt to<br>shifts of mental attention.<br>

Recent methods to automatically calibrate stationary eye trackers were shown<br>to effectively reduce inherent calibration distortion. However, these methods<br>require additional information, such as mouse clicks or on-screen content. We<br>propose the first method that only requires users' eye movements to reduce<br>calibration distortion in the background while users naturally look at an<br>interface. Our method exploits that calibration distortion makes straight<br>saccade trajectories appear curved between the saccadic start and end points.<br>We show that this curving effect is systematic and the result of distorted gaze<br>projection plane. To mitigate calibration distortion, our method undistorts<br>this plane by straightening saccade trajectories using image warping. We show<br>that this approach improves over the common six-point calibration and is<br>promising for reducing distortion. As such, it provides a non-intrusive<br>solution to alleviating accuracy decrease of eye tracker during long-term use.<br>

We consider general discrete Markov Random Fields(MRFs) with additional<br>bottleneck potentials which penalize the maximum (instead of the sum) over<br>local potential value taken by the MRF-assignment. Bottleneck potentials or<br>analogous constructions have been considered in (i) combinatorial optimization<br>(e.g. bottleneck shortest path problem, the minimum bottleneck spanning tree<br>problem, bottleneck function minimization in greedoids), (ii) inverse problems<br>with $L_{\infty}$-norm regularization, and (iii) valued constraint satisfaction<br>on the $(\min,\max)$-pre-semirings. Bottleneck potentials for general discrete<br>MRFs are a natural generalization of the above direction of modeling work to<br>Maximum-A-Posteriori (MAP) inference in MRFs. To this end, we propose MRFs<br>whose objective consists of two parts: terms that factorize according to (i)<br>$(\min,+)$, i.e. potentials as in plain MRFs, and (ii) $(\min,\max)$, i.e.<br>bottleneck potentials. To solve the ensuing inference problem, we propose<br>high-quality relaxations and efficient algorithms for solving them. We<br>empirically show efficacy of our approach on large scale seismic horizon<br>tracking problems.<br>

Generative Adversarial Networks (GANs) can achieve state-of-the-art sample<br>quality in generative modelling tasks but suffer from the mode collapse<br>problem. Variational Autoencoders (VAE) on the other hand explicitly maximize a<br>reconstruction-based data log-likelihood forcing it to cover all modes, but<br>suffer from poorer sample quality. Recent works have proposed hybrid VAE-GAN<br>frameworks which integrate a GAN-based synthetic likelihood to the VAE<br>objective to address both the mode collapse and sample quality issues, with<br>limited success. This is because the VAE objective forces a trade-off between<br>the data log-likelihood and divergence to the latent prior. The synthetic<br>likelihood ratio term also shows instability during training. We propose a<br>novel objective with a "Best-of-Many-Samples" reconstruction cost and a stable<br>direct estimate of the synthetic likelihood. This enables our hybrid VAE-GAN<br>framework to achieve high data log-likelihood and low divergence to the latent<br>prior at the same time and shows significant improvement over both hybrid<br>VAE-GANS and plain GANs in mode coverage and quality.<br>

Meta-learning has been shown to be an effective strategy for few-shot<br>learning. The key idea is to leverage a large number of similar few-shot tasks<br>in order to meta-learn how to best initiate a (single) base-learner for novel<br>few-shot tasks. While meta-learning how to initialize a base-learner has shown<br>promising results, it is well known that hyperparameter settings such as the<br>learning rate and the weighting of the regularization term are important to<br>achieve best performance. We thus propose to also meta-learn these<br>hyperparameters and in fact learn a time- and layer-varying scheme for learning<br>a base-learner on novel tasks. Additionally, we propose to learn not only a<br>single base-learner but an ensemble of several base-learners to obtain more<br>robust results. While ensembles of learners have shown to improve performance<br>in various settings, this is challenging for few-shot learning tasks due to the<br>limited number of training samples. Therefore, our approach also aims to<br>meta-learn how to effectively combine several base-learners. We conduct<br>extensive experiments and report top performance for five-class few-shot<br>recognition tasks on two challenging benchmarks: miniImageNet and<br>Fewshot-CIFAR100 (FC100).<br>

Understanding physical phenomena is a key competence that enables humans and<br>animals to act and interact under uncertain perception in previously unseen<br>environments containing novel objects and their configurations. In this work,<br>we consider the problem of autonomous block stacking and explore solutions to<br>learning manipulation under physics constraints with visual perception inherent<br>to the task. Inspired by the intuitive physics in humans, we first present an<br>end-to-end learning-based approach to predict stability directly from<br>appearance, contrasting a more traditional model-based approach with explicit<br>3D representations and physical simulation. We study the model's behavior<br>together with an accompanied human subject test. It is then integrated into a<br>real-world robotic system to guide the placement of a single wood block into<br>the scene without collapsing existing tower structure. To further automate the<br>process of consecutive blocks stacking, we present an alternative approach<br>where the model learns the physics constraint through the interaction with the<br>environment, bypassing the dedicated physics learning as in the former part of<br>this work. In particular, we are interested in the type of tasks that require<br>the agent to reach a given goal state that may be different for every new<br>trial. Thereby we propose a deep reinforcement learning framework that learns<br>policies for stacking tasks which are parametrized by a target structure.<br>

Today's deep learning systems deliver high performance based on end-to-end<br>training. While they deliver strong performance, these systems are hard to<br>interpret. To address this issue, we propose Semantic Bottleneck Networks<br>(SBN): deep networks with semantically interpretable intermediate layers that<br>all downstream results are based on. As a consequence, the analysis on what the<br>final prediction is based on is transparent to the engineer and failure cases<br>and modes can be analyzed and avoided by high-level reasoning. We present a<br>case study on street scene segmentation to demonstrate the feasibility and<br>power of SBN. In particular, we start from a well performing classic deep<br>network which we adapt to house a SB-Layer containing task related semantic<br>concepts (such as object-parts and materials). Importantly, we can recover<br>state of the art performance despite a drastic dimensionality reduction from<br>1000s (non-semantic feature) to 10s (semantic concept) channels. Additionally<br>we show how the activations of the SB-Layer can be used for both the<br>interpretation of failure cases of the network as well as for confidence<br>prediction of the resulting output. For the first time, e.g., we show<br>interpretable segmentation results for most predictions at over 99% accuracy.<br>

Image-to-image (I2I) translation is a pixel-level mapping that requires a<br>large number of paired training data and often suffers from the problems of<br>high diversity and strong category bias in image scenes. In order to tackle<br>these problems, we propose a novel BiLevel (BiL) learning paradigm that<br>alternates the learning of two models, respectively at an instance-specific<br>(IS) and a general-purpose (GP) level. In each scene, the IS model learns to<br>maintain the specific scene attributes. It is initialized by the GP model that<br>learns from all the scenes to obtain the generalizable translation knowledge.<br>This GP initialization gives the IS model an efficient starting point, thus<br>enabling its fast adaptation to the new scene with scarce training data. We<br>conduct extensive I2I translation experiments on human face and street view<br>datasets. Quantitative results validate that our approach can significantly<br>boost the performance of classical I2I translation models, such as PG2 and<br>Pix2Pix. Our visualization results show both higher image quality and more<br>appropriate instance-specific details, e.g., the translated image of a person<br>looks more like that person in terms of identity.<br>

We present a real-time approach for multi-person 3D motion capture at over 30<br>fps using a single RGB camera. It operates in generic scenes and is robust to<br>difficult occlusions both by other people and objects. Our method operates in<br>subsequent stages. The first stage is a convolutional neural network (CNN) that<br>estimates 2D and 3D pose features along with identity assignments for all<br>visible joints of all individuals. We contribute a new architecture for this<br>CNN, called SelecSLS Net, that uses novel selective long and short range skip<br>connections to improve the information flow allowing for a drastically faster<br>network without compromising accuracy. In the second stage, a fully-connected<br>neural network turns the possibly partial (on account of occlusion) 2D pose and<br>3D pose features for each subject into a complete 3D pose estimate per<br>individual. The third stage applies space-time skeletal model fitting to the<br>predicted 2D and 3D pose per subject to further reconcile the 2D and 3D pose,<br>and enforce temporal coherence. Our method returns the full skeletal pose in<br>joint angles for each subject. This is a further key distinction from previous<br>work that neither extracted global body positions nor joint angle results of a<br>coherent skeleton in real time for multi-person scenes. The proposed system<br>runs on consumer hardware at a previously unseen speed of more than 30 fps<br>given 512x320 images as input while achieving state-of-the-art accuracy, which<br>we will demonstrate on a range of challenging real-world scenes.<br>

Modern approaches to pose and body shape estimation have recently achieved<br>strong performance even under challenging real-world conditions. Even from a<br>single image of a clothed person, a realistic looking body shape can be<br>inferred that captures a users' weight group and body shape type well. This<br>opens up a whole spectrum of applications -- in particular in fashion -- where<br>virtual try-on and recommendation systems can make use of these new and<br>automatized cues. However, a realistic depiction of the undressed body is<br>regarded highly private and therefore might not be consented by most people.<br>Hence, we ask if the automatic extraction of such information can be<br>effectively evaded. While adversarial perturbations have been shown to be<br>effective for manipulating the output of machine learning models -- in<br>particular, end-to-end deep learning approaches -- state of the art shape<br>estimation methods are composed of multiple stages. We perform the first<br>investigation of different strategies that can be used to effectively<br>manipulate the automatic shape estimation while preserving the overall<br>appearance of the original image.<br>

Visual search is an important task, and it is part of daily human life. Thus, it has been a long-standing goal in Computer Vision to develop methods aiming at analysing human search intent and preferences. As the target of the search only exists in mind of the person, search intent prediction remains challenging for machine perception. In this thesis, we focus on advancing techniques for search target and preference prediction from implicit human cues. First, we propose a search target inference algorithm from human fixation data recorded during visual search. In contrast to previous work that has focused on individual instances as a search target in a closed world, we propose the first approach to predict the search target in open-world settings by learning the compatibility between observed fixations and potential search targets. Second, we further broaden the scope of search target prediction to categorical classes, such as object categories and attributes. However, state of the art models for categorical recognition, in general, require large amounts of training data, which is prohibitive for gaze data. To address this challenge, we propose a novel Gaze Pooling Layer that integrates gaze information into CNN-based architectures as an attention mechanism – incorporating both spatial and temporal aspects of human gaze behaviour. Third, we go one step further and investigate the feasibility of combining our gaze embedding approach, with the power of generative image models to visually decode, i.e. create a visual representation of, the search target. Forth, for the first time, we studied the effect of body shape on people preferences of outfits. We propose a novel and robust multi-photo approach to estimate the body shapes of each user and build a conditional model of clothing categories given body-shape. We demonstrate that in real-world data, clothing categories and body-shapes are correlated. We show that our approach estimates a realistic looking body shape that captures a user’s weight group and body shape type, even from a single image of a clothed person. However, an accurate depiction of the naked body is considered highly private and therefore, might not be consented by most people. First, we studied the perception of such technology via a user study. Then, in the last part of this thesis, we ask if the automatic extraction of such information can be effectively evaded. In summary, this thesis addresses several different tasks that aims to enable the vision system to analyse human search intent and preferences in real-world scenarios. In particular, the thesis proposes several novel ideas and models in visual search target prediction from human fixation data, for the first time studied the correlation between shape and clothing categories opening a new direction in clothing recommendation systems, and introduces a new topic in privacy and computer vision, aimed at preventing automatic 3D shape extraction from images.

Eye tracking and gaze-based human-computer interfaces have become a practical modality in desktop settings, since remote eye tracking is efficient and affordable. However, remote eye tracking remains constrained to indoor, laboratory-like conditions, in which lighting and user position need to be controlled. Mobile eye tracking has the potential to overcome these limitations and to allow people to move around freely and to use eye tracking on a daily basis during their everyday routine. However, mobile eye tracking currently faces two fundamental challenges that prevent it from being practically usable and that, consequently, have to be addressed before mobile eye tracking can truly be used by everyone: Mobile eye tracking needs to be advanced and made fully functional in unconstrained environments, and it needs to be made socially acceptable. Numerous sensing and analysis methods were initially developed for remote eye tracking and have been successfully applied for decades. Unfortunately, these methods are limited in terms of functionality and correctness, or even unsuitable for application in mobile eye tracking. Therefore, the majority of fundamental definitions, eye tracking methods, and gaze estimation approaches cannot be borrowed from remote eye tracking without adaptation. For example, the definitions of specific eye movements, like classical fixations, need to be extended to mobile settings where natural user and head motion are omnipresent. Corresponding analytical methods need to be adjusted or completely reimplemented based on novel approaches encoding the human gaze behaviour. Apart from these technical challenges, an entirely new, and yet under-explored, topic required for the breakthrough of mobile eye tracking as everyday technology is the overcoming of social obstacles. A first crucial key issue to defuse social objections is the building of acceptance towards mobile eye tracking. Hence, it is essential to replace the bulky appearance of current head-mounted eye trackers with an unobtrusive, appealing, and trendy design. The second high-priority theme of increasing importance for everyone is privacy and its protection, given that research and industry have not focused on or taken care of this problem at all. To establish true confidence, future devices have to find a fine balance between protecting users’ and bystanders’ privacy and attracting and convincing users of their necessity, utility, and potential with useful and beneficial features. The solution of technical challenges and social obstacles is the prerequisite for the development of a variety of novel and exciting applications in order to establish mobile eye tracking as a new paradigm, which ease our everyday life. This thesis addresses core technical challenges of mobile eye tracking that currently prevent it from being widely adopted. Specifically, this thesis proves that 3D data used for the calibration of mobile eye trackers improves gaze estimation and significantly reduces the parallax error. Further, it presents the first effective fixation detection method for head-mounted devices that is robust against the prevalence of user and gaze target motion. In order to achieve social acceptability, this thesis proposes an innovative and unobtrusive design for future mobile eye tracking devices and builds the first prototype with fully frame-embedded eye cameras combined with a calibration-free deep-trained appearance-based gaze estimation approach. To protect users’ and bystanders’ privacy in the presence of head-mounted eye trackers, this thesis presents another first-of-its-kind prototype. It is able to identify privacy-sensitive situations to automatically enable and disable the eye tracker’s first-person camera by means of a mechanical shutter, leveraging the combination of deep scene and eye movement features. Nevertheless, solving technical challenges and social obstacles alone is not sufficient to make mobile eye tracking attractive for the masses. The key to success is the development of convincingly useful, innovative, and essential applications. To extend the protection of users’ privacy on the software side as well, this thesis presents the first privacy-aware VR gaze interface using differential privacy. This method adds noise to recorded eye tracking data so that privacy-sensitive information like a user’s gender or identity is protected without impeding the utility of the data itself. In addition, the first large-scale online survey is conducted to understand users’ concerns with eye tracking. To develop and evaluate novel applications, this thesis presents the first publicly available long-term eye tracking datasets. They are used to show the unsupervised detection of users’ activities from eye movements alone using novel and efficient video-based encoding approaches as well as to propose the first proof-of-concept method to forecast users’ attentive behaviour during everyday mobile interactions from phone-integrated and body-worn sensors. This opens up possibilities for the development of a variety of novel and exciting applications. With more advanced features, accompanied by technological progress and sensor miniaturisation, eye tracking is increasingly integrated into conventional glasses as well as virtual and augmented reality (VR/AR) head-mounted displays, becoming an integral component of mobile interfaces. This thesis paves the way for the development of socially acceptable, privacy-aware, but highly functional mobile eye tracking devices and novel applications, so that mobile eye tracking can develop its full potential to become an everyday technology for everyone.

Adversarial training is the standard to train models robust against<br>adversarial examples. However, especially for complex datasets, adversarial<br>training incurs a significant loss in accuracy and is known to generalize<br>poorly to stronger attacks, e.g., larger perturbations or other threat models.<br>In this paper, we introduce confidence-calibrated adversarial training (CCAT)<br>where the key idea is to enforce that the confidence on adversarial examples<br>decays with their distance to the attacked examples. We show that CCAT<br>preserves better the accuracy of normal training while robustness against<br>adversarial examples is achieved via confidence thresholding, i.e., detecting<br>adversarial examples based on their confidence. Most importantly, in strong<br>contrast to adversarial training, the robustness of CCAT generalizes to larger<br>perturbations and other threat models, not encountered during training. For<br>evaluation, we extend the commonly used robust test error to our detection<br>setting, present an adaptive attack with backtracking and allow the attacker to<br>select, per test example, the worst-case adversarial example from multiple<br>black- and white-box attacks. We present experimental results using $L_\infty$,<br>$L_2$, $L_1$ and $L_0$ attacks on MNIST, SVHN and Cifar10.<br>

While great progress has been made recently in automatic image manipulation,<br>it has been limited to object centric images like faces or structured scene<br>datasets. In this work, we take a step towards general scene-level image<br>editing by developing an automatic interaction-free object removal model. Our<br>model learns to find and remove objects from general scene images using<br>image-level labels and unpaired data in a generative adversarial network (GAN)<br>framework. We achieve this with two key contributions: a two-stage editor<br>architecture consisting of a mask generator and image in-painter that<br>co-operate to remove objects, and a novel GAN based prior for the mask<br>generator that allows us to flexibly incorporate knowledge about object shapes.<br>We experimentally show on two datasets that our method effectively removes a<br>wide variety of objects using weak supervision only<br>

Deep neural perception and control networks have become key com-<br>ponents of self-driving vehicles. User acceptance is likely to benefit from easy-<br>to-interpret textual explanations which allow end-users to understand what trig-<br>gered a particular behavior. Explanations may be triggered by the neural con-<br>troller, namely<br>introspective explanations<br>, or informed by the neural controller’s<br>output, namely<br>rationalizations<br>. We propose a new approach to introspective ex-<br>planations which consists of two parts. First, we use a visual (spatial) attention<br>model to train a convolutional network end-to-end from images to the vehicle<br>control commands,<br>i<br>.<br>e<br>., acceleration and change of course. The controller’s at-<br>tention identifies image regions that potentially influence the network’s output.<br>Second, we use an attention-based video-to-text model to produce textual ex-<br>planations of model actions. The attention maps of controller and explanation<br>model are aligned so that explanations are grounded in the parts of the scene that<br>mattered to the controller. We explore two approaches to attention alignment,<br>strong- and weak-alignment. Finally, we explore a version of our model that<br>generates rationalizations, and compare with introspective explanations on the<br>same video segments. We evaluate these models on a novel driving dataset with<br>ground-truth human explanations, the Berkeley DeepDrive eXplanation (BDD-<br>X) dataset. Code is available at<br>https://github.com/JinkyuKimUCB/explainable-deep-driving