Publications - The Year Before Last

Deep learning models have proven to be successful in a wide
range of machine learning tasks. Yet, they are often highly sensitive to
perturbations on the input data which can lead to incorrect decisions
with high confidence, hampering their deployment for practical
use-cases. Thus, finding architectures that are (more) robust against
perturbations has received much attention in recent years. Just like the
search for well-performing architectures in terms of clean accuracy,
this usually involves a tedious trial-and-error process with one
additional challenge: the evaluation of a network's robustness is
significantly more expensive than its evaluation for clean accuracy.
Thus, the aim of this paper is to facilitate better streamlined research
on architectural design choices with respect to their impact on
robustness as well as, for example, the evaluation of surrogate measures
for robustness. We therefore borrow one of the most commonly considered
search spaces for neural architecture search for image classification,
NAS-Bench-201, which contains a manageable size of 6466 non-isomorphic
network designs. We evaluate all these networks on a range of common
adversarial attacks and corruption types and introduce a database on
neural architecture design and robustness evaluations. We further
present three exemplary use cases of this dataset, in which we (i)
benchmark robustness measurements based on Jacobian and Hessian matrices
for their robustness predictability, (ii) perform neural architecture
search on robust accuracies, and (iii) provide an initial analysis of
how architectural design choices affect robustness. We find that
carefully crafting the topology of a network can have substantial impact
on its robustness, where networks with the same parameter count range in
mean adversarial robust accuracy from 20%-41%.

Local coherence is essential for long-form text generation models. We identify two important aspects of local coherence within the visual storytelling task: (1) the model needs to represent re-occurrences of characters within the image sequence in order to mention them correctly in the story; (2) character representations should enable us to find instances of the same characters and distinguish different characters. In this paper, we propose a loss function inspired by a linguistic theory of coherence for self-supervised learning for image sequence representations. We further propose combining features from an object and a face detector to construct stronger character features. To evaluate input-output relevance that current reference-based metrics don't measure, we propose a character matching metric to check whether the models generate referring expressions correctly for characters in input image sequences. Experiments on a visual story generation dataset show that our proposed features and loss function are effective for generating more coherent and visually grounded stories.

Human performance capture is a highly important computer vision problem with
many applications in movie production and virtual/augmented reality. Many
previous performance capture approaches either required expensive multi-view
setups or did not recover dense space-time coherent geometry with
frame-to-frame correspondences. We propose a novel deep learning approach for
monocular dense human performance capture. Our method is trained in a weakly
supervised manner based on multi-view supervision completely removing the need
for training data with 3D ground truth annotations. The network architecture is
based on two separate networks that disentangle the task into a pose estimation
and a non-rigid surface deformation step. Extensive qualitative and
quantitative evaluations show that our approach outperforms the state of the
art in terms of quality and robustness. This work is an extended version of
DeepCap where we provide more detailed explanations, comparisons and results as
well as applications.

Minimum cost lifted multicut problem is a generalization of the multicut problem and is a means to optimizing a decomposition of a graph w.r.t. both positive and negative edge costs. Its main advantage is that multicut-based formulations do not require the number of components given a priori; instead, it is deduced from the solution. However, the standard multicut cost function is limited to pairwise relationships between nodes, while several important applications either require or can benefit from a higher-order cost function, i.e. hyper-edges. In this paper, we propose a pseudo-boolean formulation for a multiple model fitting problem. It is based on a formulation of any-order minimum cost lifted multicuts, which allows to partition an undirected graph with pairwise connectivity such as to minimize costs defined over any set of hyper-edges. As the proposed formulation is NP-hard and the branch-and-bound algorithm is too slow in practice, we propose an efficient local search algorithm for inference into resulting problems. We demonstrate versatility and effectiveness of our approach in several applications: geometric multiple model fitting, homography and motion estimation, motion segmentation.

Transformers increasingly dominate the machine learning landscape across many
tasks and domains, which increases the importance for understanding their
outputs. While their attention modules provide partial insight into their inner
workings, the attention scores have been shown to be insufficient for
explaining the models as a whole. To address this, we propose B-cos
transformers, which inherently provide holistic explanations for their
decisions. Specifically, we formulate each model component - such as the
multi-layer perceptrons, attention layers, and the tokenisation module - to be
dynamic linear, which allows us to faithfully summarise the entire transformer
via a single linear transform. We apply our proposed design to Vision
Transformers (ViTs) and show that the resulting models, dubbed Bcos-ViTs, are
highly interpretable and perform competitively to baseline ViTs on ImageNet.
Code will be made available soon.

Much of the world's most valued data is stored in relational databases and
data warehouses, where the data is organized into many tables connected by
primary-foreign key relations. However, building machine learning models using
this data is both challenging and time consuming. The core problem is that no
machine learning method is capable of learning on multiple tables
interconnected by primary-foreign key relations. Current methods can only learn
from a single table, so the data must first be manually joined and aggregated
into a single training table, the process known as feature engineering. Feature
engineering is slow, error prone and leads to suboptimal models. Here we
introduce an end-to-end deep representation learning approach to directly learn
on data laid out across multiple tables. We name our approach Relational Deep
Learning (RDL). The core idea is to view relational databases as a temporal,
heterogeneous graph, with a node for each row in each table, and edges
specified by primary-foreign key links. Message Passing Graph Neural Networks
can then automatically learn across the graph to extract representations that
leverage all input data, without any manual feature engineering. Relational
Deep Learning leads to more accurate models that can be built much faster. To
facilitate research in this area, we develop RelBench, a set of benchmark
datasets and an implementation of Relational Deep Learning. The data covers a
wide spectrum, from discussions on Stack Exchange to book reviews on the Amazon
Product Catalog. Overall, we define a new research area that generalizes graph
machine learning and broadens its applicability to a wide set of AI use cases.

Convolutional neural networks encode images through a sequence of
convolutions, normalizations and non-linearities as well as downsampling
operations into potentially strong semantic embeddings. Yet, previous work
showed that even slight mistakes during sampling, leading to aliasing, can be
directly attributed to the networks' lack in robustness. To address such issues
and facilitate simpler and faster adversarial training, [12] recently proposed
FLC pooling, a method for provably alias-free downsampling - in theory. In this
work, we conduct a further analysis through the lens of signal processing and
find that such current pooling methods, which address aliasing in the frequency
domain, are still prone to spectral leakage artifacts. Hence, we propose
aliasing and spectral artifact-free pooling, short ASAP. While only introducing
a few modifications to FLC pooling, networks using ASAP as downsampling method
exhibit higher native robustness against common corruptions, a property that
FLC pooling was missing. ASAP also increases native robustness against
adversarial attacks on high and low resolution data while maintaining similar
clean accuracy or even outperforming the baseline.