2021
Bit Error Robustness for Energy-Efficient DNN Accelerators
D. Stutz, N. Chandramoorthy, M. Hein and B. Schiele
Proceedings of Machine Learning and Systems (MLSys 2021), 2021
(Accepted/in press) D. Stutz, N. Chandramoorthy, M. Hein and B. Schiele
Proceedings of Machine Learning and Systems (MLSys 2021), 2021
Abstract
Deep neural network (DNN) accelerators received considerable attention in
past years due to saved energy compared to mainstream hardware. Low-voltage
operation of DNN accelerators allows to further reduce energy consumption
significantly, however, causes bit-level failures in the memory storing the
quantized DNN weights. In this paper, we show that a combination of robust
fixed-point quantization, weight clipping, and random bit error training
(RandBET) improves robustness against random bit errors in (quantized) DNN
weights significantly. This leads to high energy savings from both low-voltage
operation as well as low-precision quantization. Our approach generalizes
across operating voltages and accelerators, as demonstrated on bit errors from
profiled SRAM arrays. We also discuss why weight clipping alone is already a
quite effective way to achieve robustness against bit errors. Moreover, we
specifically discuss the involved trade-offs regarding accuracy, robustness and
precision: Without losing more than 1% in accuracy compared to a normally
trained 8-bit DNN, we can reduce energy consumption on CIFAR-10 by 20%. Higher
energy savings of, e.g., 30%, are possible at the cost of 2.5% accuracy, even
for 4-bit DNNs.
2020
LoopReg: Self-supervised Learning of Implicit Surface Correspondences, Pose and Shape for 3D Human Mesh Registration
B. L. Bhatnagar, C. Sminchisescu, C. Theobalt and G. Pons-Moll
Advances in Neural Information Processing Systems 33 (NIPS 2020), 2020
B. L. Bhatnagar, C. Sminchisescu, C. Theobalt and G. Pons-Moll
Advances in Neural Information Processing Systems 33 (NIPS 2020), 2020
Neural Unsigned Distance Fields for Implicit Function Learning
J. Chibane, A. Mir and G. Pons-Moll
Advances in Neural Information Processing Systems 33 (NIPS 2020), 2020
J. Chibane, A. Mir and G. Pons-Moll
Advances in Neural Information Processing Systems 33 (NIPS 2020), 2020
Deep Wiener Deconvolution: Wiener Meets Deep Learning for Image Deblurring
J. Dong, S. Roth and B. Schiele
Advances in Neural Information Processing Systems 33 (NIPS 2020), 2020
J. Dong, S. Roth and B. Schiele
Advances in Neural Information Processing Systems 33 (NIPS 2020), 2020
Attribute Prototype Network for Zero-Shot Learning
W. Xu, Y. Xian, J. Wang, B. Schiele and Z. Akata
Advances in Neural Information Processing Systems 33 (NIPS 2020), 2020
W. Xu, Y. Xian, J. Wang, B. Schiele and Z. Akata
Advances in Neural Information Processing Systems 33 (NIPS 2020), 2020
Body Shape Privacy in Images: Understanding Privacy and Preventing Automatic Shape Extraction
H. Sattar, K. Krombholz, G. Pons-Moll and M. Fritz
Computer Vision -- ECCV Workshops 2020, 2020
H. Sattar, K. Krombholz, G. Pons-Moll and M. Fritz
Computer Vision -- ECCV Workshops 2020, 2020
Abstract
Modern approaches to pose and body shape estimation have recently achieved
strong performance even under challenging real-world conditions. Even from a
single image of a clothed person, a realistic looking body shape can be
inferred that captures a users' weight group and body shape type well. This
opens up a whole spectrum of applications -- in particular in fashion -- where
virtual try-on and recommendation systems can make use of these new and
automatized cues. However, a realistic depiction of the undressed body is
regarded highly private and therefore might not be consented by most people.
Hence, we ask if the automatic extraction of such information can be
effectively evaded. While adversarial perturbations have been shown to be
effective for manipulating the output of machine learning models -- in
particular, end-to-end deep learning approaches -- state of the art shape
estimation methods are composed of multiple stages. We perform the first
investigation of different strategies that can be used to effectively
manipulate the automatic shape estimation while preserving the overall
appearance of the original image.
SelfPose: 3D Egocentric Pose Estimation from a Headset Mounted Camera
D. Tome, T. Alldieck, P. Peluse, G. Pons-Moll, L. Agapito, H. Badino and F. de la Torre
IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020
D. Tome, T. Alldieck, P. Peluse, G. Pons-Moll, L. Agapito, H. Badino and F. de la Torre
IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020
Learning Robust Representations via Multi-View Information Bottleneck
M. Federici, A. Dutta, P. Forré, N. Kushman and Z. Akata
International Conference on Learning Representations (ICLR 2020), 2020
M. Federici, A. Dutta, P. Forré, N. Kushman and Z. Akata
International Conference on Learning Representations (ICLR 2020), 2020
Prediction Poisoning: Towards Defenses Against DNN Model Stealing Attacks
T. Orekondy, B. Schiele and M. Fritz
International Conference on Learning Representations (ICLR 2020), 2020
T. Orekondy, B. Schiele and M. Fritz
International Conference on Learning Representations (ICLR 2020), 2020
Semantically Tied Paired Cycle Consistency for Any-Shot Sketch-based Image Retrieval
A. Dutta and Z. Akata
International Journal of Computer Vision, Volume 128, 2020
A. Dutta and Z. Akata
International Journal of Computer Vision, Volume 128, 2020
Diverse and Relevant Visual Storytelling with Scene Graph Embeddings
X. Hong, R. Shetty, A. Sayeed, K. Mehra, V. Demberg and B. Schiele
Proceedings of the 24th Conference on Computational Natural Language Learning (CoNLL 2020), 2020
X. Hong, R. Shetty, A. Sayeed, K. Mehra, V. Demberg and B. Schiele
Proceedings of the 24th Conference on Computational Natural Language Learning (CoNLL 2020), 2020
Lifted Disjoint Paths with Application in Multiple Object Tracking
A. Horňáková, R. Henschel, B. Rosenhahn and P. Swoboda
Proceedings of the 37th International Conference on Machine Learning (ICML 2020), 2020
A. Horňáková, R. Henschel, B. Rosenhahn and P. Swoboda
Proceedings of the 37th International Conference on Machine Learning (ICML 2020), 2020
Confidence-Calibrated Adversarial Training: Generalizing to Unseen Attacks
D. Stutz, M. Hein and B. Schiele
Proceedings of the 37th International Conference on Machine Learning (ICML 2020), 2020
D. Stutz, M. Hein and B. Schiele
Proceedings of the 37th International Conference on Machine Learning (ICML 2020), 2020
A Primal-Dual Solver for Large-Scale Tracking-by-Assignment
S. Haller, M. Prakash, L. Hutschenreiter, T. Pietzsch, C. Rother, F. Jug, P. Swoboda and B. Savchynskyy
Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics (AISTATS 2020), 2020
S. Haller, M. Prakash, L. Hutschenreiter, T. Pietzsch, C. Rother, F. Jug, P. Swoboda and B. Savchynskyy
Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics (AISTATS 2020), 2020
PoseTrackReID: Dataset Description
A. Doering, D. Chen, S. Zhang, B. Schiele and J. Gall
Technical Report, 2020
(arXiv: 2011.06243) A. Doering, D. Chen, S. Zhang, B. Schiele and J. Gall
Technical Report, 2020
Abstract
Current datasets for video-based person re-identification (re-ID) do not
include structural knowledge in form of human pose annotations for the persons
of interest. Nonetheless, pose information is very helpful to disentangle
useful feature information from background or occlusion noise. Especially
real-world scenarios, such as surveillance, contain a lot of occlusions in
human crowds or by obstacles. On the other hand, video-based person re-ID can
benefit other tasks such as multi-person pose tracking in terms of robust
feature matching. For that reason, we present PoseTrackReID, a large-scale
dataset for multi-person pose tracking and video-based person re-ID. With
PoseTrackReID, we want to bridge the gap between person re-ID and multi-person
pose tracking. Additionally, this dataset provides a good benchmark for current
state-of-the-art methods on multi-frame person re-ID.
Analyzing the Dependency of ConvNets on Spatial Information
Y. Fan, Y. Xian, M. M. Losch and B. Schiele
Technical Report, 2020
(arXiv: 2002.01827) Y. Fan, Y. Xian, M. M. Losch and B. Schiele
Technical Report, 2020
Abstract
Intuitively, image classification should profit from using spatial
information. Recent work, however, suggests that this might be overrated in
standard CNNs. In this paper, we are pushing the envelope and aim to further
investigate the reliance on spatial information. We propose spatial shuffling
and GAP+FC to destroy spatial information during both training and testing
phases. Interestingly, we observe that spatial information can be deleted from
later layers with small performance drops, which indicates spatial information
at later layers is not necessary for good performance. For example, test
accuracy of VGG-16 only drops by 0.03% and 2.66% with spatial information
completely removed from the last 30% and 53% layers on CIFAR100, respectively.
Evaluation on several object recognition datasets (CIFAR100, Small-ImageNet,
ImageNet) with a wide range of CNN architectures (VGG16, ResNet50, ResNet152)
shows an overall consistent pattern.
Improved Methods and Analysis for Semantic Image Segmentation
Y. He
PhD Thesis, Universität des Saarlandes, 2020
Y. He
PhD Thesis, Universität des Saarlandes, 2020
Abstract
Modern deep learning has enabled amazing developments of computer vision in recent years (Hinton and Salakhutdinov, 2006; Krizhevsky et al., 2012). As a fundamental task, semantic segmentation aims to predict class labels for each pixel of images, which empowers machines perception of the visual world. In spite of recent successes of fully convolutional networks (Long etal., 2015), several challenges remain to be addressed. In this thesis, we focus on this topic, under different kinds of input formats and various types of scenes. Specifically, our study contains two aspects: (1) Data-driven neural modules for improved performance. (2) Leverage of datasets w.r.t.training systems with higher performances and better data privacy guarantees. In the first part of this thesis, we improve semantic segmentation by designing new modules which are compatible with existing architectures. First, we develop a spatio-temporal data-driven pooling, which brings additional information of data (i.e. superpixels) into neural networks, benefiting the training of neural networks as well as the inference on novel data. We investigate our approach in RGB-D videos for segmenting indoor scenes, where depth provides complementary cues to colors and our model performs particularly well. Second, we design learnable dilated convolutions, which are the extension of standard dilated convolutions, whose dilation factors (Yu and Koltun, 2016) need to be carefully determined by hand to obtain decent performance. We present a method to learn dilation factors together with filter weights of convolutions to avoid a complicated search of dilation factors. We explore extensive studies on challenging street scenes, across various baselines with different complexity as well as several datasets at varying image resolutions. In the second part, we investigate how to utilize expensive training data. First, we start from the generative modelling and study the network architectures and the learning pipeline for generating multiple examples. We aim to improve the diversity of generated examples but also to preserve the comparable quality of the examples. Second, we develop a generative model for synthesizing features of a network. With a mixture of real images and synthetic features, we are able to train a segmentation model with better generalization capability. Our approach is evaluated on different scene parsing tasks to demonstrate the effectiveness of the proposed method. Finally, we study membership inference on the semantic segmentation task. We propose the first membership inference attack system against black-box semantic segmentation models, that tries to infer if a data pair is used as training data or not. From our observations, information on training data is indeed leaking. To mitigate the leakage, we leverage our synthetic features to perform prediction obfuscations, reducing the posterior distribution gaps between a training and a testing set. Consequently, our study provides not only an approach for detecting illegal use of data, but also the foundations for a safer use of semantic segmentation models.
Multicut Optimization Guarantees & Geometry of Lifted Multicuts
J.-H. Lange
PhD Thesis, Universität des Saarlandes, 2020
J.-H. Lange
PhD Thesis, Universität des Saarlandes, 2020
D-NeRF: Neural Radiance Fields for Dynamic Scenes
A. Pumarola, E. Corona, G. Pons-Moll and F. Moreno-Noguer
Technical Report, 2020
(arXiv: 2011.13961) A. Pumarola, E. Corona, G. Pons-Moll and F. Moreno-Noguer
Technical Report, 2020
Abstract
Neural rendering techniques combining machine learning with geometric
reasoning have arisen as one of the most promising approaches for synthesizing
novel views of a scene from a sparse set of images. Among these, stands out the
Neural radiance fields (NeRF), which trains a deep network to map 5D input
coordinates (representing spatial location and viewing direction) into a volume
density and view-dependent emitted radiance. However, despite achieving an
unprecedented level of photorealism on the generated images, NeRF is only
applicable to static scenes, where the same spatial location can be queried
from different images. In this paper we introduce D-NeRF, a method that extends
neural radiance fields to a dynamic domain, allowing to reconstruct and render
novel images of objects under rigid and non-rigid motions from a \emph{single}
camera moving around the scene. For this purpose we consider time as an
additional input to the system, and split the learning process in two main
stages: one that encodes the scene into a canonical space and another that maps
this canonical representation into the deformed scene at a particular time.
Both mappings are simultaneously learned using fully-connected networks. Once
the networks are trained, D-NeRF can render novel images, controlling both the
camera view and the time variable, and thus, the object movement. We
demonstrate the effectiveness of our approach on scenes with objects under
rigid, articulated and non-rigid motions. Code, model weights and the dynamic
scenes dataset will be released.
InfoScrub: Towards Attribute Privacy by Targeted Obfuscation
H.-P. Wang, T. Orekondy and M. Fritz
Technical Report, 2020
(arXiv: 2005.10329) H.-P. Wang, T. Orekondy and M. Fritz
Technical Report, 2020
Abstract
Personal photos of individuals when shared online, apart from exhibiting a
myriad of memorable details, also reveals a wide range of private information
and potentially entails privacy risks (e.g., online harassment, tracking). To
mitigate such risks, it is crucial to study techniques that allow individuals
to limit the private information leaked in visual data. We tackle this problem
in a novel image obfuscation framework: to maximize entropy on inferences over
targeted privacy attributes, while retaining image fidelity. We approach the
problem based on an encoder-decoder style architecture, with two key novelties:
(a) introducing a discriminator to perform bi-directional translation
simultaneously from multiple unpaired domains; (b) predicting an image
interpolation which maximizes uncertainty over a target set of attributes. We
find our approach generates obfuscated images faithful to the original input
images, and additionally increase uncertainty by 6.2$\times$ (or up to 0.85
bits) over the non-obfuscated counterparts.
Learning from Limited Labeled Data - Zero-Shot and Few-Shot Learning
Y. Xian
PhD Thesis, Universität des Saarlandes, 2020
Y. Xian
PhD Thesis, Universität des Saarlandes, 2020
Generalized Many-Way Few-Shot Video Classification
Y. Xian, B. Korbar, M. Douze, B. Schiele, Z. Akata and L. Torresani
Technical Report, 2020
(arXiv: 2007.04755) Y. Xian, B. Korbar, M. Douze, B. Schiele, Z. Akata and L. Torresani
Technical Report, 2020
Abstract
Few-shot learning methods operate in low data regimes. The aim is to learn
with few training examples per class. Although significant progress has been
made in few-shot image classification, few-shot video recognition is relatively
unexplored and methods based on 2D CNNs are unable to learn temporal
information. In this work we thus develop a simple 3D CNN baseline, surpassing
existing methods by a large margin. To circumvent the need of labeled examples,
we propose to leverage weakly-labeled videos from a large dataset using tag
retrieval followed by selecting the best clips with visual similarities,
yielding further improvement. Our results saturate current 5-way benchmarks for
few-shot video classification and therefore we propose a new challenging
benchmark involving more classes and a mixture of classes with varying
supervision.
2019
Modeling Conceptual Understanding in Image Reference Games
R. Corona, S. Alaniz and Z. Akata
Advances in Neural Information Processing Systems 32 (NIPS 2019), 2019
R. Corona, S. Alaniz and Z. Akata
Advances in Neural Information Processing Systems 32 (NIPS 2019), 2019
Combining Generative and Discriminative Models for Hybrid Inference
V. Garcia Satorras, Z. Akata and M. Welling
Advances in Neural Information Processing Systems 32 (NIPS 2019), 2019
V. Garcia Satorras, Z. Akata and M. Welling
Advances in Neural Information Processing Systems 32 (NIPS 2019), 2019
Learning to Self-Train for Semi-Supervised Few-Shot Classification
X. Li, Q. Sun, Y. Liu, Q. Zhou, S. Zheng, T.-S. Chua and B. Schiele
Advances in Neural Information Processing Systems 32 (NIPS 2019), 2019
X. Li, Q. Sun, Y. Liu, Q. Zhou, S. Zheng, T.-S. Chua and B. Schiele
Advances in Neural Information Processing Systems 32 (NIPS 2019), 2019
Conditional Flow Variational Autoencoders for Structured Sequence Prediction
A. Bhattacharyya, M. Hanselmann, M. Fritz, B. Schiele and C.-N. Straehle
Bayesian Deep Learning NeurIPS 2019 Workshop, 2019
A. Bhattacharyya, M. Hanselmann, M. Fritz, B. Schiele and C.-N. Straehle
Bayesian Deep Learning NeurIPS 2019 Workshop, 2019
Emergent Leadership Detection Across Datasets
P. Müller and A. Bulling
ICMI’19, International Conference on Multimodal Interaction, 2019
P. Müller and A. Bulling
ICMI’19, International Conference on Multimodal Interaction, 2019
Abstract
Automatic detection of emergent leaders in small groups from nonverbal
behaviour is a growing research topic in social signal processing but existing
methods were evaluated on single datasets -- an unrealistic assumption for
real-world applications in which systems are required to also work in settings
unseen at training time. It therefore remains unclear whether current methods
for emergent leadership detection generalise to similar but new settings and to
which extent. To overcome this limitation, we are the first to study a
cross-dataset evaluation setting for the emergent leadership detection task. We
provide evaluations for within- and cross-dataset prediction using two current
datasets (PAVIS and MPIIGroupInteraction), as well as an investigation on the
robustness of commonly used feature channels (visual focus of attention, body
pose, facial action units, speaking activity) and online prediction in the
cross-dataset setting. Our evaluations show that using pose and eye contact
based features, cross-dataset prediction is possible with an accuracy of 0.68,
as such providing another important piece of the puzzle towards emergent
leadership detection in the real world.
MAP Inference via Block-Coordinate Frank-Wolfe Algorithm
P. Swoboda and V. Kolmogorov
IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2019), 2019
P. Swoboda and V. Kolmogorov
IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2019), 2019
Abstract
When labeled training data is scarce, a promising data augmentation approach is to generate visual features of unknown classes using their attributes. To learn the class conditional distribution of CNN features, these models rely on pairs of image features and class attributes. Hence, they can not make use of the abundance of unlabeled data samples. In this paper, we tackle any-shot learning problems i.e. zero-shot and few-shot, in a unified feature generating framework that operates in both inductive and transductive learning settings. We develop a conditional generative model that combines the strength of VAE and GANs and in addition, via an unconditional discriminator, learns the marginal feature distribution of unlabeled images. We empirically show that our model learns highly discriminative CNN features for five datasets, i.e. CUB, SUN, AWA and ImageNet, and establish a new state-of-the-art in any-shot learning, i.e. inductive and transductive (generalized) zero- and few-shot learning settings. We also demonstrate that our learned features are interpretable: we visualize them by inverting them back to the pixel space and we explain them by generating textual arguments of why they are associated with a certain label.
f-VAEGAN-D2: A Feature Generating Framework for Any-Shot Learning
Y. Xian, S. Sharma, B. Schiele and Z. Akata
IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2019), 2019
Y. Xian, S. Sharma, B. Schiele and Z. Akata
IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2019), 2019
Abstract
When labeled training data is scarce, a promising data augmentation approach is to generate visual features of unknown classes using their attributes. To learn the class conditional distribution of CNN features, these models rely on pairs of image features and class attributes. Hence, they can not make use of the abundance of unlabeled data samples. In this paper, we tackle any-shot learning problems i.e. zero-shot and few-shot, in a unified feature generating framework that operates in both inductive and transductive learning settings. We develop a conditional generative model that combines the strength of VAE and GANs and in addition, via an unconditional discriminator, learns the marginal feature distribution of unlabeled images. We empirically show that our model learns highly discriminative CNN features for five datasets, i.e. CUB, SUN, AWA and ImageNet, and establish a new state-of-the-art in any-shot learning, i.e. inductive and transductive (generalized) zero- and few-shot learning settings. We also demonstrate that our learned features are interpretable: we visualize them by inverting them back to the pixel space and we explain them by generating textual arguments of why they are associated with a certain label.
Zero-shot Learning - A Comprehensive Evaluation of the Good, the Bad and the Ugly
Y. Xian, C. H. Lampert, B. Schiele and Z. Akata
IEEE Transactions on Pattern Analysis and Machine Intelligence, Volume 41, Number 9, 2019
Y. Xian, C. H. Lampert, B. Schiele and Z. Akata
IEEE Transactions on Pattern Analysis and Machine Intelligence, Volume 41, Number 9, 2019
Abstract
Due to the importance of zero-shot learning, i.e. classifying images where
there is a lack of labeled training data, the number of proposed approaches has
recently increased steadily. We argue that it is time to take a step back and
to analyze the status quo of the area. The purpose of this paper is three-fold.
First, given the fact that there is no agreed upon zero-shot learning
benchmark, we first define a new benchmark by unifying both the evaluation
protocols and data splits of publicly available datasets used for this task.
This is an important contribution as published results are often not comparable
and sometimes even flawed due to, e.g. pre-training on zero-shot test classes.
Moreover, we propose a new zero-shot learning dataset, the Animals with
Attributes 2 (AWA2) dataset which we make publicly available both in terms of
image features and the images themselves. Second, we compare and analyze a
significant number of the state-of-the-art methods in depth, both in the
classic zero-shot setting but also in the more realistic generalized zero-shot
setting. Finally, we discuss in detail the limitations of the current status of
the area which can be taken as a basis for advancing it.
Tex2Shape: Detailed Full Human Body Geometry from a Single Image
T. Alldieck, G. Pons-Moll, C. Theobalt and M. A. Magnor
International Conference on Computer Vision (ICCV 2019), 2019
T. Alldieck, G. Pons-Moll, C. Theobalt and M. A. Magnor
International Conference on Computer Vision (ICCV 2019), 2019
Abstract
We present a simple yet effective method to infer detailed full human body
shape from only a single photograph. Our model can infer full-body shape
including face, hair, and clothing including wrinkles at interactive
frame-rates. Results feature details even on parts that are occluded in the
input image. Our main idea is to turn shape regression into an aligned
image-to-image translation problem. The input to our method is a partial
texture map of the visible region obtained from off-the-shelf methods. From a
partial texture, we estimate detailed normal and vector displacement maps,
which can be applied to a low-resolution smooth body model to add detail and
clothing. Despite being trained purely with synthetic data, our model
generalizes well to real-world photographs. Numerous results demonstrate the
versatility and robustness of our method.
Bayesian Prediction of Future Street Scenes using Synthetic Likelihoods
A. Bhattacharyya, M. Fritz and B. Schiele
International Conference on Learning Representations (ICLR 2019), 2019
(Accepted/in press) A. Bhattacharyya, M. Fritz and B. Schiele
International Conference on Learning Representations (ICLR 2019), 2019
Lucid Data Dreaming for Video Object Segmentation
A. Khoreva, R. Benenson, E. Ilg, T. Brox and B. Schiele
International Journal of Computer Vision, Volume 127, Number 9, 2019
A. Khoreva, R. Benenson, E. Ilg, T. Brox and B. Schiele
International Journal of Computer Vision, Volume 127, Number 9, 2019
Moment-to-Moment Detection of Internal Thought from Eye Vergence Behaviour
M. X. Huang, J. Li, G. Ngai, H. V. Leong and A. Bulling
MM’19, 27th ACM International Conference on Multimedia, 2019
M. X. Huang, J. Li, G. Ngai, H. V. Leong and A. Bulling
MM’19, 27th ACM International Conference on Multimedia, 2019
Abstract
Internal thought refers to the process of directing attention away from a
primary visual task to internal cognitive processing. Internal thought is a
pervasive mental activity and closely related to primary task performance. As
such, automatic detection of internal thought has significant potential for
user modelling in intelligent interfaces, particularly for e-learning
applications. Despite the close link between the eyes and the human mind, only
a few studies have investigated vergence behaviour during internal thought and
none has studied moment-to-moment detection of internal thought from gaze.
While prior studies relied on long-term data analysis and required a large
number of gaze characteristics, we describe a novel method that is
computationally light-weight and that only requires eye vergence information
that is readily available from binocular eye trackers. We further propose a
novel paradigm to obtain ground truth internal thought annotations that
exploits human blur perception. We evaluate our method for three increasingly
challenging detection tasks: (1) during a controlled math-solving task, (2)
during natural viewing of lecture videos, and (3) during daily activities, such
as coding, browsing, and reading. Results from these evaluations demonstrate
the performance and robustness of vergence-based detection of internal thought
and, as such, open up new directions for research on interfaces that adapt to
shifts of mental attention.
SacCalib: Reducing Calibration Distortion for Stationary Eye Trackers Using Saccadic Eye Movements
M. X. Huang and A. Bulling
Proceedings ETRA 2019, 2019
M. X. Huang and A. Bulling
Proceedings ETRA 2019, 2019
Abstract
Recent methods to automatically calibrate stationary eye trackers were shown
to effectively reduce inherent calibration distortion. However, these methods
require additional information, such as mouse clicks or on-screen content. We
propose the first method that only requires users' eye movements to reduce
calibration distortion in the background while users naturally look at an
interface. Our method exploits that calibration distortion makes straight
saccade trajectories appear curved between the saccadic start and end points.
We show that this curving effect is systematic and the result of distorted gaze
projection plane. To mitigate calibration distortion, our method undistorts
this plane by straightening saccade trajectories using image warping. We show
that this approach improves over the common six-point calibration and is
promising for reducing distortion. As such, it provides a non-intrusive
solution to alleviating accuracy decrease of eye tracker during long-term use.
Gradient-Leaks: Understanding Deanonymization in Federated Learning
T. Orekondy, S. J. Oh, Y. Zhang, B. Schiele and M. Fritz
The 2nd International Workshop on Federated Learning for Data Privacy and Confidentiality (in Conjunction with NeurIPS 2019) (FL-NeurIPS 2019), 2019
(Accepted/in press) T. Orekondy, S. J. Oh, Y. Zhang, B. Schiele and M. Fritz
The 2nd International Workshop on Federated Learning for Data Privacy and Confidentiality (in Conjunction with NeurIPS 2019) (FL-NeurIPS 2019), 2019
Bottleneck Potentials in Markov Random Fields
A. Abbas and P. Swoboda
Technical Report, 2019
(arXiv: 1904.08080) A. Abbas and P. Swoboda
Technical Report, 2019
Abstract
We consider general discrete Markov Random Fields(MRFs) with additional
bottleneck potentials which penalize the maximum (instead of the sum) over
local potential value taken by the MRF-assignment. Bottleneck potentials or
analogous constructions have been considered in (i) combinatorial optimization
(e.g. bottleneck shortest path problem, the minimum bottleneck spanning tree
problem, bottleneck function minimization in greedoids), (ii) inverse problems
with $L_{\infty}$-norm regularization, and (iii) valued constraint satisfaction
on the $(\min,\max)$-pre-semirings. Bottleneck potentials for general discrete
MRFs are a natural generalization of the above direction of modeling work to
Maximum-A-Posteriori (MAP) inference in MRFs. To this end, we propose MRFs
whose objective consists of two parts: terms that factorize according to (i)
$(\min,+)$, i.e. potentials as in plain MRFs, and (ii) $(\min,\max)$, i.e.
bottleneck potentials. To solve the ensuing inference problem, we propose
high-quality relaxations and efficient algorithms for solving them. We
empirically show efficacy of our approach on large scale seismic horizon
tracking problems.
“Best-of-Many-Samples” Distribution Matching
A. Bhattacharyya, M. Fritz and B. Schiele
Technical Report, 2019
(arXiv: 1909.12598) A. Bhattacharyya, M. Fritz and B. Schiele
Technical Report, 2019
Abstract
Generative Adversarial Networks (GANs) can achieve state-of-the-art sample
quality in generative modelling tasks but suffer from the mode collapse
problem. Variational Autoencoders (VAE) on the other hand explicitly maximize a
reconstruction-based data log-likelihood forcing it to cover all modes, but
suffer from poorer sample quality. Recent works have proposed hybrid VAE-GAN
frameworks which integrate a GAN-based synthetic likelihood to the VAE
objective to address both the mode collapse and sample quality issues, with
limited success. This is because the VAE objective forces a trade-off between
the data log-likelihood and divergence to the latent prior. The synthetic
likelihood ratio term also shows instability during training. We propose a
novel objective with a "Best-of-Many-Samples" reconstruction cost and a stable
direct estimate of the synthetic likelihood. This enables our hybrid VAE-GAN
framework to achieve high data log-likelihood and low divergence to the latent
prior at the same time and shows significant improvement over both hybrid
VAE-GANS and plain GANs in mode coverage and quality.
SampleFix: Learning to Correct Programs by Sampling Diverse Fixes
H. Hajipour, A. Bhattacharyya and M. Fritz
Technical Report, 2019
(arXiv: 1906.10502) H. Hajipour, A. Bhattacharyya and M. Fritz
Technical Report, 2019
Abstract
Automatic program correction is an active topic of research, which holds the
potential of dramatically improving productivity of programmers during the
software development process and correctness of software in general. Recent
advances in machine learning, deep learning and NLP have rekindled the hope to
eventually fully automate the process of repairing programs. A key challenge is
ambiguity, as multiple codes -- or fixes -- can implement the same
functionality. In addition, datasets by nature fail to capture the variance
introduced by such ambiguities. Therefore, we propose a deep generative model
to automatically correct programming errors by learning a distribution of
potential fixes. Our model is formulated as a deep conditional variational
autoencoder that samples diverse fixes for the given erroneous programs. In
order to account for ambiguity and inherent lack of representative datasets, we
propose a novel regularizer to encourage the model to generate diverse fixes.
Our evaluations on common programming errors show for the first time the
generation of diverse fixes and strong improvements over the state-of-the-art
approaches by fixing up to 65% of the mistakes.
LCC: Learning to Customize and Combine Neural Networks for Few-Shot Learning
Y. Liu, Q. Sun, A.-A. Liu, Y. Su, B. Schiele and T.-S. Chua
Technical Report, 2019
(arXiv: 1904.08479) Y. Liu, Q. Sun, A.-A. Liu, Y. Su, B. Schiele and T.-S. Chua
Technical Report, 2019
Abstract
Meta-learning has been shown to be an effective strategy for few-shot
learning. The key idea is to leverage a large number of similar few-shot tasks
in order to meta-learn how to best initiate a (single) base-learner for novel
few-shot tasks. While meta-learning how to initialize a base-learner has shown
promising results, it is well known that hyperparameter settings such as the
learning rate and the weighting of the regularization term are important to
achieve best performance. We thus propose to also meta-learn these
hyperparameters and in fact learn a time- and layer-varying scheme for learning
a base-learner on novel tasks. Additionally, we propose to learn not only a
single base-learner but an ensemble of several base-learners to obtain more
robust results. While ensembles of learners have shown to improve performance
in various settings, this is challenging for few-shot learning tasks due to the
limited number of training samples. Therefore, our approach also aims to
meta-learn how to effectively combine several base-learners. We conduct
extensive experiments and report top performance for five-class few-shot
recognition tasks on two challenging benchmarks: miniImageNet and
Fewshot-CIFAR100 (FC100).
Learning Manipulation under Physics Constraints with Visual Perception
W. Li, A. Leonardis, J. Bohg and M. Fritz
Technical Report, 2019
(arXiv: 1904.09860) W. Li, A. Leonardis, J. Bohg and M. Fritz
Technical Report, 2019
Abstract
Understanding physical phenomena is a key competence that enables humans and
animals to act and interact under uncertain perception in previously unseen
environments containing novel objects and their configurations. In this work,
we consider the problem of autonomous block stacking and explore solutions to
learning manipulation under physics constraints with visual perception inherent
to the task. Inspired by the intuitive physics in humans, we first present an
end-to-end learning-based approach to predict stability directly from
appearance, contrasting a more traditional model-based approach with explicit
3D representations and physical simulation. We study the model's behavior
together with an accompanied human subject test. It is then integrated into a
real-world robotic system to guide the placement of a single wood block into
the scene without collapsing existing tower structure. To further automate the
process of consecutive blocks stacking, we present an alternative approach
where the model learns the physics constraint through the interaction with the
environment, bypassing the dedicated physics learning as in the former part of
this work. In particular, we are interested in the type of tasks that require
the agent to reach a given goal state that may be different for every new
trial. Thereby we propose a deep reinforcement learning framework that learns
policies for stacking tasks which are parametrized by a target structure.
Interpretability Beyond Classification Output: Semantic Bottleneck Networks
M. Losch, M. Fritz and B. Schiele
Technical Report, 2019
(arXiv: 1907.10882) M. Losch, M. Fritz and B. Schiele
Technical Report, 2019
Abstract
Today's deep learning systems deliver high performance based on end-to-end
training. While they deliver strong performance, these systems are hard to
interpret. To address this issue, we propose Semantic Bottleneck Networks
(SBN): deep networks with semantically interpretable intermediate layers that
all downstream results are based on. As a consequence, the analysis on what the
final prediction is based on is transparent to the engineer and failure cases
and modes can be analyzed and avoided by high-level reasoning. We present a
case study on street scene segmentation to demonstrate the feasibility and
power of SBN. In particular, we start from a well performing classic deep
network which we adapt to house a SB-Layer containing task related semantic
concepts (such as object-parts and materials). Importantly, we can recover
state of the art performance despite a drastic dimensionality reduction from
1000s (non-semantic feature) to 10s (semantic concept) channels. Additionally
we show how the activations of the SB-Layer can be used for both the
interpretation of failure cases of the network as well as for confidence
prediction of the resulting output. For the first time, e.g., we show
interpretable segmentation results for most predictions at over 99% accuracy.
A Novel BiLevel Paradigm for Image-to-Image Translation
L. Ma, Q. Sun, B. Schiele and L. Van Gool
Technical Report, 2019
(arXiv: 1904.09028) L. Ma, Q. Sun, B. Schiele and L. Van Gool
Technical Report, 2019
Abstract
Image-to-image (I2I) translation is a pixel-level mapping that requires a
large number of paired training data and often suffers from the problems of
high diversity and strong category bias in image scenes. In order to tackle
these problems, we propose a novel BiLevel (BiL) learning paradigm that
alternates the learning of two models, respectively at an instance-specific
(IS) and a general-purpose (GP) level. In each scene, the IS model learns to
maintain the specific scene attributes. It is initialized by the GP model that
learns from all the scenes to obtain the generalizable translation knowledge.
This GP initialization gives the IS model an efficient starting point, thus
enabling its fast adaptation to the new scene with scarce training data. We
conduct extensive I2I translation experiments on human face and street view
datasets. Quantitative results validate that our approach can significantly
boost the performance of classical I2I translation models, such as PG2 and
Pix2Pix. Our visualization results show both higher image quality and more
appropriate instance-specific details, e.g., the translated image of a person
looks more like that person in terms of identity.
XNect: Real-time Multi-person 3D Human Pose Estimation with a Single RGB Camera
D. Mehta, O. Sotnychenko, F. Mueller, W. Xu, M. Elgharib, P. Fua, H.-P. Seidel, H. Rhodin, G. Pons-Moll and C. Theobalt
Technical Report, 2019
(arXiv: 1907.00837) D. Mehta, O. Sotnychenko, F. Mueller, W. Xu, M. Elgharib, P. Fua, H.-P. Seidel, H. Rhodin, G. Pons-Moll and C. Theobalt
Technical Report, 2019
Abstract
We present a real-time approach for multi-person 3D motion capture at over 30
fps using a single RGB camera. It operates in generic scenes and is robust to
difficult occlusions both by other people and objects. Our method operates in
subsequent stages. The first stage is a convolutional neural network (CNN) that
estimates 2D and 3D pose features along with identity assignments for all
visible joints of all individuals. We contribute a new architecture for this
CNN, called SelecSLS Net, that uses novel selective long and short range skip
connections to improve the information flow allowing for a drastically faster
network without compromising accuracy. In the second stage, a fully-connected
neural network turns the possibly partial (on account of occlusion) 2D pose and
3D pose features for each subject into a complete 3D pose estimate per
individual. The third stage applies space-time skeletal model fitting to the
predicted 2D and 3D pose per subject to further reconcile the 2D and 3D pose,
and enforce temporal coherence. Our method returns the full skeletal pose in
joint angles for each subject. This is a further key distinction from previous
work that neither extracted global body positions nor joint angle results of a
coherent skeleton in real time for multi-person scenes. The proposed system
runs on consumer hardware at a previously unseen speed of more than 30 fps
given 512x320 images as input while achieving state-of-the-art accuracy, which
we will demonstrate on a range of challenging real-world scenes.
Updates-Leak: Data Set Inference and Reconstruction Attacks in Online Learning
A. M. G. Salem, A. Bhattacharyya, M. Backes, M. Fritz and Y. Zhang
Technical Report, 2019
(arXiv: 1904.01067) A. M. G. Salem, A. Bhattacharyya, M. Backes, M. Fritz and Y. Zhang
Technical Report, 2019
Abstract
Machine learning (ML) has progressed rapidly during the past decade and the
major factor that drives such development is the unprecedented large-scale
data. As data generation is a continuous process, this leads to ML service
providers updating their models frequently with newly-collected data in an
online learning scenario. In consequence, if an ML model is queried with the
same set of data samples at two different points in time, it will provide
different results.
In this paper, we investigate whether the change in the output of a black-box
ML model before and after being updated can leak information of the dataset
used to perform the update. This constitutes a new attack surface against
black-box ML models and such information leakage severely damages the
intellectual property and data privacy of the ML model owner/provider. In
contrast to membership inference attacks, we use an encoder-decoder formulation
that allows inferring diverse information ranging from detailed characteristics
to full reconstruction of the dataset. Our new attacks are facilitated by
state-of-the-art deep learning techniques. In particular, we propose a hybrid
generative model (BM-GAN) that is based on generative adversarial networks
(GANs) but includes a reconstructive loss that allows generating accurate
samples. Our experiments show effective prediction of dataset characteristics
and even full reconstruction in challenging conditions.
Intents and Preferences Prediction Based on Implicit Human Cues
H. Sattar
PhD Thesis, Universität des Saarlandes, 2019
H. Sattar
PhD Thesis, Universität des Saarlandes, 2019
Abstract
Visual search is an important task, and it is part of daily human life. Thus, it has been a long-standing goal in Computer Vision to develop methods aiming at analysing human search intent and preferences. As the target of the search only exists in mind of the person, search intent prediction remains challenging for machine perception. In this thesis, we focus on advancing techniques for search target and preference prediction from implicit human cues. First, we propose a search target inference algorithm from human fixation data recorded during visual search. In contrast to previous work that has focused on individual instances as a search target in a closed world, we propose the first approach to predict the search target in open-world settings by learning the compatibility between observed fixations and potential search targets. Second, we further broaden the scope of search target prediction to categorical classes, such as object categories and attributes. However, state of the art models for categorical recognition, in general, require large amounts of training data, which is prohibitive for gaze data. To address this challenge, we propose a novel Gaze Pooling Layer that integrates gaze information into CNN-based architectures as an attention mechanism – incorporating both spatial and temporal aspects of human gaze behaviour. Third, we go one step further and investigate the feasibility of combining our gaze embedding approach, with the power of generative image models to visually decode, i.e. create a visual representation of, the search target. Forth, for the first time, we studied the effect of body shape on people preferences of outfits. We propose a novel and robust multi-photo approach to estimate the body shapes of each user and build a conditional model of clothing categories given body-shape. We demonstrate that in real-world data, clothing categories and body-shapes are correlated. We show that our approach estimates a realistic looking body shape that captures a user’s weight group and body shape type, even from a single image of a clothed person. However, an accurate depiction of the naked body is considered highly private and therefore, might not be consented by most people. First, we studied the perception of such technology via a user study. Then, in the last part of this thesis, we ask if the automatic extraction of such information can be effectively evaded. In summary, this thesis addresses several different tasks that aims to enable the vision system to analyse human search intent and preferences in real-world scenarios. In particular, the thesis proposes several novel ideas and models in visual search target prediction from human fixation data, for the first time studied the correlation between shape and clothing categories opening a new direction in clothing recommendation systems, and introduces a new topic in privacy and computer vision, aimed at preventing automatic 3D shape extraction from images.
Shape Evasion: Preventing Body Shape Inference of Multi-Stage Approaches
H. Sattar, K. Krombholz, G. Pons-Moll and M. Fritz
Technical Report, 2019
(arXiv: 1905.11503) H. Sattar, K. Krombholz, G. Pons-Moll and M. Fritz
Technical Report, 2019
Abstract
Modern approaches to pose and body shape estimation have recently achieved
strong performance even under challenging real-world conditions. Even from a
single image of a clothed person, a realistic looking body shape can be
inferred that captures a users' weight group and body shape type well. This
opens up a whole spectrum of applications -- in particular in fashion -- where
virtual try-on and recommendation systems can make use of these new and
automatized cues. However, a realistic depiction of the undressed body is
regarded highly private and therefore might not be consented by most people.
Hence, we ask if the automatic extraction of such information can be
effectively evaded. While adversarial perturbations have been shown to be
effective for manipulating the output of machine learning models -- in
particular, end-to-end deep learning approaches -- state of the art shape
estimation methods are composed of multiple stages. We perform the first
investigation of different strategies that can be used to effectively
manipulate the automatic shape estimation while preserving the overall
appearance of the original image.
Mobile Eye Tracking for Everyone
J. Steil
PhD Thesis, Universität des Saarlandes, 2019
J. Steil
PhD Thesis, Universität des Saarlandes, 2019
Abstract
Eye tracking and gaze-based human-computer interfaces have become a practical modality in desktop settings, since remote eye tracking is efficient and affordable. However, remote eye tracking remains constrained to indoor, laboratory-like conditions, in which lighting and user position need to be controlled. Mobile eye tracking has the potential to overcome these limitations and to allow people to move around freely and to use eye tracking on a daily basis during their everyday routine. However, mobile eye tracking currently faces two fundamental challenges that prevent it from being practically usable and that, consequently, have to be addressed before mobile eye tracking can truly be used by everyone: Mobile eye tracking needs to be advanced and made fully functional in unconstrained environments, and it needs to be made socially acceptable. Numerous sensing and analysis methods were initially developed for remote eye tracking and have been successfully applied for decades. Unfortunately, these methods are limited in terms of functionality and correctness, or even unsuitable for application in mobile eye tracking. Therefore, the majority of fundamental definitions, eye tracking methods, and gaze estimation approaches cannot be borrowed from remote eye tracking without adaptation. For example, the definitions of specific eye movements, like classical fixations, need to be extended to mobile settings where natural user and head motion are omnipresent. Corresponding analytical methods need to be adjusted or completely reimplemented based on novel approaches encoding the human gaze behaviour. Apart from these technical challenges, an entirely new, and yet under-explored, topic required for the breakthrough of mobile eye tracking as everyday technology is the overcoming of social obstacles. A first crucial key issue to defuse social objections is the building of acceptance towards mobile eye tracking. Hence, it is essential to replace the bulky appearance of current head-mounted eye trackers with an unobtrusive, appealing, and trendy design. The second high-priority theme of increasing importance for everyone is privacy and its protection, given that research and industry have not focused on or taken care of this problem at all. To establish true confidence, future devices have to find a fine balance between protecting users’ and bystanders’ privacy and attracting and convincing users of their necessity, utility, and potential with useful and beneficial features. The solution of technical challenges and social obstacles is the prerequisite for the development of a variety of novel and exciting applications in order to establish mobile eye tracking as a new paradigm, which ease our everyday life. This thesis addresses core technical challenges of mobile eye tracking that currently prevent it from being widely adopted. Specifically, this thesis proves that 3D data used for the calibration of mobile eye trackers improves gaze estimation and significantly reduces the parallax error. Further, it presents the first effective fixation detection method for head-mounted devices that is robust against the prevalence of user and gaze target motion. In order to achieve social acceptability, this thesis proposes an innovative and unobtrusive design for future mobile eye tracking devices and builds the first prototype with fully frame-embedded eye cameras combined with a calibration-free deep-trained appearance-based gaze estimation approach. To protect users’ and bystanders’ privacy in the presence of head-mounted eye trackers, this thesis presents another first-of-its-kind prototype. It is able to identify privacy-sensitive situations to automatically enable and disable the eye tracker’s first-person camera by means of a mechanical shutter, leveraging the combination of deep scene and eye movement features. Nevertheless, solving technical challenges and social obstacles alone is not sufficient to make mobile eye tracking attractive for the masses. The key to success is the development of convincingly useful, innovative, and essential applications. To extend the protection of users’ privacy on the software side as well, this thesis presents the first privacy-aware VR gaze interface using differential privacy. This method adds noise to recorded eye tracking data so that privacy-sensitive information like a user’s gender or identity is protected without impeding the utility of the data itself. In addition, the first large-scale online survey is conducted to understand users’ concerns with eye tracking. To develop and evaluate novel applications, this thesis presents the first publicly available long-term eye tracking datasets. They are used to show the unsupervised detection of users’ activities from eye movements alone using novel and efficient video-based encoding approaches as well as to propose the first proof-of-concept method to forecast users’ attentive behaviour during everyday mobile interactions from phone-integrated and body-worn sensors. This opens up possibilities for the development of a variety of novel and exciting applications. With more advanced features, accompanied by technological progress and sensor miniaturisation, eye tracking is increasingly integrated into conventional glasses as well as virtual and augmented reality (VR/AR) head-mounted displays, becoming an integral component of mobile interfaces. This thesis paves the way for the development of socially acceptable, privacy-aware, but highly functional mobile eye tracking devices and novel applications, so that mobile eye tracking can develop its full potential to become an everyday technology for everyone.
Confidence-Calibrated Adversarial Training and Detection: More Robust Models Generalizing Beyond the Attack Used During Training
D. Stutz, M. Hein and B. Schiele
Technical Report, 2019
(arXiv: 1910.06259) D. Stutz, M. Hein and B. Schiele
Technical Report, 2019
Abstract
Adversarial training is the standard to train models robust against
adversarial examples. However, especially for complex datasets, adversarial
training incurs a significant loss in accuracy and is known to generalize
poorly to stronger attacks, e.g., larger perturbations or other threat models.
In this paper, we introduce confidence-calibrated adversarial training (CCAT)
where the key idea is to enforce that the confidence on adversarial examples
decays with their distance to the attacked examples. We show that CCAT
preserves better the accuracy of normal training while robustness against
adversarial examples is achieved via confidence thresholding, i.e., detecting
adversarial examples based on their confidence. Most importantly, in strong
contrast to adversarial training, the robustness of CCAT generalizes to larger
perturbations and other threat models, not encountered during training. For
evaluation, we extend the commonly used robust test error to our detection
setting, present an adaptive attack with backtracking and allow the attacker to
select, per test example, the worst-case adversarial example from multiple
black- and white-box attacks. We present experimental results using $L_\infty$,
$L_2$, $L_1$ and $L_0$ attacks on MNIST, SVHN and Cifar10.
2018
Sequential Attacks on Agents for Long-Term Adversarial Goals
E. Tretschk, S. J. Oh and M. Fritz
2. ACM Computer Science in Cars Symposium (CSCS 2018), 2018
E. Tretschk, S. J. Oh and M. Fritz
2. ACM Computer Science in Cars Symposium (CSCS 2018), 2018
Adversarial Scene Editing: Automatic Object Removal from Weak Supervision
R. Shetty, M. Fritz and B. Schiele
Advances in Neural Information Processing Systems 31, 2018
R. Shetty, M. Fritz and B. Schiele
Advances in Neural Information Processing Systems 31, 2018
Abstract
While great progress has been made recently in automatic image manipulation,
it has been limited to object centric images like faces or structured scene
datasets. In this work, we take a step towards general scene-level image
editing by developing an automatic interaction-free object removal model. Our
model learns to find and remove objects from general scene images using
image-level labels and unpaired data in a generative adversarial network (GAN)
framework. We achieve this with two key contributions: a two-stage editor
architecture consisting of a mask generator and image in-painter that
co-operate to remove objects, and a novel GAN based prior for the mask
generator that allows us to flexibly incorporate knowledge about object shapes.
We experimentally show on two datasets that our method effectively removes a
wide variety of objects using weak supervision only
Unsupervised Learning of Shape and Pose with Differentiable Point Clouds
E. Insafutdinov and A. Dosovitskiy
Advances in Neural Information Processing Systems 31 (NIPS 2018), 2018
E. Insafutdinov and A. Dosovitskiy
Advances in Neural Information Processing Systems 31 (NIPS 2018), 2018
JAMI: Fast Computation of Conditional Mutual Information for ceRNA Network Analysis
A. Horňáková, M. List, J. Vreeken and M. H. Schulz
Bioinformatics, Volume 34, Number 17, 2018
A. Horňáková, M. List, J. Vreeken and M. H. Schulz
Bioinformatics, Volume 34, Number 17, 2018
Textual Explanations for Self-Driving Vehicles
J. Kim, A. Rohrbach, T. Darrell, J. Canny and Z. Akata
Computer Vision -- ECCV 2018, 2018
J. Kim, A. Rohrbach, T. Darrell, J. Canny and Z. Akata
Computer Vision -- ECCV 2018, 2018
Abstract
Deep neural perception and control networks have become key com-
ponents of self-driving vehicles. User acceptance is likely to benefit from easy-
to-interpret textual explanations which allow end-users to understand what trig-
gered a particular behavior. Explanations may be triggered by the neural con-
troller, namely
introspective explanations
, or informed by the neural controller’s
output, namely
rationalizations
. We propose a new approach to introspective ex-
planations which consists of two parts. First, we use a visual (spatial) attention
model to train a convolutional network end-to-end from images to the vehicle
control commands,
i
.
e
., acceleration and change of course. The controller’s at-
tention identifies image regions that potentially influence the network’s output.
Second, we use an attention-based video-to-text model to produce textual ex-
planations of model actions. The attention maps of controller and explanation
model are aligned so that explanations are grounded in the parts of the scene that
mattered to the controller. We explore two approaches to attention alignment,
strong- and weak-alignment. Finally, we explore a version of our model that
generates rationalizations, and compare with introspective explanations on the
same video segments. We evaluate these models on a novel driving dataset with
ground-truth human explanations, the Berkeley DeepDrive eXplanation (BDD-
X) dataset. Code is available at
https://github.com/JinkyuKimUCB/explainable-deep-driving
A Vision-grounded Dataset for Predicting Typical Locations for Verbs
N. Mukuze, A. Rohrbach, V. Demberg and B. Schiele
Eleventh International Conference on Language Resources and Evaluation (LREC 2018), 2018
N. Mukuze, A. Rohrbach, V. Demberg and B. Schiele
Eleventh International Conference on Language Resources and Evaluation (LREC 2018), 2018
Eye Movements During Everyday Behavior Predict Personality Traits
S. Hoppe, T. Loetscher, S. Morey and A. Bulling
Frontiers in Human Neuroscience, Volume 12, 2018
S. Hoppe, T. Loetscher, S. Morey and A. Bulling
Frontiers in Human Neuroscience, Volume 12, 2018
Learning to Refine Human Pose Estimation
M. Fieraru, A. Khoreva, L. Pishchulin and B. Schiele
IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW 2018), 2018
M. Fieraru, A. Khoreva, L. Pishchulin and B. Schiele
IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW 2018), 2018
Towards Reaching Human Performance in Pedestrian Detection
S. Zhang, R. Benenson, M. Omran, J. Hosang and B. Schiele
IEEE Transactions on Pattern Analysis and Machine Intelligence, Volume 40, Number 4, 2018
S. Zhang, R. Benenson, M. Omran, J. Hosang and B. Schiele
IEEE Transactions on Pattern Analysis and Machine Intelligence, Volume 40, Number 4, 2018
Abstract
Encouraged by the recent progress in pedestrian detection, we investigate the gap between current state-of-the-art methods
and the “perfect single frame detector”. We enable our analysis by creating a human baseline for pedestrian detection (over the Caltech
pedestrian dataset). After manually clustering the frequent errors of a top detector, we characterise both localisation and background-
versus-foreground errors.
To address localisation errors we study the impact of training annotation noise on the detector performance, and show that we can
improve results even with a small portion of sanitised training data. To address background/foreground discrimination, we study convnets
for pedestrian detection, and discuss which factors affect their performance.
Other than our in-depth analysis, we report top performance on the Caltech pedestrian dataset, and provide a new sanitised set of
training and test annotations.
Learning 3D Shape Completion under Weak Supervision
D. Stutz and A. Geiger
International Journal of Computer Vision, Volume 128, 2018
D. Stutz and A. Geiger
International Journal of Computer Vision, Volume 128, 2018
A4NT: Author Attribute Anonymity by Adversarial Training of Neural Machine Translation
R. Shetty, B. Schiele and M. Fritz
Proceedings of the 27th USENIX Security Symposium, 2018
R. Shetty, B. Schiele and M. Fritz
Proceedings of the 27th USENIX Security Symposium, 2018
Partial Optimality and Fast Lower Bounds for Weighted Correlation Clustering
J.-H. Lange, A. Karrenbauer and B. Andres
Proceedings of the 35th International Conference on Machine Learning (ICML 2018), 2018
J.-H. Lange, A. Karrenbauer and B. Andres
Proceedings of the 35th International Conference on Machine Learning (ICML 2018), 2018
A Multimodal Corpus of Expert Gaze and Behavior during Phonetic Segmentation Tasks
A. Khan, I. Steiner, Y. Sugano, A. Bulling and R. Macdonald
Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), 2018
A. Khan, I. Steiner, Y. Sugano, A. Bulling and R. Macdonald
Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), 2018
Generating Counterfactual Explanations with Natural Language
L. A. Hendricks, R. Hu, T. Darrell and Z. Akata
Proceedings of the 2018 ICML Workshop on Human Interpretability in Machine Learning (WHI 2018), 2018
(arXiv: 1806.09809) L. A. Hendricks, R. Hu, T. Darrell and Z. Akata
Proceedings of the 2018 ICML Workshop on Human Interpretability in Machine Learning (WHI 2018), 2018
Abstract
Natural language explanations of deep neural network decisions provide an
intuitive way for a AI agent to articulate a reasoning process. Current textual
explanations learn to discuss class discriminative features in an image.
However, it is also helpful to understand which attributes might change a
classification decision if present in an image (e.g., "This is not a Scarlet
Tanager because it does not have black wings.") We call such textual
explanations counterfactual explanations, and propose an intuitive method to
generate counterfactual explanations by inspecting which evidence in an input
is missing, but might contribute to a different classification decision if
present in the image. To demonstrate our method we consider a fine-grained
image classification task in which we take as input an image and a
counterfactual class and output text which explains why the image does not
belong to a counterfactual class. We then analyze our generated counterfactual
explanations both qualitatively and quantitatively using proposed automatic
metrics.
Advanced Steel Microstructure Classification by Deep Learning Methods
S. M. Azimi, D. Britz, M. Engstler, M. Fritz and F. Mücklich
Scientific Reports, Volume 8, 2018
S. M. Azimi, D. Britz, M. Engstler, M. Fritz and F. Mücklich
Scientific Reports, Volume 8, 2018
Abstract
The inner structure of a material is called microstructure. It stores the
genesis of a material and determines all its physical and chemical properties.
While microstructural characterization is widely spread and well known, the
microstructural classification is mostly done manually by human experts, which
opens doors for huge uncertainties. Since the microstructure could be a
combination of different phases with complex substructures its automatic
classification is very challenging and just a little work in this field has
been carried out. Prior related works apply mostly designed and engineered
features by experts and classify microstructure separately from feature
extraction step. Recently Deep Learning methods have shown surprisingly good
performance in vision applications by learning the features from data together
with the classification step. In this work, we propose a deep learning method
for microstructure classification in the examples of certain microstructural
constituents of low carbon steel. This novel method employs pixel-wise
segmentation via Fully Convolutional Neural Networks (FCNN) accompanied by
max-voting scheme. Our system achieves 93.94% classification accuracy,
drastically outperforming the state-of-the-art method of 48.89% accuracy,
indicating the effectiveness of pixel-wise approaches. Beyond the success
presented in this paper, this line of research offers a more robust and first
of all objective way for the difficult task of steel quality appreciation.
Towards Reverse-Engineering Black-Box Neural Networks
S. J. Oh, M. Augustin, B. Schiele and M. Fritz
Sixth International Conference on Learning Representations (ICLR 2018), 2018
S. J. Oh, M. Augustin, B. Schiele and M. Fritz
Sixth International Conference on Learning Representations (ICLR 2018), 2018
Higher-order Projected Power Iterations for Scalable Multi-Matching
F. Bernard, J. Thunberg, P. Swoboda and C. Theobalt
Technical Report, 2018
(arXiv: 1811.10541) F. Bernard, J. Thunberg, P. Swoboda and C. Theobalt
Technical Report, 2018
Abstract
The matching of multiple objects (e.g. shapes or images) is a fundamental
problem in vision and graphics. In order to robustly handle ambiguities, noise
and repetitive patterns in challenging real-world settings, it is essential to
take geometric consistency between points into account. Computationally, the
multi-matching problem is difficult. It can be phrased as simultaneously
solving multiple (NP-hard) quadratic assignment problems (QAPs) that are
coupled via cycle-consistency constraints. The main limitations of existing
multi-matching methods are that they either ignore geometric consistency and
thus have limited robustness, or they are restricted to small-scale problems
due to their (relatively) high computational cost. We address these
shortcomings by introducing a Higher-order Projected Power Iteration method,
which is (i) efficient and scales to tens of thousands of points, (ii)
straightforward to implement, (iii) able to incorporate geometric consistency,
and (iv) guarantees cycle-consistent multi-matchings. Experimentally we show
that our approach is superior to existing methods.
Bayesian Prediction of Future Street Scenes through Importance Sampling based Optimization
A. Bhattacharyya, M. Fritz and B. Schiele
Technical Report, 2018
(arXiv: 1806.06939) A. Bhattacharyya, M. Fritz and B. Schiele
Technical Report, 2018
Abstract
For autonomous agents to successfully operate in the real world, anticipation
of future events and states of their environment is a key competence. This
problem can be formalized as a sequence prediction problem, where a number of
observations are used to predict the sequence into the future. However,
real-world scenarios demand a model of uncertainty of such predictions, as
future states become increasingly uncertain and multi-modal -- in particular on
long time horizons. This makes modelling and learning challenging. We cast
state of the art semantic segmentation and future prediction models based on
deep learning into a Bayesian formulation that in turn allows for a full
Bayesian treatment of the prediction problem. We present a new sampling scheme
for this model that draws from the success of variational autoencoders by
incorporating a recognition network. In the experiments we show that our model
outperforms prior work in accuracy of the predicted segmentation and provides
calibrated probabilities that also better capture the multi-modal aspects of
possible future states of street scenes.
Primal-Dual Wasserstein GAN
M. Gemici, Z. Akata and M. Welling
Technical Report, 2018
(arXiv: 1805.09575) M. Gemici, Z. Akata and M. Welling
Technical Report, 2018
Abstract
We introduce Primal-Dual Wasserstein GAN, a new learning algorithm for
building latent variable models of the data distribution based on the primal
and the dual formulations of the optimal transport (OT) problem. We utilize the
primal formulation to learn a flexible inference mechanism and to create an
optimal approximate coupling between the data distribution and the generative
model. In order to learn the generative model, we use the dual formulation and
train the decoder adversarially through a critic network that is regularized by
the approximate coupling obtained from the primal. Unlike previous methods that
violate various properties of the optimal critic, we regularize the norm and
the direction of the gradients of the critic function. Our model shares many of
the desirable properties of auto-encoding models in terms of mode coverage and
latent structure, while avoiding their undesirable averaging properties, e.g.
their inability to capture sharp visual features when modeling real images. We
compare our algorithm with several other generative modeling techniques that
utilize Wasserstein distances on Frechet Inception Distance (FID) and Inception
Scores (IS).
MLCapsule: Guarded Offline Deployment of Machine Learning as a Service
L. Hanzlik, Y. Zhang, K. Grosse, A. Salem, M. Augustin, M. Backes and M. Fritz
Technical Report, 2018
(arXiv: 1808.00590) L. Hanzlik, Y. Zhang, K. Grosse, A. Salem, M. Augustin, M. Backes and M. Fritz
Technical Report, 2018
Abstract
With the widespread use of machine learning (ML) techniques, ML as a service
has become increasingly popular. In this setting, an ML model resides on a
server and users can query the model with their data via an API. However, if
the user's input is sensitive, sending it to the server is not an option.
Equally, the service provider does not want to share the model by sending it to
the client for protecting its intellectual property and pay-per-query business
model. In this paper, we propose MLCapsule, a guarded offline deployment of
machine learning as a service. MLCapsule executes the machine learning model
locally on the user's client and therefore the data never leaves the client.
Meanwhile, MLCapsule offers the service provider the same level of control and
security of its model as the commonly used server-side execution. In addition,
MLCapsule is applicable to offline applications that require local execution.
Beyond protecting against direct model access, we demonstrate that MLCapsule
allows for implementing defenses against advanced attacks on machine learning
models such as model stealing/reverse engineering and membership inference.
Manipulating Attributes of Natural Scenes via Hallucination
L. Karacan, Z. Akata, A. Erdem and E. Erdem
Technical Report, 2018
(arXiv: 1808.07413) L. Karacan, Z. Akata, A. Erdem and E. Erdem
Technical Report, 2018
Abstract
In this study, we explore building a two-stage framework for enabling users
to directly manipulate high-level attributes of a natural scene. The key to our
approach is a deep generative network which can hallucinate images of a scene
as if they were taken at a different season (e.g. during winter), weather
condition (e.g. in a cloudy day) or time of the day (e.g. at sunset). Once the
scene is hallucinated with the given attributes, the corresponding look is then
transferred to the input image while preserving the semantic details intact,
giving a photo-realistic manipulation result. As the proposed framework
hallucinates what the scene will look like, it does not require any reference
style image as commonly utilized in most of the appearance or style transfer
approaches. Moreover, it allows to simultaneously manipulate a given scene
according to a diverse set of transient attributes within a single model,
eliminating the need of training multiple networks per each translation task.
Our comprehensive set of qualitative and quantitative results demonstrate the
effectiveness of our approach against the competing methods.
Learning a Disentangled Embedding for Monocular 3D Shape Retrieval and Pose Estimation
K. Z. Lin, W. Xu, Q. Sun, C. Theobalt and T.-S. Chua
Technical Report, 2018
(arXiv: 1812.09899) K. Z. Lin, W. Xu, Q. Sun, C. Theobalt and T.-S. Chua
Technical Report, 2018
Abstract
We propose a novel approach to jointly perform 3D object retrieval and pose
estimation from monocular images.In order to make the method robust to real
world scene variations in the images, e.g. texture, lighting and background,we
learn an embedding space from 3D data that only includes the relevant
information, namely the shape and pose.Our method can then be trained for
robustness under real world scene variations without having to render a large
training set simulating these variations. Our learned embedding explicitly
disentangles a shape vector and a pose vector, which alleviates both pose bias
for 3D shape retrieval and categorical bias for pose estimation. Having the
learned disentangled embedding, we train a CNN to map the images to the
embedding space, and then retrieve the closest 3D shape from the database and
estimate the 6D pose of the object using the embedding vectors. Our method
achieves 10.8 median error for pose estimation and 0.514 top-1-accuracy for
category agnostic 3D object retrieval on the Pascal3D+ dataset. It therefore
outperforms the previous state-of-the-art methods on both tasks.
From Perception over Anticipation to Manipulation
W. Li
PhD Thesis, Universität des Saarlandes, 2018
W. Li
PhD Thesis, Universität des Saarlandes, 2018
Abstract
From autonomous driving cars to surgical robots, robotic system has enjoyed significant growth over the past decade. With the rapid development in robotics alongside the evolution in the related fields, such as computer vision and machine learning, integrating perception, anticipation and manipulation is key to the success of future robotic system. In this thesis, we explore different ways of such integration to extend the capabilities of a robotic system to take on more challenging real world tasks. On anticipation and perception, we address the recognition of ongoing activity from videos. In particular we focus on long-duration and complex activities and hence propose a new challenging dataset to facilitate the work. We introduce hierarchical labels over the activity classes and investigate the temporal accuracy-specificity trade-offs. We propose a new method based on recurrent neural networks that learns to predict over this hierarchy and realize accuracy specificity trade-offs. Our method outperforms several baselines on this new challenge. On manipulation with perception, we propose an efficient framework for programming a robot to use human tools. We first present a novel and compact model for using tools described by a tip model. Then we explore a strategy of utilizing a dual-gripper approach for manipulating tools – motivated by the absence of dexterous hands on widely available general purpose robots. Afterwards, we embed the tool use learning into a hierarchical architecture and evaluate it on a Baxter research robot. Finally, combining perception, anticipation and manipulation, we focus on a block stacking task. First we explore how to guide robot to place a single block into the scene without collapsing the existing structure. We introduce a mechanism to predict physical stability directly from visual input and evaluate it first on a synthetic data and then on real-world block stacking. Further, we introduce the target stacking task where the agent stacks blocks to reproduce a tower shown in an image. To do so, we create a synthetic block stacking environment with physics simulation in which the agent can learn block stacking end-to-end through trial and error, bypassing to explicitly model the corresponding physics knowledge. We propose a goal-parametrized GDQN model to plan with respect to the specific goal. We validate the model on both a navigation task in a classic gridworld environment and the block stacking task.
Deep Appearance Maps
M. Maximov, T. Ritschel and M. Fritz
Technical Report, 2018
(arXiv: 1804.00863) M. Maximov, T. Ritschel and M. Fritz
Technical Report, 2018
Abstract
We propose a deep representation of appearance, i. e. the relation of color,
surface orientation, viewer position, material and illumination. Previous
approaches have used deep learning to extract classic appearance
representations relating to reflectance model parameters (e. g. Phong) or
illumination (e. g. HDR environment maps). We suggest to directly represent
appearance itself as a network we call a deep appearance map (DAM). This is a
4D generalization over 2D reflectance maps, which held the view direction
fixed. First, we show how a DAM can be learned from images or video frames and
later be used to synthesize appearance, given new surface orientations and
viewer positions. Second, we demonstrate how another network can be used to map
from an image or video frames to a DAM network to reproduce this appearance,
without using a lengthy optimization such as stochastic gradient descent
(learning-to-learn). Finally, we generalize this to an appearance
estimation-and-segmentation task, where we map from an image showing multiple
materials to multiple networks reproducing their appearance, as well as
per-pixel segmentation.
Image Manipulation against Learned Models Privacy and Security Implications
S. J. Oh
PhD Thesis, Universität des Saarlandes, 2018
S. J. Oh
PhD Thesis, Universität des Saarlandes, 2018
Abstract
Machine learning is transforming the world. Its application areas span privacy
sensitive and security critical tasks such as human identification and self-driving
cars. These applications raise privacy and security related questions that are not
fully understood or answered yet: Can automatic person recognisers identify people
in photos even when their faces are blurred? How easy is it to find an adversarial
input for a self-driving car that makes it drive off the road?
This thesis contributes one of the first steps towards a better understanding of
such concerns. We observe that many privacy and security critical scenarios for
learned models involve input data manipulation: users obfuscate their identity by
blurring their faces and adversaries inject imperceptible perturbations to the input
signal. We introduce a data manipulator framework as a tool for collectively describing
and analysing privacy and security relevant scenarios involving learned models.
A data manipulator introduces a shift in data distribution for achieving privacy or
security related goals, and feeds the transformed input to the target model. This
framework provides a common perspective on the studies presented in the thesis.
We begin the studies from the user’s privacy point of view. We analyse the
efficacy of common obfuscation methods like face blurring, and show that they
are surprisingly ineffective against state of the art person recognition systems. We
then propose alternatives based on head inpainting and adversarial examples. By
studying the user privacy, we also study the dual problem: model security. In model
security perspective, a model ought to be robust and reliable against small amounts
of data manipulation. In both cases, data are manipulated with the goal of changing
the target model prediction. User privacy and model security problems can be
described with the same objective.
We then study the knowledge aspect of the data manipulation problem. The more
one knows about the target model, the more effective manipulations one can craft.
We propose a game theoretic manipulation framework to systematically represent
the knowledge level on the target model and derive privacy and security guarantees.
We then discuss ways to increase knowledge about a black-box model by only querying
it, deriving implications that are relevant to both privacy and security perspectives.
Understanding and Controlling User Linkability in Decentralized Learning
T. Orekondy, S. J. Oh, B. Schiele and M. Fritz
Technical Report, 2018
(arXiv: 1805.05838) T. Orekondy, S. J. Oh, B. Schiele and M. Fritz
Technical Report, 2018
Abstract
Machine Learning techniques are widely used by online services (e.g. Google,
Apple) in order to analyze and make predictions on user data. As many of the
provided services are user-centric (e.g. personal photo collections, speech
recognition, personal assistance), user data generated on personal devices is
key to provide the service. In order to protect the data and the privacy of the
user, federated learning techniques have been proposed where the data never
leaves the user's device and "only" model updates are communicated back to the
server. In our work, we propose a new threat model that is not concerned with
learning about the content - but rather is concerned with the linkability of
users during such decentralized learning scenarios.
We show that model updates are characteristic for users and therefore lend
themselves to linkability attacks. We show identification and matching of users
across devices in closed and open world scenarios. In our experiments, we find
our attacks to be highly effective, achieving 20x-175x chance-level
performance.
In order to mitigate the risks of linkability attacks, we study various
strategies. As adding random noise does not offer convincing operation points,
we propose strategies based on using calibrated domain-specific data; we find
these strategies offers substantial protection against linkability threats with
little effect to utility.
End-to-end Learning for Graph Decomposition
J. Song, B. Andres, M. Black, O. Hilliges and S. Tang
Technical Report, 2018
(arXiv: 1812.09737) J. Song, B. Andres, M. Black, O. Hilliges and S. Tang
Technical Report, 2018
Abstract
We propose a novel end-to-end trainable framework for the graph decomposition
problem. The minimum cost multicut problem is first converted to an
unconstrained binary cubic formulation where cycle consistency constraints are
incorporated into the objective function. The new optimization problem can be
viewed as a Conditional Random Field (CRF) in which the random variables are
associated with the binary edge labels of the initial graph and the hard
constraints are introduced in the CRF as high-order potentials. The parameters
of a standard Neural Network and the fully differentiable CRF are optimized in
an end-to-end manner. Furthermore, our method utilizes the cycle constraints as
meta-supervisory signals during the learning of the deep feature
representations by taking the dependencies between the output random variables
into account. We present analyses of the end-to-end learned representations,
showing the impact of the joint training, on the task of clustering images of
MNIST. We also validate the effectiveness of our approach both for the feature
learning and the final clustering on the challenging task of real-world
multi-person pose estimation.
PrivacEye: Privacy-Preserving First-Person Vision Using Image Features and Eye Movement Analysis
J. Steil, M. Koelle, W. Heuten, S. Boll and A. Bulling
Technical Report, 2018
(arXiv: 1801.04457) J. Steil, M. Koelle, W. Heuten, S. Boll and A. Bulling
Technical Report, 2018
Abstract
As first-person cameras in head-mounted displays become increasingly
prevalent, so does the problem of infringing user and bystander privacy. To
address this challenge, we present PrivacEye, a proof-of-concept system that
detects privacysensitive everyday situations and automatically enables and
disables the first-person camera using a mechanical shutter. To close the
shutter, PrivacEye detects sensitive situations from first-person camera videos
using an end-to-end deep-learning model. To open the shutter without visual
input, PrivacEye uses a separate, smaller eye camera to detect changes in
users' eye movements to gauge changes in the "privacy level" of the current
situation. We evaluate PrivacEye on a dataset of first-person videos recorded
in the daily life of 17 participants that they annotated with privacy
sensitivity levels. We discuss the strengths and weaknesses of our
proof-of-concept system based on a quantitative technical evaluation as well as
qualitative insights from semi-structured interviews.
Gaze Estimation and Interaction in Real-World Environments
X. Zhang
PhD Thesis, Universität des Saarlandes, 2018a
X. Zhang
PhD Thesis, Universität des Saarlandes, 2018a
Gaze estimation and interaction in real-world environments
X. Zhang
PhD Thesis, Universität des Saarlandes, 2018b
X. Zhang
PhD Thesis, Universität des Saarlandes, 2018b
Abstract
Following a period of expedited progress in the capabilities of digital systems, the society begins to realize that systems designed to assist people in various tasks can also harm individuals and society. Mediating access to information and explicitly or implicitly ranking people in increasingly many applications, search systems have a substantial potential to contribute to such unwanted outcomes. Since they collect vast amounts of data about both searchers and search subjects, they have the potential to violate the privacy of both of these groups of users. Moreover, in applications where rankings influence people's economic livelihood outside of the platform, such as sharing economy or hiring support websites, search engines have an immense economic power over their users in that they control user exposure in ranked results. This thesis develops new models and methods broadly covering different aspects of privacy and fairness in search systems for both searchers and search subjects. Specifically, it makes the following contributions: (1) We propose a model for computing individually fair rankings where search subjects get exposure proportional to their relevance. The exposure is amortized over time using constrained optimization to overcome searcher attention biases while preserving ranking utility. (2) We propose a model for computing sensitive search exposure where each subject gets to know the sensitive queries that lead to her profile in the top-k search results. The problem of finding exposing queries is technically modeled as reverse nearest neighbor search, followed by a weekly-supervised learning to rank model ordering the queries by privacy-sensitivity. (3) We propose a model for quantifying privacy risks from textual data in online communities. The method builds on a topic model where each topic is annotated by a crowdsourced sensitivity score, and privacy risks are associated with a user's relevance to sensitive topics. We propose relevance measures capturing different dimensions of user interest in a topic and show how they correlate with human risk perceptions. (4) We propose a model for privacy-preserving personalized search where search queries of different users are split and merged into synthetic profiles. The model mediates the privacy-utility trade-off by keeping semantically coherent fragments of search histories within individual profiles, while trying to minimize the similarity of any of the synthetic profiles to the original user profiles. The models are evaluated using information retrieval techniques and user studies over a variety of datasets, ranging from query logs, through social media and community question answering postings, to item listings from sharing economy platforms.
2017
Long-Term On-Board Prediction of Pedestrians in Traffic Scenes
A. Bhattacharyya, M. Fritz and B. Schiele
1st Conference on Robot Learning (CoRL 2017), 2017
A. Bhattacharyya, M. Fritz and B. Schiele
1st Conference on Robot Learning (CoRL 2017), 2017
Gradient-free Policy Architecture Search and Adaptation
S. Ebrahimi, A. Rohrbach and T. Darrell
1st Conference on Robot Learning (CoRL 2017), 2017
S. Ebrahimi, A. Rohrbach and T. Darrell
1st Conference on Robot Learning (CoRL 2017), 2017
CityPersons: A Diverse Dataset for Pedestrian Detection
S. Zhang, R. Benenson and B. Schiele
30th IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2017), 2017
S. Zhang, R. Benenson and B. Schiele
30th IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2017), 2017
Abstract
Convnets have enabled significant progress in pedestrian detection recently,
but there are still open questions regarding suitable architectures and
training data. We revisit CNN design and point out key adaptations, enabling
plain FasterRCNN to obtain state-of-the-art results on the Caltech dataset.
To achieve further improvement from more and better data, we introduce
CityPersons, a new set of person annotations on top of the Cityscapes dataset.
The diversity of CityPersons allows us for the first time to train one single
CNN model that generalizes well over multiple benchmarks. Moreover, with
additional training with CityPersons, we obtain top results using FasterRCNN on
Caltech, improving especially for more difficult cases (heavy occlusion and
small scale) and providing higher localization quality.
Visual Stability Prediction and Its Application to Manipulation
W. Li, A. Leonardis and M. Fritz
AAAI 2017 Spring Symposia 05, Interactive Multisensory Object Perception for Embodied Agents, 2017
W. Li, A. Leonardis and M. Fritz
AAAI 2017 Spring Symposia 05, Interactive Multisensory Object Perception for Embodied Agents, 2017
Pose Guided Person Image Generation
L. Ma, X. Jia, Q. Sun, B. Schiele, T. Tuytelaars and L. Van Gool
Advances in Neural Information Processing Systems 30 (NIPS 2017), 2017
L. Ma, X. Jia, Q. Sun, B. Schiele, T. Tuytelaars and L. Van Gool
Advances in Neural Information Processing Systems 30 (NIPS 2017), 2017
Lucid Data Dreaming for Object Tracking
A. Khoreva, R. Benenson, E. Ilg, T. Brox and B. Schiele
DAVIS Challenge on Video Object Segmentation 2017, 2017
A. Khoreva, R. Benenson, E. Ilg, T. Brox and B. Schiele
DAVIS Challenge on Video Object Segmentation 2017, 2017
Predicting the Category and Attributes of Visual Search Targets Using Deep Gaze Pooling
H. Sattar, A. Bulling and M. Fritz
2017 IEEE International Conference on Computer Vision Workshops (MBCC @ICCV 2017), 2017
H. Sattar, A. Bulling and M. Fritz
2017 IEEE International Conference on Computer Vision Workshops (MBCC @ICCV 2017), 2017
Abstract
Previous work focused on predicting visual search targets from human
fixations but, in the real world, a specific target is often not known, e.g.
when searching for a present for a friend. In this work we instead study the
problem of predicting the mental picture, i.e. only an abstract idea instead of
a specific target. This task is significantly more challenging given that
mental pictures of the same target category can vary widely depending on
personal biases, and given that characteristic target attributes can often not
be verbalised explicitly. We instead propose to use gaze information as
implicit information on users' mental picture and present a novel gaze pooling
layer to seamlessly integrate semantic and localized fixation information into
a deep image representation. We show that we can robustly predict both the
mental picture's category as well as attributes on a novel dataset containing
fixation data of 14 users searching for targets on a subset of the DeepFahion
dataset. Our results have important implications for future search interfaces
and suggest deep gaze pooling as a general-purpose approach for gaze-supported
computer vision systems.
MARCOnI -- ConvNet-Based MARker-Less Motion Capture in Outdoor and Indoor Scenes
A. Elhayek, E. de Aguiar, A. Jain, J. Tompson, L. Pishchulin, M. Andriluka, C. Bregler, B. Schiele and C. Theobalt
IEEE Transactions on Pattern Analysis and Machine Intelligence, Volume 39, Number 3, 2017
A. Elhayek, E. de Aguiar, A. Jain, J. Tompson, L. Pishchulin, M. Andriluka, C. Bregler, B. Schiele and C. Theobalt
IEEE Transactions on Pattern Analysis and Machine Intelligence, Volume 39, Number 3, 2017
A Compact Representation of Human Actions by Sliding Coordinate Coding
R. Ding, Q. Sun, M. Liu and H. Liu
International Journal of Advanced Robotic Systems, Volume 14, Number 6, 2017
R. Ding, Q. Sun, M. Liu and H. Liu
International Journal of Advanced Robotic Systems, Volume 14, Number 6, 2017
Movie Description
A. Rohrbach, A. Torabi, M. Rohrbach, N. Tandon, C. Pal, H. Larochelle, A. Courville and B. Schiele
International Journal of Computer Vision, Volume 123, Number 1, 2017
A. Rohrbach, A. Torabi, M. Rohrbach, N. Tandon, C. Pal, H. Larochelle, A. Courville and B. Schiele
International Journal of Computer Vision, Volume 123, Number 1, 2017
Abstract
Audio Description (AD) provides linguistic descriptions of movies and allows
visually impaired people to follow a movie along with their peers. Such
descriptions are by design mainly visual and thus naturally form an interesting
data source for computer vision and computational linguistics. In this work we
propose a novel dataset which contains transcribed ADs, which are temporally
aligned to full length movies. In addition we also collected and aligned movie
scripts used in prior work and compare the two sources of descriptions. In
total the Large Scale Movie Description Challenge (LSMDC) contains a parallel
corpus of 118,114 sentences and video clips from 202 movies. First we
characterize the dataset by benchmarking different approaches for generating
video descriptions. Comparing ADs to scripts, we find that ADs are indeed more
visual and describe precisely what is shown rather than what should happen
according to the scripts created prior to movie production. Furthermore, we
present and compare the results of several teams who participated in a
challenge organized in the context of the workshop "Describing and
Understanding Video & The Large Scale Movie Description Challenge (LSMDC)", at
ICCV 2015.
Look Together: Using Gaze for Assisting Co-located Collaborative Search
Y. Zhang, K. Pfeuffer, M. K. Chong, J. Alexander, A. Bulling and H. Gellersen
Personal and Ubiquitous Computing, Volume 21, Number 1, 2017
Y. Zhang, K. Pfeuffer, M. K. Chong, J. Alexander, A. Bulling and H. Gellersen
Personal and Ubiquitous Computing, Volume 21, Number 1, 2017
Analysis and Optimization of Graph Decompositions by Lifted Multicuts
A. Horňáková, J.-H. Lange and B. Andres
Proceedings of the 34th International Conference on Machine Learning (ICML 2017), 2017
A. Horňáková, J.-H. Lange and B. Andres
Proceedings of the 34th International Conference on Machine Learning (ICML 2017), 2017
Analysis and Improvement of the Visual Object Detection Pipeline
J. Hosang
PhD Thesis, Universität des Saarlandes, 2017
J. Hosang
PhD Thesis, Universität des Saarlandes, 2017
Abstract
Visual object detection has seen substantial improvements during the last years due to the possibilities enabled by deep learning. While research on image classification provides continuous progress on how to learn image representations and classifiers jointly, object detection research focuses on identifying how to properly use deep learning technology to effectively localise objects. In this thesis, we analyse and improve different aspects of the commonly used detection pipeline. We analyse ten years of research on pedestrian detection and find that improvement of feature representations was the driving factor. Motivated by this finding, we adapt an end-to-end learned detector architecture from general object detection to pedestrian detection. Our deep network outperforms all previous neural networks for pedestrian detection by a large margin, even without using additional training data. After substantial improvements on pedestrian detection in recent years, we investigate the gap between human performance and state-of-the-art pedestrian detectors. We find that pedestrian detectors still have a long way to go before they reach human performance, and we diagnose failure modes of several top performing detectors, giving direction to future research. As a side-effect we publish new, better localised annotations for the Caltech pedestrian benchmark. We analyse detection proposals as a preprocessing step for object detectors. We establish different metrics and compare a wide range of methods according to these metrics. By examining the relationship between localisation of proposals and final object detection performance, we define and experimentally verify a metric that can be used as a proxy for detector performance. Furthermore, we address a structural weakness of virtually all object detection pipelines: non-maximum suppression. We analyse why it is necessary and what the shortcomings of the most common approach are. To address these problems, we present work to overcome these shortcomings and to replace typical non-maximum suppression with a learnable alternative. The introduced paradigm paves the way to true end-to-end learning of object detectors without any post-processing. In summary, this thesis provides analyses of recent pedestrian detectors and detection proposals, improves pedestrian detection by employing deep neural networks, and presents a viable alternative to traditional non-maximum suppression.
Lucid Data Dreaming for Multiple Object Tracking
A. Khoreva, R. Benenson, E. Ilg, T. Brox and B. Schiele
Technical Report, 2017
(arXiv: 1703.09554) A. Khoreva, R. Benenson, E. Ilg, T. Brox and B. Schiele
Technical Report, 2017
Abstract
Convolutional networks reach top quality in pixel-level object tracking but
require a large amount of training data (1k ~ 10k) to deliver such results. We
propose a new training strategy which achieves state-of-the-art results across
three evaluation datasets while using 20x ~ 100x less annotated data than
competing methods. Instead of using large training sets hoping to generalize
across domains, we generate in-domain training data using the provided
annotation on the first frame of each video to synthesize ("lucid dream")
plausible future video frames. In-domain per-video training data allows us to
train high quality appearance- and motion-based models, as well as tune the
post-processing stage. This approach allows to reach competitive results even
when training from only a single annotated frame, without ImageNet
pre-training. Our results indicate that using a larger training set is not
automatically better, and that for the tracking task a smaller training set
that is closer to the target domain is more effective. This changes the mindset
regarding how many training samples and general "objectness" knowledge are
required for the object tracking task.
Learning to Segment in Images and Videos with Different Forms of Supervision
A. Khoreva
PhD Thesis, Universität des Saarlandes, 2017
A. Khoreva
PhD Thesis, Universität des Saarlandes, 2017
Abstract
Much progress has been made in image and video segmentation
over the last years. To a large extent, the success can be attributed to
the strong appearance models completely learned from data, in particular
using deep learning methods. However,to perform best these methods require
large representative datasets for training with expensive pixel-level
annotations, which in case of videos are prohibitive to obtain. Therefore,
there is a need to relax this constraint and to consider alternative forms
of supervision, which are easier and cheaper to collect. In this thesis,
we aim to develop algorithms for learning to segment in images and videos
with different levels of supervision.
First, we develop approaches for training convolutional networks with weaker
forms of supervision, such as bounding boxes or image labels, for object
boundary estimation and semantic/instance labelling tasks. We propose to
generate pixel-level approximate groundtruth from these weaker forms of
annotations to train a network, which allows to achieve high-quality
results comparable to the full supervision quality without any
modifications of the network architecture or the training procedure.
Second, we address the problem of the excessive computational and memory
costs inherent to solving video segmentation via graphs. We propose
approaches to improve the runtime and memory efficiency as well as the
output segmentation quality by learning from the available training data
the best representation of the graph. In particular, we contribute with
learning must-link constraints, the topology and edge weights of the graph
as well as enhancing the graph nodes - superpixels - themselves.
Third, we tackle the task of pixel-level object tracking and address the
problem of the limited amount of densely annotated video data for training
convolutional networks. We introduce an architecture which allows training
with static images only and propose an elaborate data synthesis scheme
which creates a large number of training examples close to the target
domain from the given first frame mask. With the proposed techniques we
show that densely annotated consequent video data is not necessary to
achieve high-quality temporally coherent video segmentationresults.
In summary, this thesis advances the state of the art in weakly supervised
image segmentation, graph-based video segmentation and pixel-level object
tracking and contributes with the new ways of training convolutional
networks with a limited amount of pixel-level annotated training data.
Image Classification with Limited Training Data and Class Ambiguity
M. Lapin
PhD Thesis, Universität des Saarlandes, 2017
M. Lapin
PhD Thesis, Universität des Saarlandes, 2017
Abstract
Modern image classification methods are based on supervised learning algorithms that require labeled training data. However, only a limited amount of annotated data may be available in certain applications due to scarcity of the data itself or high costs associated with human annotation. Introduction of additional information and structural constraints can help improve the performance of a learning algorithm. In this thesis, we study the framework of learning using privileged information and demonstrate its relation to learning with instance weights. We also consider multitask feature learning and develop an efficient dual optimization scheme that is particularly well suited to problems with high dimensional image descriptors. Scaling annotation to a large number of image categories leads to the problem of class ambiguity where clear distinction between the classes is no longer possible. Many real world images are naturally multilabel yet the existing annotation might only contain a single label. In this thesis, we propose and analyze a number of loss functions that allow for a certain tolerance in top k predictions of a learner. Our results indicate consistent improvements over the standard loss functions that put more penalty on the first incorrect prediction compared to the proposed losses. All proposed learning methods are complemented with efficient optimization schemes that are based on stochastic dual coordinate ascent for convex problems and on gradient descent for nonconvex formulations.
Acquiring Target Stacking Skills by Goal-Parameterized Deep Reinforcement Learning
W. Li, J. Bohg and M. Fritz
Technical Report, 2017
(arXiv: 1711.00267) W. Li, J. Bohg and M. Fritz
Technical Report, 2017
Abstract
Understanding physical phenomena is a key component of human intelligence and
enables physical interaction with previously unseen environments. In this
paper, we study how an artificial agent can autonomously acquire this intuition
through interaction with the environment. We created a synthetic block stacking
environment with physics simulation in which the agent can learn a policy
end-to-end through trial and error. Thereby, we bypass to explicitly model
physical knowledge within the policy. We are specifically interested in tasks
that require the agent to reach a given goal state that may be different for
every new trial. To this end, we propose a deep reinforcement learning
framework that learns policies which are parametrized by a goal. We validated
the model on a toy example navigating in a grid world with different target
positions and in a block stacking task with different target structures of the
final tower. In contrast to prior work, our policies show better generalization
across different goals.
Towards Holistic Machines: From Visual Recognition To Question Answering About Real-world Image
M. Malinowski
PhD Thesis, Universität des Saarlandes, 2017
M. Malinowski
PhD Thesis, Universität des Saarlandes, 2017
Abstract
Computer Vision has undergone major changes over the recent five years. Here, we investigate if the performance of such architectures generalizes to more complex tasks that require a more holistic approach to scene comprehension. The presented work focuses on learning spatial and multi-modal representations, and the foundations of a Visual Turing Test, where the scene understanding is tested by a series of questions about its content. In our studies, we propose DAQUAR, the first ‘question answering about real-world images’ dataset together with methods, termed a symbolic-based and a neural-based visual question answering architectures, that address the problem. The symbolic-based method relies on a semantic parser, a database of visual facts, and a bayesian formulation that accounts for various interpretations of the visual scene. The neural-based method is an end-to-end architecture composed of a question encoder, image encoder, multimodal embedding, and answer decoder. This architecture has proven to be effective in capturing language-based biases. It also becomes the standard component of other visual question answering architectures. Along with the methods, we also investigate various evaluation metrics that embraces uncertainty in word's meaning, and various interpretations of the scene and the question.
Whitening Black-Box Neural Networks
S. J. Oh, M. Augustin, B. Schiele and M. Fritz
Technical Report, 2017
(arXiv: 1711.01768) S. J. Oh, M. Augustin, B. Schiele and M. Fritz
Technical Report, 2017
Abstract
Many deployed learned models are black boxes: given input, returns output.
Internal information about the model, such as the architecture, optimisation
procedure, or training data, is not disclosed explicitly as it might contain
proprietary information or make the system more vulnerable. This work shows
that such attributes of neural networks can be exposed from a sequence of
queries. This has multiple implications. On the one hand, our work exposes the
vulnerability of black-box neural networks to different types of attacks -- we
show that the revealed internal information helps generate more effective
adversarial examples against the black box model. On the other hand, this
technique can be used for better protection of private content from automatic
recognition models using adversarial examples. Our paper suggests that it is
actually hard to draw a line between white box and black box models.
Person Recognition in Social Media Photos
S. J. Oh, R. Benenson, M. Fritz and B. Schiele
Technical Report, 2017
(arXiv: 1710.03224) S. J. Oh, R. Benenson, M. Fritz and B. Schiele
Technical Report, 2017
Abstract
People nowadays share large parts of their personal lives through social
media. Being able to automatically recognise people in personal photos may
greatly enhance user convenience by easing photo album organisation. For human
identification task, however, traditional focus of computer vision has been
face recognition and pedestrian re-identification. Person recognition in social
media photos sets new challenges for computer vision, including non-cooperative
subjects (e.g. backward viewpoints, unusual poses) and great changes in
appearance. To tackle this problem, we build a simple person recognition
framework that leverages convnet features from multiple image regions (head,
body, etc.). We propose new recognition scenarios that focus on the time and
appearance gap between training and testing samples. We present an in-depth
analysis of the importance of different features according to time and
viewpoint generalisability. In the process, we verify that our simple approach
achieves the state of the art result on the PIPA benchmark, arguably the
largest social media based benchmark for person recognition to date with
diverse poses, viewpoints, social groups, and events.
Compared the conference version of the paper, this paper additionally
presents (1) analysis of a face recogniser (DeepID2+), (2) new method naeil2
that combines the conference version method naeil and DeepID2+ to achieve state
of the art results even compared to post-conference works, (3) discussion of
related work since the conference version, (4) additional analysis including
the head viewpoint-wise breakdown of performance, and (5) results on the
open-world setup.
Attentive Explanations: Justifying Decisions and Pointing to the Evidence (Extended Abstract)
D. H. Park, L. A. Hendricks, Z. Akata, A. Rohrbach, B. Schiele, T. Darrell and M. Rohrbach
Technical Report, 2017
(arXiv: 1711.07373) D. H. Park, L. A. Hendricks, Z. Akata, A. Rohrbach, B. Schiele, T. Darrell and M. Rohrbach
Technical Report, 2017
Abstract
Deep models are the defacto standard in visual decision problems due to their
impressive performance on a wide array of visual tasks. On the other hand,
their opaqueness has led to a surge of interest in explainable systems. In this
work, we emphasize the importance of model explanation in various forms such as
visual pointing and textual justification. The lack of data with justification
annotations is one of the bottlenecks of generating multimodal explanations.
Thus, we propose two large-scale datasets with annotations that visually and
textually justify a classification decision for various activities, i.e. ACT-X,
and for question answering, i.e. VQA-X. We also introduce a multimodal
methodology for generating visual and textual explanations simultaneously. We
quantitatively show that training with the textual explanations not only yields
better textual justification models, but also models that better localize the
evidence that support their decision.
Generation and Grounding of Natural Language Descriptions for Visual Data
A. Rohrbach
PhD Thesis, Universität des Saarlandes, 2017
A. Rohrbach
PhD Thesis, Universität des Saarlandes, 2017
Abstract
Generating natural language descriptions for visual data links computer vision and computational linguistics. Being able to generate a concise and human-readable description of a video is a step towards visual understanding. At the same time, grounding natural language in visual data provides disambiguation for the linguistic concepts, necessary for many applications. This thesis focuses on both directions and tackles three specific problems. First, we develop recognition approaches to understand video of complex cooking activities. We propose an approach to generate coherent multi-sentence descriptions for our videos. Furthermore, we tackle the new task of describing videos at variable level of detail. Second, we present a large-scale dataset of movies and aligned professional descriptions. We propose an approach, which learns from videos and sentences to describe movie clips relying on robust recognition of visual semantic concepts. Third, we propose an approach to ground textual phrases in images with little or no localization supervision, which we further improve by introducing Multimodal Compact Bilinear Pooling for combining language and vision representations. Finally, we jointly address the task of describing videos and grounding the described people. To summarize, this thesis advances the state-of-the-art in automatic video description and visual grounding and also contributes large datasets for studying the intersection of computer vision and computational linguistics.
Visual Decoding of Targets During Visual Search From Human Eye Fixations
H. Sattar, M. Fritz and A. Bulling
Technical Report, 2017
(arXiv: 1706.05993) H. Sattar, M. Fritz and A. Bulling
Technical Report, 2017
Abstract
What does human gaze reveal about a users' intents and to which extend can
these intents be inferred or even visualized? Gaze was proposed as an implicit
source of information to predict the target of visual search and, more
recently, to predict the object class and attributes of the search target. In
this work, we go one step further and investigate the feasibility of combining
recent advances in encoding human gaze information using deep convolutional
neural networks with the power of generative image models to visually decode,
i.e. create a visual representation of, the search target. Such visual decoding
is challenging for two reasons: 1) the search target only resides in the user's
mind as a subjective visual pattern, and can most often not even be described
verbally by the person, and 2) it is, as of yet, unclear if gaze fixations
contain sufficient information for this task at all. We show, for the first
time, that visual representations of search targets can indeed be decoded only
from human gaze fixations. We propose to first encode fixations into a semantic
representation and then decode this representation into an image. We evaluate
our method on a recent gaze dataset of 14 participants searching for clothing
in image collages and validate the model's predictions using two human studies.
Our results show that 62% (Chance level = 10%) of the time users were able to
select the categories of the decoded image right. In our second studies we show
the importance of a local gaze encoding for decoding visual search targets of
user
People detection and tracking in crowded scenes
S. Tang
PhD Thesis, Universität des Saarlandes, 2017
S. Tang
PhD Thesis, Universität des Saarlandes, 2017
Abstract
People are often a central element of visual scenes, particularly in real-world street scenes. Thus it has been a long-standing goal in Computer Vision to develop methods aiming at analyzing humans in visual data. Due to the complexity of real-world scenes, visual understanding of people remains challenging for machine perception. In this thesis we focus on advancing the techniques for people detection and tracking in crowded street scenes. We also propose new models for human pose estimation and motion segmentation in realistic images and videos. First, we propose detection models that are jointly trained to detect single person as well as pairs of people under varying degrees of occlusion. The learning algorithm of our joint detector facilitates a tight integration of tracking and detection, because it is designed to address common failure cases during tracking due to long-term inter-object occlusions. Second, we propose novel multi person tracking models that formulate tracking as a graph partitioning problem. Our models jointly cluster detection hypotheses in space and time, eliminating the need for a heuristic non-maximum suppression. Furthermore, for crowded scenes, our tracking model encodes long-range person re-identification information into the detection clustering process in a unified and rigorous manner. Third, we explore the visual tracking task in different granularity. We present a tracking model that simultaneously clusters object bounding boxes and pixel level trajectories over time. This approach provides a rich understanding of the motion of objects in the scene. Last, we extend our tracking model for the multi person pose estimation task. We introduce a joint subset partitioning and labelling model where we simultaneously estimate the poses of all the people in the scene. In summary, this thesis addresses a number of diverse tasks that aim to enable vision systems to analyze people in realistic images and videos. In particular, the thesis proposes several novel ideas and rigorous mathematical formulations, pushes the boundary of state-of-the-arts and results in superior performance.
2016
Weakly Supervised Object Boundaries
A. Khoreva, R. Benenson, M. Omran, M. Hein and B. Schiele
29th IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2016), 2016
A. Khoreva, R. Benenson, M. Omran, M. Hein and B. Schiele
29th IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2016), 2016
Abstract
State-of-the-art learning based boundary detection methods require extensive
training data. Since labelling object boundaries is one of the most expensive
types of annotations, there is a need to relax the requirement to carefully
annotate images to make both the training more affordable and to extend the
amount of training data. In this paper we propose a technique to generate
weakly supervised annotations and show that bounding box annotations alone
suffice to reach high-quality object boundaries without using any
object-specific boundary annotations. With the proposed weak supervision
techniques we achieve the top performance on the object boundary detection
task, outperforming by a large margin the current fully supervised
state-of-the-art methods.
Deep Reflectance Maps
K. Rematas, T. Ritschel, M. Fritz, E. Gavves and T. Tuytelaars
29th IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2016), 2016
K. Rematas, T. Ritschel, M. Fritz, E. Gavves and T. Tuytelaars
29th IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2016), 2016
Abstract
Undoing the image formation process and therefore decomposing appearance into
its intrinsic properties is a challenging task due to the under-constraint
nature of this inverse problem. While significant progress has been made on
inferring shape, materials and illumination from images only, progress in an
unconstrained setting is still limited. We propose a convolutional neural
architecture to estimate reflectance maps of specular materials in natural
lighting conditions. We achieve this in an end-to-end learning formulation that
directly predicts a reflectance map from the image itself. We show how to
improve estimates by facilitating additional supervision in an indirect scheme
that first predicts surface orientation and afterwards predicts the reflectance
map by a learning-based sparse data interpolation.
In order to analyze performance on this difficult task, we propose a new
challenge of Specular MAterials on SHapes with complex IllumiNation (SMASHINg)
using both synthetic and real images. Furthermore, we show the application of
our method to a range of image-based editing tasks on real images.
Learning What and Where to Draw
S. Reed, Z. Akata, S. Mohan, S. Tenka, B. Schiele and L. Honglak
Advances in Neural Information Processing Systems 29 (NIPS 2016), 2016
S. Reed, Z. Akata, S. Mohan, S. Tenka, B. Schiele and L. Honglak
Advances in Neural Information Processing Systems 29 (NIPS 2016), 2016
On the Verge: Voluntary Convergences for Accurate and Precise Timing of Gaze Input
D. Kirst and A. Bulling
CHI 2016 Extended Abstracts, 2016
D. Kirst and A. Bulling
CHI 2016 Extended Abstracts, 2016
Abstract
Rotations performed with the index finger and thumb involve some of the most complex motor action among common multi-touch gestures, yet little is known about the factors affecting performance and ergonomics. This note presents results from a study where the angle, direction, diameter, and position of rotations were systematically manipulated. Subjects were asked to perform the rotations as quickly as possible without losing contact with the display, and were allowed to skip rotations that were too uncomfortable. The data show surprising interaction effects among the variables, and help us identify whole categories of rotations that are slow and cumbersome for users.
Generating Visual Explanations
L. A. Hendricks, Z. Akata, M. Rohrbach, J. Donahue, B. Schiele and T. Darrell
Computer Vision -- ECCV 2016, 2016
L. A. Hendricks, Z. Akata, M. Rohrbach, J. Donahue, B. Schiele and T. Darrell
Computer Vision -- ECCV 2016, 2016
Abstract
Clearly explaining a rationale for a classification decision to an end-user
can be as important as the decision itself. Existing approaches for deep visual
recognition are generally opaque and do not output any justification text;
contemporary vision-language models can describe image content but fail to take
into account class-discriminative image aspects which justify visual
predictions. We propose a new model that focuses on the discriminating
properties of the visible object, jointly predicts a class label, and explains
why the predicted label is appropriate for the image. We propose a novel loss
function based on sampling and reinforcement learning that learns to generate
sentences that realize a global sentence property, such as class specificity.
Our results on a fine-grained bird species classification dataset show that our
model is able to generate explanations which are not only consistent with an
image but also more discriminative than descriptions produced by existing
captioning methods.
DeeperCut: A Deeper, Stronger, and Faster Multi-Person Pose Estimation Model
E. Insafutdinov, L. Pishchulin, B. Andres, M. Andriluka and B. Schiele
Computer Vision -- ECCV 2016, 2016
E. Insafutdinov, L. Pishchulin, B. Andres, M. Andriluka and B. Schiele
Computer Vision -- ECCV 2016, 2016
Abstract
The goal of this paper is to advance the state-of-the-art of articulated pose
estimation in scenes with multiple people. To that end we contribute on three
fronts. We propose (1) improved body part detectors that generate effective
bottom-up proposals for body parts; (2) novel image-conditioned pairwise terms
that allow to assemble the proposals into a variable number of consistent body
part configurations; and (3) an incremental optimization strategy that explores
the search space more efficiently thus leading both to better performance and
significant speed-up factors. We evaluate our approach on two single-person and
two multi-person pose estimation benchmarks. The proposed approach
significantly outperforms best known multi-person pose estimation results while
demonstrating competitive performance on the task of single person pose
estimation. Models and code available at http://pose.mpi-inf.mpg.de
VConv-DAE: Deep Volumetric Shape Learning Without Object Labels
A. Sharma, O. Grau and M. Fritz
Computer Vision - ECCV 2016 Workshops, 2016
A. Sharma, O. Grau and M. Fritz
Computer Vision - ECCV 2016 Workshops, 2016
Abstract
With the advent of affordable depth sensors, 3D capture becomes more and more
ubiquitous and already has made its way into commercial products. Yet,
capturing the geometry or complete shapes of everyday objects using scanning
devices (eg. Kinect) still comes with several challenges that result in noise
or even incomplete shapes. Recent success in deep learning has shown how to
learn complex shape distributions in a data-driven way from large scale 3D CAD
Model collections and to utilize them for 3D processing on volumetric
representations and thereby circumventing problems of topology and
tessellation. Prior work has shown encouraging results on problems ranging from
shape completion to recognition. We provide an analysis of such approaches and
discover that training as well as the resulting representation are strongly and
unnecessarily tied to the notion of object labels. Furthermore, deep learning
research argues ~\cite{Vincent08} that learning representation with
over-complete model are more prone to overfitting compared to the approach that
learns from noisy data. Thus, we investigate a full convolutional volumetric
denoising auto encoder that is trained in a unsupervised fashion. It
outperforms prior work on recognition as well as more challenging tasks like
denoising and shape completion. In addition, our approach is atleast two order
of magnitude faster at test time and thus, provides a path to scaling up 3D
deep learning.
Improved Image Boundaries for Better Video Segmentation
A. Khoreva, R. Benenson, F. Galasso, M. Hein and B. Schiele
Computer Vision -- ECCV 2016 Workshops, 2016
A. Khoreva, R. Benenson, F. Galasso, M. Hein and B. Schiele
Computer Vision -- ECCV 2016 Workshops, 2016
Abstract
Graph-based video segmentation methods rely on superpixels as starting point.
While most previous work has focused on the construction of the graph edges and
weights as well as solving the graph partitioning problem, this paper focuses
on better superpixels for video segmentation. We demonstrate by a comparative
analysis that superpixels extracted from boundaries perform best, and show that
boundary estimation can be significantly improved via image and time domain
cues. With superpixels generated from our better boundaries we observe
consistent improvement for two video segmentation methods in two different
datasets.
Eyewear Computing -- Augmenting the Human with Head-mounted Wearable Assistants
A. Bulling, O. Cakmakci, K. Kunze and J. M. Rehg (Eds.)
Schloss Dagstuhl, 2016
A. Bulling, O. Cakmakci, K. Kunze and J. M. Rehg (Eds.)
Schloss Dagstuhl, 2016
Ask Your Neurons Again: Analysis of Deep Methods with Global Image Representation
M. Malinowski, M. Rohrbach and M. Fritz
IEEE Conference on Computer Vision and Pattern Recognition Workshops (VQA 2016), 2016
(Accepted/in press) M. Malinowski, M. Rohrbach and M. Fritz
IEEE Conference on Computer Vision and Pattern Recognition Workshops (VQA 2016), 2016
Abstract
We are addressing an open-ended question answering task
about real-world images. With the help of currently available methods
developed in Computer Vision and Natural Language Processing, we would
like to push an architecture with a global visual representation to its
limits. In our contribution, we show how to achieve competitive
performance on VQA with global visual features (Residual Net) together
with a carefully desgined architecture.
Long Term Boundary Extrapolation for Deterministic Motion
A. Bhattacharyya, M. Malinowski and M. Fritz
NIPS Workshop on Intuitive Physics, 2016
A. Bhattacharyya, M. Malinowski and M. Fritz
NIPS Workshop on Intuitive Physics, 2016
A Convnet for Non-maximum Suppression
J. Hosang, R. Benenson and B. Schiele
Pattern Recognition (GCPR 2016), 2016
J. Hosang, R. Benenson and B. Schiele
Pattern Recognition (GCPR 2016), 2016
Abstract
Non-maximum suppression (NMS) is used in virtually all state-of-the-art
object detection pipelines. While essential object detection ingredients such
as features, classifiers, and proposal methods have been extensively researched
surprisingly little work has aimed to systematically address NMS. The de-facto
standard for NMS is based on greedy clustering with a fixed distance threshold,
which forces to trade-off recall versus precision. We propose a convnet
designed to perform NMS of a given set of detections. We report experiments on
a synthetic setup, and results on crowded pedestrian detection scenes. Our
approach overcomes the intrinsic limitations of greedy NMS, obtaining better
recall and precision.
Generative Adversarial Text to Image Synthesis
S. Reed, Z. Akata, X. Yan, L. Logeswaran, B. Schiele and H. Lee
Proceedings of the 33rd International Conference on Machine Learning (ICML 2016), 2016
S. Reed, Z. Akata, X. Yan, L. Logeswaran, B. Schiele and H. Lee
Proceedings of the 33rd International Conference on Machine Learning (ICML 2016), 2016
Mean Box Pooling: A Rich Image Representation and Output Embedding for the Visual Madlibs Task
A. Mokarian Forooshani, M. Malinowski and M. Fritz
Proceedings of the British Machine Vision Conference (BMVC 2016), 2016
A. Mokarian Forooshani, M. Malinowski and M. Fritz
Proceedings of the British Machine Vision Conference (BMVC 2016), 2016
Commonsense in Parts: Mining Part-Whole Relations from the Web and Image Tags
N. Tandon, C. D. Hariman, J. Urbani, A. Rohrbach, M. Rohrbach and G. Weikum
Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, 2016
N. Tandon, C. D. Hariman, J. Urbani, A. Rohrbach, M. Rohrbach and G. Weikum
Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, 2016
Spatio-Temporal Image Boundary Extrapolation
A. Bhattacharyya, M. Malinowski and M. Fritz
Technical Report, 2016
(arXiv: 1605.07363) A. Bhattacharyya, M. Malinowski and M. Fritz
Technical Report, 2016
Abstract
Boundary prediction in images as well as video has been a very active topic
of research and organizing visual information into boundaries and segments is
believed to be a corner stone of visual perception. While prior work has
focused on predicting boundaries for observed frames, our work aims at
predicting boundaries of future unobserved frames. This requires our model to
learn about the fate of boundaries and extrapolate motion patterns. We
experiment on established real-world video segmentation dataset, which provides
a testbed for this new task. We show for the first time spatio-temporal
boundary extrapolation in this challenging scenario. Furthermore, we show
long-term prediction of boundaries in situations where the motion is governed
by the laws of physics. We successfully predict boundaries in a billiard
scenario without any assumptions of a strong parametric model or any object
notion. We argue that our model has with minimalistic model assumptions derived
a notion of 'intuitive physics' that can be applied to novel scenes.
Bayesian Non-Parametrics for Multi-Modal Segmentation
W.-C. Chiu
PhD Thesis, Universität des Saarlandes, 2016
W.-C. Chiu
PhD Thesis, Universität des Saarlandes, 2016
Natural Illumination from Multiple Materials Using Deep Learning
S. Georgoulis, K. Rematas, T. Ritschel, M. Fritz, T. Tuytelaars and L. Van Gool
Technical Report, 2016
(arXiv: 1611.09325) S. Georgoulis, K. Rematas, T. Ritschel, M. Fritz, T. Tuytelaars and L. Van Gool
Technical Report, 2016
Abstract
Recovering natural illumination from a single Low-Dynamic Range (LDR) image
is a challenging task. To remedy this situation we exploit two properties often
found in everyday images. First, images rarely show a single material, but
rather multiple ones that all reflect the same illumination. However, the
appearance of each material is observed only for some surface orientations, not
all. Second, parts of the illumination are often directly observed in the
background, without being affected by reflection. Typically, this directly
observed part of the illumination is even smaller. We propose a deep
Convolutional Neural Network (CNN) that combines prior knowledge about the
statistics of illumination and reflectance with an input that makes explicit
use of these two observations. Our approach maps multiple partial LDR material
observations represented as reflectance maps and a background image to a
spherical High-Dynamic Range (HDR) illumination map. For training and testing
we propose a new data set comprising of synthetic and real images with multiple
materials observed under the same illumination. Qualitative and quantitative
evidence shows how both multi-material and using a background are essential to
improve illumination estimations.
DeLight-Net: Decomposing Reflectance Maps into Specular Materials and Natural Illumination
S. Georgoulis, K. Rematas, T. Ritschel, M. Fritz, L. Van Gool and T. Tuytelaars
Technical Report, 2016
(arXiv: 1603.08240) S. Georgoulis, K. Rematas, T. Ritschel, M. Fritz, L. Van Gool and T. Tuytelaars
Technical Report, 2016
Abstract
In this paper we are extracting surface reflectance and natural environmental
illumination from a reflectance map, i.e. from a single 2D image of a sphere of
one material under one illumination. This is a notoriously difficult problem,
yet key to various re-rendering applications. With the recent advances in
estimating reflectance maps from 2D images their further decomposition has
become increasingly relevant.
To this end, we propose a Convolutional Neural Network (CNN) architecture to
reconstruct both material parameters (i.e. Phong) as well as illumination (i.e.
high-resolution spherical illumination maps), that is solely trained on
synthetic data. We demonstrate that decomposition of synthetic as well as real
photographs of reflectance maps, both in High Dynamic Range (HDR), and, for the
first time, on Low Dynamic Range (LDR) as well. Results are compared to
previous approaches quantitatively as well as qualitatively in terms of
re-renderings where illumination, material, view or shape are changed.
RGBD Semantic Segmentation Using Spatio-Temporal Data-Driven Pooling
Y. He, W.-C. Chiu, M. Keuper and M. Fritz
Technical Report, 2016
(arXiv: 1604.02388) Y. He, W.-C. Chiu, M. Keuper and M. Fritz
Technical Report, 2016
Abstract
Beyond the success in classification, neural networks have recently shown
strong results on pixel-wise prediction tasks like image semantic segmentation
on RGBD data. However, the commonly used deconvolutional layers for upsampling
intermediate representations to the full-resolution output still show different
failure modes, like imprecise segmentation boundaries and label mistakes in
particular on large, weakly textured objects (e.g. fridge, whiteboard, door).
We attribute these errors in part to the rigid way, current network aggregate
information, that can be either too local (missing context) or too global
(inaccurate boundaries). Therefore we propose a data-driven pooling layer that
integrates with fully convolutional architectures and utilizes boundary
detection from RGBD image segmentation approaches. We extend our approach to
leverage region-level correspondences across images with an additional temporal
pooling stage. We evaluate our approach on the NYU-Depth-V2 dataset comprised
of indoor RGBD video sequences and compare it to various state-of-the-art
baselines. Besides a general improvement over the state-of-the-art, our
approach shows particularly good results in terms of accuracy of the predicted
boundaries and in segmenting previously problematic classes.
End-to-End Eye Movement Detection Using Convolutional Neural Networks
S. Hoppe and A. Bulling
Technical Report, 2016
(arXiv: 1609.02452) S. Hoppe and A. Bulling
Technical Report, 2016
Abstract
Common computational methods for automated eye movement detection - i.e. the
task of detecting different types of eye movement in a continuous stream of
gaze data - are limited in that they either involve thresholding on
hand-crafted signal features, require individual detectors each only detecting
a single movement, or require pre-segmented data. We propose a novel approach
for eye movement detection that only involves learning a single detector
end-to-end, i.e. directly from the continuous gaze data stream and
simultaneously for different eye movements without any manual feature crafting
or segmentation. Our method is based on convolutional neural networks (CNN)
that recently demonstrated superior performance in a variety of tasks in
computer vision, signal processing, and machine learning. We further introduce
a novel multi-participant dataset that contains scripted and free-viewing
sequences of ground-truth annotated saccades, fixations, and smooth pursuits.
We show that our CNN-based method outperforms state-of-the-art baselines by a
large margin on this challenging dataset, thereby underlining the significant
potential of this approach for holistic, robust, and accurate eye movement
protocol analysis.
A Multi-cut Formulation for Joint Segmentation and Tracking of Multiple Objects
M. Keuper, S. Tang, Z. Yu, B. Andres, T. Brox and B. Schiele
Technical Report, 2016
(arXiv: 1607.06317) M. Keuper, S. Tang, Z. Yu, B. Andres, T. Brox and B. Schiele
Technical Report, 2016
Abstract
Recently, Minimum Cost Multicut Formulations have been proposed and proven to
be successful in both motion trajectory segmentation and multi-target tracking
scenarios. Both tasks benefit from decomposing a graphical model into an
optimal number of connected components based on attractive and repulsive
pairwise terms. The two tasks are formulated on different levels of granularity
and, accordingly, leverage mostly local information for motion segmentation and
mostly high-level information for multi-target tracking. In this paper we argue
that point trajectories and their local relationships can contribute to the
high-level task of multi-target tracking and also argue that high-level cues
from object detection and tracking are helpful to solve motion segmentation. We
propose a joint graphical model for point trajectories and object detections
whose Multicuts are solutions to motion segmentation {\it and} multi-target
tracking problems at once. Results on the FBMS59 motion segmentation benchmark
as well as on pedestrian tracking sequences from the 2D MOT 2015 benchmark
demonstrate the promise of this joint approach.
To Fall Or Not To Fall: A Visual Approach to Physical Stability Prediction
W. Li, S. Azimi, A. Leonardis and M. Fritz
Technical Report, 2016
(arXiv: 1604.00066) W. Li, S. Azimi, A. Leonardis and M. Fritz
Technical Report, 2016
Abstract
Understanding physical phenomena is a key competence that enables humans and
animals to act and interact under uncertain perception in previously unseen
environments containing novel object and their configurations. Developmental
psychology has shown that such skills are acquired by infants from observations
at a very early stage.
In this paper, we contrast a more traditional approach of taking a
model-based route with explicit 3D representations and physical simulation by
an end-to-end approach that directly predicts stability and related quantities
from appearance. We ask the question if and to what extent and quality such a
skill can directly be acquired in a data-driven way bypassing the need for an
explicit simulation.
We present a learning-based approach based on simulated data that predicts
stability of towers comprised of wooden blocks under different conditions and
quantities related to the potential fall of the towers. The evaluation is
carried out on synthetic data and compared to human judgments on the same
stimuli.
Tutorial on Answering Questions about Images with Deep Learning
M. Malinowski and M. Fritz
Technical Report, 2016
(arXiv: 1610.01076) M. Malinowski and M. Fritz
Technical Report, 2016
Abstract
Together with the development of more accurate methods in Computer Vision and
Natural Language Understanding, holistic architectures that answer on questions
about the content of real-world images have emerged. In this tutorial, we build
a neural-based approach to answer questions about images. We base our tutorial
on two datasets: (mostly on) DAQUAR, and (a bit on) VQA. With small tweaks the
models that we present here can achieve a competitive performance on both
datasets, in fact, they are among the best methods that use a combination of
LSTM with a global, full frame CNN representation of an image. We hope that
after reading this tutorial, the reader will be able to use Deep Learning
frameworks, such as Keras and introduced Kraino, to build various architectures
that will lead to a further performance improvement on this challenging task.
Attentive Explanations: Justifying Decisions and Pointing to the Evidence
D. H. Park, L. A. Hendricks, Z. Akata, B. Schiele, T. Darrell and M. Rohrbach
Technical Report, 2016
(arXiv: 1612.04757) D. H. Park, L. A. Hendricks, Z. Akata, B. Schiele, T. Darrell and M. Rohrbach
Technical Report, 2016
Abstract
Deep models are the defacto standard in visual decision models due to their
impressive performance on a wide array of visual tasks. However, they are
frequently seen as opaque and are unable to explain their decisions. In
contrast, humans can justify their decisions with natural language and point to
the evidence in the visual world which led to their decisions. We postulate
that deep models can do this as well and propose our Pointing and Justification
(PJ-X) model which can justify its decision with a sentence and point to the
evidence by introspecting its decision and explanation process using an
attention mechanism. Unfortunately there is no dataset available with reference
explanations for visual decision making. We thus collect two datasets in two
domains where it is interesting and challenging to explain decisions. First, we
extend the visual question answering task to not only provide an answer but
also a natural language explanation for the answer. Second, we focus on
explaining human activities which is traditionally more challenging than object
classification. We extensively evaluate our PJ-X model, both on the
justification and pointing tasks, by comparing it to prior models and ablations
using both automatic and human evaluations.
Articulated People Detection and Pose Estimation in Challenging Real World Environments
L. Pishchulin
PhD Thesis, Universität des Saarlandes, 2016
L. Pishchulin
PhD Thesis, Universität des Saarlandes, 2016
EgoCap: Egocentric Marker-less Motion Capture with Two Fisheye Cameras (Extended Abstract)
H. Rhodin, C. Richardt, D. Casas, E. Insafutdinov, M. Shafiei, H.-P. Seidel, B. Schiele and C. Theobalt
Technical Report, 2016b
(arXiv: 1701.00142) H. Rhodin, C. Richardt, D. Casas, E. Insafutdinov, M. Shafiei, H.-P. Seidel, B. Schiele and C. Theobalt
Technical Report, 2016b
Abstract
Marker-based and marker-less optical skeletal motion-capture methods use an
outside-in arrangement of cameras placed around a scene, with viewpoints
converging on the center. They often create discomfort by possibly needed
marker suits, and their recording volume is severely restricted and often
constrained to indoor scenes with controlled backgrounds. We therefore propose
a new method for real-time, marker-less and egocentric motion capture which
estimates the full-body skeleton pose from a lightweight stereo pair of fisheye
cameras that are attached to a helmet or virtual-reality headset. It combines
the strength of a new generative pose estimation framework for fisheye views
with a ConvNet-based body-part detector trained on a new automatically
annotated and augmented dataset. Our inside-in method captures full-body motion
in general indoor and outdoor scenes, and also crowded scenes.
Seeing with Humans: Gaze-Assisted Neural Image Captioning
Y. Sugano and A. Bulling
Technical Report, 2016
(arXiv: 1608.05203) Y. Sugano and A. Bulling
Technical Report, 2016
Abstract
Gaze reflects how humans process visual scenes and is therefore increasingly
used in computer vision systems. Previous works demonstrated the potential of
gaze for object-centric tasks, such as object localization and recognition, but
it remains unclear if gaze can also be beneficial for scene-centric tasks, such
as image captioning. We present a new perspective on gaze-assisted image
captioning by studying the interplay between human gaze and the attention
mechanism of deep neural networks. Using a public large-scale gaze dataset, we
first assess the relationship between state-of-the-art object and scene
recognition models, bottom-up visual saliency, and human gaze. We then propose
a novel split attention model for image captioning. Our model integrates human
gaze information into an attention-based long short-term memory architecture,
and allows the algorithm to allocate attention selectively to both fixated and
non-fixated image regions. Through evaluation on the COCO/SALICON datasets we
show that our method improves image captioning performance and that gaze can
complement machine attention for semantic scene understanding tasks.
2015
On the Interplay between Spontaneous Spoken Instructions and Human Visual Behaviour in an Indoor Guidance Task
N. Koleva, S. Hoppe, M. M. Moniri, M. Staudte and A. Bulling
37th Annual Meeting of the Cognitive Science Society (COGSCI 2015), 2015
N. Koleva, S. Hoppe, M. M. Moniri, M. Staudte and A. Bulling
37th Annual Meeting of the Cognitive Science Society (COGSCI 2015), 2015
Efficient Output Kernel Learning for Multiple Tasks
P. Jawanpuria, M. Lapin, M. Hein and B. Schiele
Advances in Neural Information Processing Systems 28 (NIPS 2015), 2015
P. Jawanpuria, M. Lapin, M. Hein and B. Schiele
Advances in Neural Information Processing Systems 28 (NIPS 2015), 2015
Top-k Multiclass SVM
M. Lapin, M. Hein and B. Schiele
Advances in Neural Information Processing Systems 28 (NIPS 2015), 2015
M. Lapin, M. Hein and B. Schiele
Advances in Neural Information Processing Systems 28 (NIPS 2015), 2015
Rekonstruktion zerebraler Gefässnetzwerke aus in-vivo μMRA mittels physiologischem Vorwissen zur lokalen Gefässgeometrie
M. Rempfler, M. Schneider, G. D. Ielacqua, T. Sprenger, X. Xiao, S. R. Stock, J. Klohs, G. Székely, B. Andres and B. H. Menze
Bildverarbeitung für die Medizin 2015 (BVM 2015), 2015
M. Rempfler, M. Schneider, G. D. Ielacqua, T. Sprenger, X. Xiao, S. R. Stock, J. Klohs, G. Székely, B. Andres and B. H. Menze
Bildverarbeitung für die Medizin 2015 (BVM 2015), 2015
A Study on the Natural History of Scanning Behaviour in Patients with Visual Field Defects after Stroke
T. Loetscher, C. Chen, S. Wignall, A. Bulling, S. Hoppe, O. Churches, N. A. Thomas, M. E. R. Nicholls and A. Lee
BMC Neurology, Volume 15, 2015
T. Loetscher, C. Chen, S. Wignall, A. Bulling, S. Hoppe, O. Churches, N. A. Thomas, M. E. R. Nicholls and A. Lee
BMC Neurology, Volume 15, 2015
The Royal Corgi: Exploring Social Gaze Interaction for Immersive Gameplay
M. Vidal, R. Bismuth, A. Bulling and H. Gellersen
CHI 2015, 33rd Annual ACM Conference on Human Factors in Computing Systems, 2015
M. Vidal, R. Bismuth, A. Bulling and H. Gellersen
CHI 2015, 33rd Annual ACM Conference on Human Factors in Computing Systems, 2015
Abstract
The eyes are a rich channel for non-verbal communication in
our daily interactions. We propose social gaze interaction as a game
mechanic to enhance user interactions with virtual characters. We
develop a game from the ground-up in which characters are esigned to be
reactive to the player’s gaze in social ways, such as etting annoyed
when the player seems distracted or changing their dialogue depending on
the player’s apparent focus of ttention. Results from a qualitative user
study provide insights bout how social gaze interaction is intuitive for
users, elicits deep feelings of immersion, and highlight the players’
self-consciousness of their own eye movements through their strong
reactions to the characters
Computational Modelling and Prediction of Gaze Estimation Error for Head-mounted Eye Trackers
M. Barz, A. Bulling and F. Daiber
Technical Report, 2015
M. Barz, A. Bulling and F. Daiber
Technical Report, 2015
Abstract
Head-mounted eye tracking has significant potential for
mobile gaze-based interaction with ambient displays but current
interfaces lack information about the tracker\'s gaze estimation error.
Consequently, current interfaces do not exploit the full potential of
gaze input as the inherent estimation error can not be dealt with. The
error depends on the physical properties of the display and constantly
varies with changes in position and distance of the user to the display.
In this work we present a computational model of gaze estimation error
for head-mounted eye trackers. Our model covers the full processing
pipeline for mobile gaze estimation, namely mapping of pupil positions
to scene camera coordinates, marker-based display detection, and display
mapping. We build the model based on a series of controlled measurements
of a sample state-of-the-art monocular head-mounted eye tracker. Results
show that our model can predict gaze estimation error with a root mean
squared error of 17.99~px ($1.96^\\circ$).
GazeProjector: Location-independent Gaze Interaction on and Across Multiple Displays
C. Lander, S. Gehring, A. Krüger, S. Boring and A. Bulling
Technical Report, 2015
C. Lander, S. Gehring, A. Krüger, S. Boring and A. Bulling
Technical Report, 2015
Abstract
Mobile gaze-based interaction with multiple displays may
occur from arbitrary positions and orientations. However, maintaining
high gaze estimation accuracy still represents a significant challenge.
To address this, we present GazeProjector, a system that combines
accurate point-of-gaze estimation with natural feature tracking on
displays to determine the mobile eye tracker’s position relative to a
display. The detected eye positions are transformed onto that display
allowing for gaze-based interaction. This allows for seamless gaze
estimation and interaction on (1) multiple displays of arbitrary sizes,
(2) independently of the user’s position and orientation to the display.
In a user study with 12 participants we compared GazeProjector to
existing well- established methods such as visual on-screen markers and
a state-of-the-art motion capture system. Our results show that our
approach is robust to varying head poses, orientations, and distances to
the display, while still providing high gaze estimation accuracy across
multiple displays without re-calibration. The system represents an
important step towards the vision of pervasive gaze-based interfaces.
See the Difference: Direct Pre-Image Reconstruction and Pose Estimation by Differentiating HOG
W.-C. Chiu and M. Fritz
ICCV 2015, IEEE International Conference on Computer Vision, 2015
W.-C. Chiu and M. Fritz
ICCV 2015, IEEE International Conference on Computer Vision, 2015
Ask Your Neurons: A Neural-based Approach to Answering Questions About Images
M. Malinowski, M. Rohrbach and M. Fritz
ICCV 2015, IEEE International Conference on Computer Vision, 2015
M. Malinowski, M. Rohrbach and M. Fritz
ICCV 2015, IEEE International Conference on Computer Vision, 2015
Rendering of Eyes for Eye-Shape Registration and Gaze Estimation
E. Wood, T. Baltrusaitis, X. Zhang, Y. Sugano, P. Robinson and A. Bulling
ICCV 2015, IEEE International Conference on Computer Vision, 2015
E. Wood, T. Baltrusaitis, X. Zhang, Y. Sugano, P. Robinson and A. Bulling
ICCV 2015, IEEE International Conference on Computer Vision, 2015
Efficient ConvNet-based Marker-less Motion Capture in General Scenes with a Low Number of Cameras
A. Elhayek, E. de Aguiar, J. Tompson, A. Jain, L. Pishchulin, M. Andriluka, C. Bregler, B. Schiele and C. Theobalt
IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2015), 2015
A. Elhayek, E. de Aguiar, J. Tompson, A. Jain, L. Pishchulin, M. Andriluka, C. Bregler, B. Schiele and C. Theobalt
IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2015), 2015
A Comparative Study of Modern Inference Techniques for Structured Discrete Energy Minimization Problems
J. H. Kappes, B. Andres, F. A. Hamprecht, C. Schnörr, S. Nowozin, D. Batra, S. Kim, B. X. Kausler, T. Kröger, J. Lellmann, N. Komodakis, B. Savchynskyy and C. Rother
International Journal of Computer Vision, Volume 115, Number 2, 2015
J. H. Kappes, B. Andres, F. A. Hamprecht, C. Schnörr, S. Nowozin, D. Batra, S. Kim, B. X. Kausler, T. Kröger, J. Lellmann, N. Komodakis, B. Savchynskyy and C. Rother
International Journal of Computer Vision, Volume 115, Number 2, 2015
Abstract
Szeliski et al. published an influential study in 2006 on energy minimization
methods for Markov Random Fields (MRF). This study provided valuable insights
in choosing the best optimization technique for certain classes of problems.
While these insights remain generally useful today, the phenomenal success of
random field models means that the kinds of inference problems that have to be
solved changed significantly. Specifically, the models today often include
higher order interactions, flexible connectivity structures, large
la\-bel-spaces of different cardinalities, or learned energy tables. To reflect
these changes, we provide a modernized and enlarged study. We present an
empirical comparison of 32 state-of-the-art optimization techniques on a corpus
of 2,453 energy minimization instances from diverse applications in computer
vision. To ensure reproducibility, we evaluate all methods in the OpenGM 2
framework and report extensive results regarding runtime and solution quality.
Key insights from our study agree with the results of Szeliski et al. for the
types of models they studied. However, on new and challenging types of models
our findings disagree and suggest that polyhedral methods and integer
programming solvers are competitive in terms of runtime and solution quality
over a large range of model types.
Bridging the Gap Between Synthetic and Real Data
M. Fritz
Machine Learning with Interdependent and Non-Identically Distributed Data, 2015
M. Fritz
Machine Learning with Interdependent and Non-Identically Distributed Data, 2015
Graphical Passwords in the Wild: Understanding How Users Choose Pictures and Passwords in Image-based Authentication Schemes
F. Alt, S. Schneegass, A. Shirazi, M. Hassib and A. Bulling
MobileHCI’15, 17th International Conference on Human-Computer Interaction with Mobile Devices and Services, 2015
F. Alt, S. Schneegass, A. Shirazi, M. Hassib and A. Bulling
MobileHCI’15, 17th International Conference on Human-Computer Interaction with Mobile Devices and Services, 2015
Characterizing Information Diets of Social Media Users
J. Kulshrestha, M. B. Zafar, L. E. Espin Noboa, K. Gummadi and S. Gosh
Proceedings of the 9th International AAAI Conference on Web and Social Media (ICWSM 2015), 2015
J. Kulshrestha, M. B. Zafar, L. E. Espin Noboa, K. Gummadi and S. Gosh
Proceedings of the 9th International AAAI Conference on Web and Social Media (ICWSM 2015), 2015
Latent Max-margin Metric Learning for Comparing Video Face Tubes
G. Sharma and P. Pérez
The IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW 2015), 2015
G. Sharma and P. Pérez
The IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW 2015), 2015
Hard to Cheat: A Turing Test based on Answering Questions about Images
M. Malinowski and M. Fritz
Twenty-Ninth AAAI Conference on Artificial Intelligence W6, Beyond the Turing Test (AAAI 2015 W6, Beyond the Turing Test), 2015
(arXiv: 1501.03302) M. Malinowski and M. Fritz
Twenty-Ninth AAAI Conference on Artificial Intelligence W6, Beyond the Turing Test (AAAI 2015 W6, Beyond the Turing Test), 2015
Abstract
Progress in language and image understanding by machines has sparkled the
interest of the research community in more open-ended, holistic tasks, and
refueled an old AI dream of building intelligent machines. We discuss a few
prominent challenges that characterize such holistic tasks and argue for
"question answering about images" as a particular appealing instance of such a
holistic task. In particular, we point out that it is a version of a Turing
Test that is likely to be more robust to over-interpretations and contrast it
with tasks like grounding and generation of descriptions. Finally, we discuss
tools to measure progress in this field.
What Makes for Effective Detection Proposals?
J. Hosang, R. Benenson, P. Dollár and B. Schiele
Technical Report, 2015
(arXiv: 1502.05082) J. Hosang, R. Benenson, P. Dollár and B. Schiele
Technical Report, 2015
Abstract
Current top performing object detectors employ detection proposals to guide
the search for objects, thereby avoiding exhaustive sliding window search
across images. Despite the popularity and widespread use of detection
proposals, it is unclear which trade-offs are made when using them during
object detection. We provide an in-depth analysis of twelve proposal methods
along with four baselines regarding proposal repeatability, ground truth
annotation recall on PASCAL and ImageNet, and impact on DPM and R-CNN detection
performance. Our analysis shows that for object detection improving proposal
localisation accuracy is as important as improving recall. We introduce a novel
metric, the average recall (AR), which rewards both high recall and good
localisation and correlates surprisingly well with detector performance. Our
findings show common strengths and weaknesses of existing methods, and provide
insights and metrics for selecting and tuning proposal methods.
Richer Object Representations for Object Class Detection in Challenging Real World Image
B. Pepik
PhD Thesis, Universität des Saarlandes, 2015
B. Pepik
PhD Thesis, Universität des Saarlandes, 2015
GazeDPM: Early Integration of Gaze Information in Deformable Part Models
I. Shcherbatyi, A. Bulling and M. Fritz
Technical Report, 2015
(arXiv: 1505.05753) I. Shcherbatyi, A. Bulling and M. Fritz
Technical Report, 2015
Abstract
An increasing number of works explore collaborative human-computer systems in
which human gaze is used to enhance computer vision systems. For object
detection these efforts were so far restricted to late integration approaches
that have inherent limitations, such as increased precision without increase in
recall. We propose an early integration approach in a deformable part model,
which constitutes a joint formulation over gaze and visual data. We show that
our GazeDPM method improves over the state-of-the-art DPM baseline by 4% and a
recent method for gaze-supported object detection by 3% on the public POET
dataset. Our approach additionally provides introspection of the learnt models,
can reveal salient image structures, and allows us to investigate the interplay
between gaze attracting and repelling areas, the importance of view-specific
models, as well as viewers' personal biases in gaze patterns. We finally study
important practical aspects of our approach, such as the impact of using
saliency maps instead of real fixations, the impact of the number of fixations,
as well as robustness to gaze estimation error.
Labeled Pupils in the Wild: A Dataset for Studying Pupil Detection in Unconstrained Environments
M. Tonsen, X. Zhang, Y. Sugano and A. Bulling
Technical Report, 2015
(arXiv: 1511.05768) M. Tonsen, X. Zhang, Y. Sugano and A. Bulling
Technical Report, 2015
Abstract
We present labelled pupils in the wild (LPW), a novel dataset of 66
high-quality, high-speed eye region videos for the development and evaluation
of pupil detection algorithms. The videos in our dataset were recorded from 22
participants in everyday locations at about 95 FPS using a state-of-the-art
dark-pupil head-mounted eye tracker. They cover people with different
ethnicities, a diverse set of everyday indoor and outdoor illumination
environments, as well as natural gaze direction distributions. The dataset also
includes participants wearing glasses, contact lenses, as well as make-up. We
benchmark five state-of-the-art pupil detection algorithms on our dataset with
respect to robustness and accuracy. We further study the influence of image
resolution, vision aids, as well as recording location (indoor, outdoor) on
pupil detection performance. Our evaluations provide valuable insights into the
general pupil detection problem and allow us to identify key challenges for
robust pupil detection on head-mounted eye trackers.
2014
Pursuits: Spontaneous Eye-based Interaction for Dynamic Interfaces
M. Vidal, A. Bulling and H. Gellersen
ACM SIGMOBILE Mobile Computing and Communications Review, Volume 18, Number 4, 2014
M. Vidal, A. Bulling and H. Gellersen
ACM SIGMOBILE Mobile Computing and Communications Review, Volume 18, Number 4, 2014
Abstract
Although gaze is an attractive modality for pervasive
interaction, real-world implementation of eye-based interfaces poses
significant challenges. In particular, user calibration is tedious and
time consuming. Pursuits is an innovative interaction technique that
enables truly spontaneous interaction with eye-based interfaces. A user
can simply walk up to the screen and readily interact with moving
targets. Instead of being based on gaze location, Pursuits correlates
eye pursuit movements with objects dynamically moving on the interface.
A Multi-world Approach to Question Answering about Real-world Scenes based on Uncertain Input
M. Malinowski and M. Fritz
Advances in Neural Information Processing Systems 27 (NIPS 2014), 2014
M. Malinowski and M. Fritz
Advances in Neural Information Processing Systems 27 (NIPS 2014), 2014
Ubic: Bridging the Gap Between Digital Cryptography and the Physical World
M. Simkin, A. Bulling, M. Fritz and D. Schröder
Computer Security - ESORICS 2014, 2014
M. Simkin, A. Bulling, M. Fritz and D. Schröder
Computer Security - ESORICS 2014, 2014
Learning Human Pose Estimation Features with Convolutional Networks
A. Jain, J. Tompson, M. Andriluka, G. W. Taylor and C. Bregler
International Conference on Learning Representations 2014 (ICLR 2014), 2014
(arXiv: 1312.7302) A. Jain, J. Tompson, M. Andriluka, G. W. Taylor and C. Bregler
International Conference on Learning Representations 2014 (ICLR 2014), 2014
Abstract
This paper introduces a new architecture for human pose estimation using a
multi- layer convolutional network architecture and a modified learning
technique that learns low-level features and higher-level weak spatial models.
Unconstrained human pose estimation is one of the hardest problems in computer
vision, and our new architecture and learning schema shows significant
improvement over the current state-of-the-art results. The main contribution of
this paper is showing, for the first time, that a specific variation of deep
learning is able to outperform all existing traditional architectures on this
task. The paper also discusses several lessons learned while researching
alternatives, most notably, that it is possible to learn strong low-level
feature detectors on features that might even just cover a few pixels in the
image. Higher-level spatial models improve somewhat the overall result, but to
a much lesser extent then expected. Many researchers previously argued that the
kinematic structure and top-down information is crucial for this domain, but
with our purely bottom up, and weak spatial model, we could improve other more
complicated architectures that currently produce the best results. This mirrors
what many other researchers, like those in the speech recognition, object
recognition, and other domains have experienced.
Multi-view Priors for Learning Detectors from Sparse Viewpoint Data
B. Pepik, M. Stark, P. Gehler and B. Schiele
International Conference on Learning Representations 2014 (ICLR 2014), 2014
(arXiv: 1312.6095) B. Pepik, M. Stark, P. Gehler and B. Schiele
International Conference on Learning Representations 2014 (ICLR 2014), 2014
Abstract
While the majority of today's object class models provide only 2D bounding
boxes, far richer output hypotheses are desirable including viewpoint,
fine-grained category, and 3D geometry estimate. However, models trained to
provide richer output require larger amounts of training data, preferably well
covering the relevant aspects such as viewpoint and fine-grained categories. In
this paper, we address this issue from the perspective of transfer learning,
and design an object class model that explicitly leverages correlations between
visual features. Specifically, our model represents prior distributions over
permissible multi-view detectors in a parametric way -- the priors are learned
once from training data of a source object class, and can later be used to
facilitate the learning of a detector for a target class. As we show in our
experiments, this transfer is not only beneficial for detectors based on
basic-level category representations, but also enables the robust learning of
detectors that represent classes at finer levels of granularity, where training
data is typically even scarcer and more unbalanced. As a result, we report
largely improved performance in simultaneous 2D object localization and
viewpoint estimation on a recent dataset of challenging street scenes.
Multi-View Priors for Learning Detectors from Sparse Viewpoint Data
B. Pepik, M. Stark, P. Gehler and B. Schiele
International Conference on Learning Representations 2014 (ICLR 2014), 2014
(arXiv: http://arxiv.org/abs/1312.6095) B. Pepik, M. Stark, P. Gehler and B. Schiele
International Conference on Learning Representations 2014 (ICLR 2014), 2014
Abstract
While the majority of today's object class models provide only 2D bounding boxes, far richer output hypotheses are desirable including viewpoint, fine-grained category, and 3D geometry estimate. However, models trained to provide richer output require larger amounts of training data, preferably well covering the relevant aspects such as viewpoint and fine-grained categories. In this paper, we address this issue from the perspective of transfer learning, and design an object class model that explicitly leverages correlations between visual features. Specifically, our model represents prior distributions over permissible multi-view detectors in a parametric way -- the priors are learned once from training data of a source object class, and can later be used to facilitate the learning of a detector for a target class. As we show in our experiments, this transfer is not only beneficial for detectors based on basic-level category representations, but also enables the robust learning of detectors that represent classes at finer levels of granularity, where training data is typically even scarcer and more unbalanced. As a result, we report largely improved performance in simultaneous 2D object localization and viewpoint estimation on a recent dataset of challenging street scenes.
Introduction to the PETMEI Special Issue
A. Bulling and R. Bednarik
Journal of Eye Movement Research, Volume 7, Number 3, 2014
A. Bulling and R. Bednarik
Journal of Eye Movement Research, Volume 7, Number 3, 2014
Candidate Sampling for Neuron Reconstruction from Anisotropic Electron Microscopy Volumes
J. Funke, J. N. P. Martel, S. Gerhard, B. Andres, D. C. Ciresan, A. Giusti, L. M. Gambardella, J. Schmidhuber, H. Pfister, A. Cardona and M. Cook
Medical Image Computing and Computer-Assisted Intervention -- MICCAI 2014, 2014
J. Funke, J. N. P. Martel, S. Gerhard, B. Andres, D. C. Ciresan, A. Giusti, L. M. Gambardella, J. Schmidhuber, H. Pfister, A. Cardona and M. Cook
Medical Image Computing and Computer-Assisted Intervention -- MICCAI 2014, 2014
Towards a Visual Turing Challenge
M. Malinowski and M. Fritz
NIPS 2014 Workshop on Learning Semantics, 2014
(arXiv: 1410.8027) M. Malinowski and M. Fritz
NIPS 2014 Workshop on Learning Semantics, 2014
Abstract
As language and visual understanding by machines progresses rapidly, we are observing an increasing interest in holistic architectures that tightly interlink both modalities in a joint learning and inference process. This trend has allowed the community to progress towards more challenging and open tasks and refueled the hope at achieving the old AI dream of building machines that could pass a turing test in open domains. In order to steadily make progress towards this goal, we realize that quantifying performance becomes increasingly difficult. Therefore we ask how we can precisely define such challenges and how we can evaluate different algorithms on this open tasks? In this paper, we summarize and discuss such challenges as well as try to give answers where appropriate options are available in the literature. We exemplify some of the solutions on a recently presented dataset of question-answering task based on real-world indoor images that establishes a visual turing challenge. Finally, we argue despite the success of unique ground-truth annotation, we likely have to step away from carefully curated dataset and rather rely on ’}social consensus{’ as the main driving force to create suitable benchmarks. Providing coverage in this inherently ambiguous output space is an emerging challenge that we face in order to make quantifiable progress in this area.
Expressive Models and Comprehensive Benchmark for 2D Human Pose Estimation
L. Pishchulin, M. Andriluka, P. Gehler and B. Schiele
Parts and Attributes (ECCV 2014 Workshop PA), 2014
L. Pishchulin, M. Andriluka, P. Gehler and B. Schiele
Parts and Attributes (ECCV 2014 Workshop PA), 2014
In the Blink of an Eye - Combining Head Motion and Eye Blink Frequency for Activity Recognition with Google Glass
S. Ishimaru, K. Kunze, K. Kise, J. Weppner, A. Dengel, P. Lukowicz and A. Bulling
Proceedings of the 5th Augmented Human International Conference (AH 2014), 2014
S. Ishimaru, K. Kunze, K. Kise, J. Weppner, A. Dengel, P. Lukowicz and A. Bulling
Proceedings of the 5th Augmented Human International Conference (AH 2014), 2014
Object Disambiguation for Augmented Reality Applications
W.-C. Chiu, G. Johnson, D. McCulley, O. Grau and M. Fritz
Proceedings of the British Machine Vision Conference (BMVC 2014), 2014
W.-C. Chiu, G. Johnson, D. McCulley, O. Grau and M. Fritz
Proceedings of the British Machine Vision Conference (BMVC 2014), 2014
How Good are Detection Proposals, really?
J. Hosang, R. Benenson and B. Schiele
Proceedings of the British Machine Vision Conference (BMVC 2014), 2014
J. Hosang, R. Benenson and B. Schiele
Proceedings of the British Machine Vision Conference (BMVC 2014), 2014
Abstract
Current top performing Pascal VOC object detectors employ detection proposals to guide the search for objects thereby avoiding exhaustive sliding window search across images. Despite the popularity of detection proposals, it is unclear which trade‐offs are made when using them during object detection. We provide an in depth analysis of ten object proposal methods along with four baselines regarding ground truth annotation recall (on Pascal VOC 2007 and ImageNet 2013), repeatability, and impact on DPM detector performance. Our findings show common weaknesses of existing methods, and provide insights to choose the most adequate method for different settings.
Scalable Multitask Representation Learning for Scene Classification
M. Lapin, B. Schiele and M. Hein
Scene Understanding Workshop (SUNw 2014), 2014
M. Lapin, B. Schiele and M. Hein
Scene Understanding Workshop (SUNw 2014), 2014
Learning People Detectors for Tracking in Crowded Scenes
S. Tang, M. Andriluka, A. Milan, K. Schindler, S. Roth and B. Schiele
Scene Understanding Workshop (SUNw 2014), 2014
S. Tang, M. Andriluka, A. Milan, K. Schindler, S. Roth and B. Schiele
Scene Understanding Workshop (SUNw 2014), 2014
High-Resolution 3D Layout from a Single View
M. Z. Zia, M. Stark and K. Schindler
Scene Understanding Workshop (SUNw 2014), 2014
M. Z. Zia, M. Stark and K. Schindler
Scene Understanding Workshop (SUNw 2014), 2014
Pupil: An Open Source Platform for Pervasive Eye Tracking and Mobile Gaze-based Interaction
M. Kassner, W. Patera and A. Bulling
UbiComp’14 Adjunct, 2014
M. Kassner, W. Patera and A. Bulling
UbiComp’14 Adjunct, 2014
Physically Grounded 3D Scene Interpretation with Detailed Object Models
M. Z. Zia, M. Stark and K. Schindler
Vision Meets Cognition Workshop: Functionality, Physics, Intentionality, and Causality (CVPR 2014 Workshop FPIC), 2014
M. Z. Zia, M. Stark and K. Schindler
Vision Meets Cognition Workshop: Functionality, Physics, Intentionality, and Causality (CVPR 2014 Workshop FPIC), 2014
Zero-Shot Learning with Structured Embeddings
Z. Akata, H. Lee and B. Schiele
Technical Report, 2014
(arXiv: 1409.8403) Z. Akata, H. Lee and B. Schiele
Technical Report, 2014
Abstract
Despite significant recent advances in image classification, fine-grained
classification remains a challenge. In the present paper, we address the
zero-shot and few-shot learning scenarios as obtaining labeled data is
especially difficult for fine-grained classification tasks. First, we embed
state-of-the-art image descriptors in a label embedding space using side
information such as attributes. We argue that learning a joint embedding space,
that maximizes the compatibility between the input and output embeddings, is
highly effective for zero/few-shot learning. We show empirically that such
embeddings significantly outperforms the current state-of-the-art methods on
two challenging datasets (Caltech-UCSD Birds and Animals with Attributes).
Second, to reduce the amount of costly manual attribute annotations, we use
alternate output embeddings based on the word-vector representations, obtained
from large text-corpora without any supervision. We report that such
unsupervised embeddings achieve encouraging results, and lead to further
improvements when combined with the supervised ones.
Learning Multi-scale Representations for Material Classification
W. Li and M. Fritz
Technical Report, 2014
(arXiv: 1408.2938) W. Li and M. Fritz
Technical Report, 2014
Abstract
The recent progress in sparse coding and deep learning has made unsupervised
feature learning methods a strong competitor to hand-crafted descriptors. In
computer vision, success stories of learned features have been predominantly
reported for object recognition tasks. In this paper, we investigate if and how
feature learning can be used for material recognition. We propose two
strategies to incorporate scale information into the learning procedure
resulting in a novel multi-scale coding procedure. Our results show that our
learned features for material recognition outperform hand-crafted descriptors
on the FMD and the KTH-TIPS2 material classification benchmarks.
A Pooling Approach to Modelling Spatial Relations for Image Retrieval and Annotation
M. Malinowski and M. Fritz
Technical Report, 2014
(arXiv: 1411.5190) M. Malinowski and M. Fritz
Technical Report, 2014
Abstract
Over the last two decades we have witnessed strong progress on modeling
visual object classes, scenes and attributes that have significantly
contributed to automated image understanding. On the other hand, surprisingly
little progress has been made on incorporating a spatial representation and
reasoning in the inference process. In this work, we propose a pooling
interpretation of spatial relations and show how it improves image retrieval
and annotations tasks involving spatial language. Due to the complexity of the
spatial language, we argue for a learning-based approach that acquires a
representation of spatial relations by learning parameters of the pooling
operator. We show improvements on previous work on two datasets and two
different tasks as well as provide additional insights on a new dataset with an
explicit focus on spatial relations.
Estimating Maximally Probable Constrained Relations by Mathematical Programming
L. Qu and B. Andres
Technical Report, 2014
(arXiv: 1408.0838) L. Qu and B. Andres
Technical Report, 2014
Abstract
Estimating a constrained relation is a fundamental problem in machine
learning. Special cases are classification (the problem of estimating a map
from a set of to-be-classified elements to a set of labels), clustering (the
problem of estimating an equivalence relation on a set) and ranking (the
problem of estimating a linear order on a set). We contribute a family of
probability measures on the set of all relations between two finite, non-empty
sets, which offers a joint abstraction of multi-label classification,
correlation clustering and ranking by linear ordering. Estimating (learning) a
maximally probable measure, given (a training set of) related and unrelated
pairs, is a convex optimization problem. Estimating (inferring) a maximally
probable relation, given a measure, is a 01-linear program. It is solved in
linear time for maps. It is NP-hard for equivalence relations and linear
orders. Practical solutions for all three cases are shown in experiments with
real data. Finally, estimating a maximally probable measure and relation
jointly is posed as a mixed-integer nonlinear program. This formulation
suggests a mathematical programming approach to semi-supervised learning.
Combining Visual Recognition and Computational Linguistics : Linguistic Knowledge for Visual Recognition and Natural Language Descriptions of Visual Content
M. Rohrbach
PhD Thesis, Universität des Saarlandes, 2014
M. Rohrbach
PhD Thesis, Universität des Saarlandes, 2014
Coherent Multi-sentence Video Description with Variable Level of Detail
A. Senina, M. Rohrbach, W. Qiu, A. Friedrich, S. Amin, M. Andriluka, M. Pinkal and B. Schiele
Technical Report, 2014
(arXiv: 1403.6173) A. Senina, M. Rohrbach, W. Qiu, A. Friedrich, S. Amin, M. Andriluka, M. Pinkal and B. Schiele
Technical Report, 2014
Abstract
Humans can easily describe what they see in a coherent way and at varying
level of detail. However, existing approaches for automatic video description
are mainly focused on single sentence generation and produce descriptions at a
fixed level of detail. In this paper, we address both of these limitations: for
a variable level of detail we produce coherent multi-sentence descriptions of
complex videos. We follow a two-step approach where we first learn to predict a
semantic representation (SR) from video and then generate natural language
descriptions from the SR. To produce consistent multi-sentence descriptions, we
model across-sentence consistency at the level of the SR by enforcing a
consistent topic. We also contribute both to the visual recognition of objects
proposing a hand-centric approach as well as to the robust generation of
sentences using a word lattice. Human judges rate our multi-sentence
descriptions as more readable, correct, and relevant than related work. To
understand the difference between more detailed and shorter descriptions, we
collect and analyze a video description corpus of three levels of detail.
2013
Transfer Learning in a Transductive Setting
M. Rohrbach, S. Ebert and B. Schiele
Advances in Neural Information Processing Systems 26 (NIPS 2013), 2013
M. Rohrbach, S. Ebert and B. Schiele
Advances in Neural Information Processing Systems 26 (NIPS 2013), 2013
Abstract
Category models for objects or activities typically rely on supervised
learning requiring sufficiently large training sets. Transferring
knowledge from known categories to novel classes with no or only
a few labels however is far less researched even though it is a common
scenario. In this work, we extend transfer learning with semi-supervised
learning to exploit unlabeled instances of (novel) categories with
no or only a few labeled instances. Our proposed approach Propagated
Semantic Transfer combines three main ingredients. First, we transfer
information from known to novel categories by incorporating external
knowledge, such as linguistic or expert-specified information, e.g.,
by a mid-level layer of semantic attributes. Second, we exploit the
manifold structure of novel classes. More specifically we adapt a
graph-based learning algorithm - so far only used for semi-supervised
learning - to zero-shot and few-shot learning. Third, we improve
the local neighborhood in such graph structures by replacing the
raw feature-based representation with a mid-level object- or attribute-based
representation. We evaluate our approach on three challenging datasets
in two different applications, namely on Animals with Attributes
and ImageNet for image classification and on MPII Composites for
activity recognition. Our approach consistently outperforms state-of-the-art
transfer and semi-supervised approaches on all datasets.
EyeContext: Recognition of High-level Contextual Cues from Human Visual Behaviour
A. Bulling, C. Weichel and H. Gellersen
CHI 2013, The 31st Annual CHI Conference on Human Factors in Computing Systems, 2013
A. Bulling, C. Weichel and H. Gellersen
CHI 2013, The 31st Annual CHI Conference on Human Factors in Computing Systems, 2013
Abstract
Automatic annotation of life logging data is challenging. In this
work we present EyeContext, a system to infer high-level contextual
cues from human visual behaviour. We conduct a user study to record
eye movements of four participants over a full day of their daily
life, totalling 42.5 hours of eye movement data. Participants were
asked to self-annotate four non-mutually exclusive cues: social (interacting
with somebody vs. no interaction), cognitive (concentrated work vs.
leisure), physical (physically active vs. not active), and spatial
(inside vs. outside a building). We evaluate a proof-of-concept EyeContext
system that combines encoding of eye movements into strings and a
spectrum string kernel support vector machine (SVM) classifier. Using
person-dependent training, we obtain a top performance of 85.3%
precision (98.0% recall) for recognising social interactions. Our
results demonstrate the large information content available in long-term
human visual behaviour and opens up new venues for research on eye-based
behavioural monitoring and life logging.
Pursuits: Eye-based Interaction with Moving Targets
M. Vidal, K. Pfeuffer, A. Bulling and H. W. Gellersen
CHI 2013 Extended Abstracts, 2013
M. Vidal, K. Pfeuffer, A. Bulling and H. W. Gellersen
CHI 2013 Extended Abstracts, 2013
Abstract
Eye-based interaction has commonly been based on estimation of eye
gaze direction, to locate objects for interaction. We introduce Pursuits,
a novel and very different eye tracking method that instead is based
on following the trajectory of eye movement and comparing this with
trajectories of objects in the field of view. Because the eyes naturally
follow the trajectory of moving objects of interest, our method is
able to detect what the user is looking at, by matching eye movement
and object movement. We illustrate Pursuits with three applications
that demonstrate how the method facilitates natural interaction with
moving targets.
Learning Smooth Pooling Regions for Visual Recognition
M. Malinowski and M. Fritz
Electronic Proceedings of the British Machine Vision Conference 2013 (BMVC 2013), 2013
M. Malinowski and M. Fritz
Electronic Proceedings of the British Machine Vision Conference 2013 (BMVC 2013), 2013
Abstract
From the early HMAX model to Spatial Pyramid Matching, spatial pooling
has played an important role in visual recognition pipelines. By
aggregating local statistics, it equips the recognition pipelines
with a certain degree of robustness to translation and deformation
yet preserving spatial information. Despite of its predominance in
current recognition systems, we have seen little progress to fully
adapt the pooling strategy to the task at hand. In this paper, we
propose a flexible parameterization of the spatial pooling step and
learn the pooling regions together with the classifier. We investigate
a smoothness regularization term that in conjuncture with an efficient
learning scheme makes learning scalable. Our framework can work with
both popular pooling operators: sum-pooling and max-pooling. Finally,
we show benefits of our approach for object recognition tasks based
on visual words and higher level event recognition tasks based on
object-bank features. In both cases, we improve over the hand-crafted
spatial pooling step showing the importance of its adaptation to
the task.
Segmenting Planar Superpixel Adjacency Graphs w.r.t. Non-planar Superpixel Affinity Graphs
B. Andres, J. Yarkony, B. S. Manjunath, S. Kirchhoff, E. Turetken, C. C. Fowlkes and H. Pfister
Energy Minimization Methods in Computer Vision and Pattern Recognition (EMMCVPR 2013), 2013
B. Andres, J. Yarkony, B. S. Manjunath, S. Kirchhoff, E. Turetken, C. C. Fowlkes and H. Pfister
Energy Minimization Methods in Computer Vision and Pattern Recognition (EMMCVPR 2013), 2013
Eye Pull, Eye Push: Moving Objects between Large Screens and Personal Devices with Gaze & Touch
J. Turner, J. Alexander, A. Bulling, S. Dominik and H. Gellersen
Human-Computer Interaction – INTERACT 2013, 2013
J. Turner, J. Alexander, A. Bulling, S. Dominik and H. Gellersen
Human-Computer Interaction – INTERACT 2013, 2013
Abstract
Previous work has validated the eyes and mobile input as a viable
approach for pointing at, and selecting out of reach objects. This
work presents Eye Pull, Eye Push, a novel interaction concept for
content transfer between public and personal devices using gaze and
touch. We present three techniques that enable this interaction:
Eye Cut & Paste, Eye Drag & Drop, and Eye Summon & Cast. We outline
and discuss several scenarios in which these techniques can be used.
In a user study we found that participants responded well to the
visual feedback provided by Eye Drag & Drop during object movement.
In contrast, we found that although Eye Summon & Cast significantly
improved performance, participants had difficulty coordinating their
hands and eyes during interaction.
Translating Video Content to Natural Language Descriptions
M. Rohrbach, W. Qiu, I. Titov, S. Thater, M. Pinkal and B. Schiele
ICCV 2013, IEEE International Conference on Computer Vision, 2013
M. Rohrbach, W. Qiu, I. Titov, S. Thater, M. Pinkal and B. Schiele
ICCV 2013, IEEE International Conference on Computer Vision, 2013
Abstract
Humans use rich natural language to describe and communicate visual
perceptions. In order to provide natural language descriptions for
visual content, this paper combines two important ingredients. First,
we generate a rich semantic representation of the visual content
including e.g. object and activity labels. To predict the semantic
representation we learn a CRF to model the relationships between
different components of the visual input. And second, we propose
to formulate the generation of natural language as a machine translation
problem using the semantic representation as source language and
the generated sentences as target language. For this we exploit the
power of a parallel corpus of videos and textual descriptions and
adapt statistical machine translation to translate between our two
languages. We evaluate our video descriptions on the TACoS dataset,
which contains video snippets aligned with sentence descriptions.
Using automatic evaluation and human judgments we show significant
improvements over several base line approaches, motivated by prior
work. Our translation approach also shows improvements over related
work on an image description task.
Learning People Detectors for Tracking in Crowded Scenes
S. Tang, M. Andriluka, A. Milan, K. Schindler, S. Roth and B. Schiele
ICCV 2013, IEEE International Conference on Computer Vision, 2013
S. Tang, M. Andriluka, A. Milan, K. Schindler, S. Roth and B. Schiele
ICCV 2013, IEEE International Conference on Computer Vision, 2013
Abstract
People tracking in crowded real-world scenes is challenging due to
frequent and long-term occlusions. Recent tracking methods obtain
the image evidence from object (people) detectors, but typically
use off-the-shelf detectors and treat them as black box components.
In this paper we argue that for best performance one should explicitly
train people detectors on failure cases of the overall tracker instead.
To that end, we first propose a novel joint people detector that
combines a state-of-the-art single person detector with a detector
for pairs of people, which explicitly exploits common patterns of
person-person occlusions across multiple viewpoints that are a common
failure case for tracking in crowded scenes. To explicitly address
remaining failure cases of the tracker we explore two methods. First,
we analyze typical failure cases of trackers and train a detector
explicitly on those failure cases. And second, we train the detector
with the people tracker in the loop, focusing on the most common
tracker failures. We show that our joint multi-person detector significantly
improves both detection accuracy as well as tracker performance,
improving the state-of-the-art on standard benchmarks.
Multi-class Video Co-segmentation with a Generative Multi-video Model
W.-C. Chiu and M. Fritz
2013 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2013), 2013
W.-C. Chiu and M. Fritz
2013 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2013), 2013
A Comparative Study of Modern Inference Techniques for Discrete Energy Minimization Problem
J. H. Kappes, B. Andres, F. A. Hamprecht, C. Schnörr, S. Nowozin, D. Batra, S. Kim, B. X. Kausler, J. Lellmann, N. Komodakis and C. Rother
2013 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2013), 2013
J. H. Kappes, B. Andres, F. A. Hamprecht, C. Schnörr, S. Nowozin, D. Batra, S. Kim, B. X. Kausler, J. Lellmann, N. Komodakis and C. Rother
2013 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2013), 2013
Learnable Pooling Regions for Image Classification
M. Malinowski and M. Fritz
International Conference on Learning Representations Workshop Proceedings (ICLR 2013), 2013
(arXiv: 1301.3516) M. Malinowski and M. Fritz
International Conference on Learning Representations Workshop Proceedings (ICLR 2013), 2013
Abstract
Biologically inspired, from the early HMAX model to Spatial Pyramid Matching,
pooling has played an important role in visual recognition pipelines. Spatial
pooling, by grouping of local codes, equips these methods with a certain degree
of robustness to translation and deformation yet preserving important spatial
information. Despite the predominance of this approach in current recognition
systems, we have seen little progress to fully adapt the pooling strategy to
the task at hand. This paper proposes a model for learning task dependent
pooling scheme -- including previously proposed hand-crafted pooling schemes as
a particular instantiation. In our work, we investigate the role of different
regularization terms showing that the smooth regularization term is crucial to
achieve strong performance using the presented architecture. Finally, we
propose an efficient and parallel method to train the model. Our experiments
show improved performance over hand-crafted pooling schemes on the CIFAR-10 and
CIFAR-100 datasets -- in particular improving the state-of-the-art to 56.29% on
the latter.
Signal Processing Technologies for Activity-aware Smart Textiles
D. Roggen, G. Tröster and A. Bulling
Multidisciplinary Know-How for Smart-Textiles Developers, 2013
D. Roggen, G. Tröster and A. Bulling
Multidisciplinary Know-How for Smart-Textiles Developers, 2013
Abstract
Garments made of smart textiles have an enormous potential for embedding sensors in close proximity to the body in an unobtrusive and comfortable manner. Combined with signal processing and pattern recognition technologies, complex high-level information about human behaviors or situations can be inferred from the sensor data. The goal of this chapter is to introduce the reader to the design of activity-aware systems that use body-worn sensors, such as those that can be made available through smart textiles. We start this chapter by emphasizing recent trends towards ‘}wearable{’ sensing and computing and we present several examples of activity-aware applications. Then we outline the role that smart textiles can play in activity-aware applications, but also the challenges that they pose. We conclude by discussing the design process followed to devise activity-aware systems: the choice of sensors, the available data processing methods, and the evaluation techniques. We discuss recent data processing methods that address the challenges resulting from the use of smart textiles.
Dynamic Feature Selection for Classification on a Budget
S. Karayev, M. Fritz and T. Darrell
Prediction with Sequential Models (ICML 2013 Workshop), 2013
S. Karayev, M. Fritz and T. Darrell
Prediction with Sequential Models (ICML 2013 Workshop), 2013
Qualitative Activity Recognition of Weight Lifting Exercises
E. Velloso, A. Bulling, H. Gellersen, W. Ugulino and H. Fuks
Proceedings of the 4th Augmented Human International Conference (AH 2013), 2013
E. Velloso, A. Bulling, H. Gellersen, W. Ugulino and H. Fuks
Proceedings of the 4th Augmented Human International Conference (AH 2013), 2013
Abstract
Research on human activity recognition has traditionally focused on
discriminating between different activities, i.e. to predict \textquoteleft}{\textquoteleft}which{\textquoteright}{\textquoteright}
activity was performed at a specific point in time. The quality of
executing an activity, the {\textquoteleft}{\textquoteleft}how (well){\textquoteright}{\textquoteright,
has only received little attention so far, even though it potentially
provides useful information for a large variety of applications,
such as sports training. In this work we first define quality of
execution and investigate three aspects that pertain to qualitative
activity recognition: the problem of specifying correct execution,
the automatic and robust detection of execution mistakes, and how
to provide feedback on the quality of execution to the user. We illustrate
our approach on the example problem of qualitatively assessing and
providing feedback on weight lifting exercises. In two user studies
we try out a sensor- and a model-based approach to qualitative activity
recognition. Our results underline the potential of model-based assessment
and the positive impact of real-time user feedback on the quality
of execution.
Collecting a Large-scale Dataset of Fine-grained Cars
J. Krause, J. Deng, M. Stark and L. Fei-Fei
Second Workshop on Fine-Grained Visual Categorization (FGVC2), 2013
J. Krause, J. Deng, M. Stark and L. Fei-Fei
Second Workshop on Fine-Grained Visual Categorization (FGVC2), 2013
Modeling Instance Appearance for Recognition - Can We Do Better Than EM?
A. Chou, H. Wang, M. Stark and D. Koller
Structured Prediction : Tractability, Learning, and Inference (CVPR 2013 Workshop SPTLI), 2013
A. Chou, H. Wang, M. Stark and D. Koller
Structured Prediction : Tractability, Learning, and Inference (CVPR 2013 Workshop SPTLI), 2013
Grounding Action Descriptions in Videos
M. Regneri, M. Rohrbach, D. Wetzel, S. Thater, B. Schiele and M. Pinkal
Transactions of the Association for Computational Linguistics, Volume 1, 2013
M. Regneri, M. Rohrbach, D. Wetzel, S. Thater, B. Schiele and M. Pinkal
Transactions of the Association for Computational Linguistics, Volume 1, 2013
Pursuit Calibration: Making Gaze Calibration Less Tedious and More Flexible
K. Pfeuffer, M. Vidal, J. Turner, A. Bulling and H. Gellersen
UIST’13, ACM Symposium on User Interface Software and Technology, 2013
K. Pfeuffer, M. Vidal, J. Turner, A. Bulling and H. Gellersen
UIST’13, ACM Symposium on User Interface Software and Technology, 2013
Abstract
Eye gaze is a compelling interaction modality but requires a user
calibration before interaction can commence. State of the art procedures
require the user to fixate on a succession of calibration markers,
a task that is often experienced as difficult and tedious. We present
a novel approach, pursuit calibration, that instead uses moving targets
for calibration. Users naturally perform smooth pursuit eye movements
when they follow a moving target, and we use correlation of eye and
target movement to detect the users attention and to sample data
for calibration. Because the method knows when the users is attending
to a target, the calibration can be performed implicitly, which enables
more flexible design of the calibration task. We demonstrate this
in application examples and user studies, and show that pursuit calibration
is tolerant to interruption, can blend naturally with applications,
and is able to calibrate users without their awareness.
Proceedings of the 4th Augmented Human International Conference
A. Schmidt, A. Bulling and C. Holz (Eds.)
ACM, 2013
A. Schmidt, A. Bulling and C. Holz (Eds.)
ACM, 2013
Abstract
We are very happy to present the proceedings of the 4th Augmented
Human International Conference (Augmented Human 2013). Augmented
Human 2013 focuses on augmenting human capabilities through technology
for increased well-being and enjoyable human experience. The conference
is in cooperation with ACM SIGCHI, with its proceedings to be archived
in ACM\textquoteright}s Digital Library. With technological advances,
computing has progressively moved beyond the desktop into new physical
and social contexts. As physical artifacts gain new computational
behaviors, they become reprogrammable, customizable, repurposable,
and interoperable in rich ecologies and diverse contexts. They also
become more complex, and require intense design effort in order to
be functional, usable, and enjoyable. Designing such systems requires
interdisciplinary thinking. Their creation must not only encompass
software, electronics, and mechanics, but also the system{\textquoterights
physical form and behavior, its social and physical milieu, and beyond.
2012
Timely Object Recognition
S. Karayev, T. Baumgarnter, M. Fritz and T. Darrell
Advances in Neural Information Processing Systems 25 (NIPS 2012), 2012
S. Karayev, T. Baumgarnter, M. Fritz and T. Darrell
Advances in Neural Information Processing Systems 25 (NIPS 2012), 2012
Detection and Tracking of Occluded People
S. Tang, M. Andriluka and B. Schiele
Electronic Proceedings of the British Machine Vision Conference 2012 (BMVC 2012), 2012
S. Tang, M. Andriluka and B. Schiele
Electronic Proceedings of the British Machine Vision Conference 2012 (BMVC 2012), 2012
Teaching 3D Geometry to Deformable Part Models
B. Pepik, M. Stark, P. Gehler and B. Schiele
2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2012), 2012
B. Pepik, M. Stark, P. Gehler and B. Schiele
2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2012), 2012
Abstract
Current object class recognition systems typically target 2D bounding box localization, encouraged by benchmark data sets, such as Pascal VOC. While this seems suitable for the detection of individual objects, higher-level applications such as 3D scene understanding or 3D object tracking would benefit from more fine-grained object hypotheses incorporating 3D geometric information, such as viewpoints or the locations of individual parts. In this paper, we help narrowing the representational gap between the ideal input of a scene understanding system and object class detector output, by designing a detector particularly tailored towards 3D geometric reasoning. In particular, we extend the successful discriminatively trained deformable part models to include both estimates of viewpoint and 3D parts that are consistent across viewpoints. We experimentally verify that adding 3D geometric information comes at minimal performance loss w.r.t. 2D bounding box localization, but outperforms prior work in 3D viewpoint estimation and ultra-wide baseline matching.
Articulated People Detection and Pose Estimation: Reshaping the Future
L. Pishchulin, A. Jain, M. Andriluka, T. Thormaehlen and B. Schiele
2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2012), 2012
L. Pishchulin, A. Jain, M. Andriluka, T. Thormaehlen and B. Schiele
2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2012), 2012
Abstract
State-of-the-art methods for human detection and pose estimation require many training samples for best performance. While large, manually collected datasets exist, the captured variations w.r.t. appearance, shape and pose are often uncontrolled thus limiting the overall performance. In order to overcome this limitation we propose a new technique to extend an existing training set that allows to
explicitly control pose and shape variations. For this we build on recent advances in
computer graphics to generate samples with realistic appearance and background
while modifying body shape and pose.
We validate the effectiveness of our approach on the task of articulated human detection and articulated pose estimation.
We report close to state of the art results on the popular Image Parsing human pose estimation benchmark and demonstrate superior performance for articulated human detection. In addition we define a new challenge of combined articulated human detection and pose estimation in real-world scenes.
Discriminative Appearance Models for Pictorial Structures
M. Andriluka, S. Roth and B. Schiele
International Journal of Computer Vision, Volume 99, Number 3, 2012
M. Andriluka, S. Roth and B. Schiele
International Journal of Computer Vision, Volume 99, Number 3, 2012
Abstract
In this paper we consider people detection and articulated pose estimation, two closely related and challenging problems in computer vision. Conceptually, both of these problems can be addressed within the pictorial structures framework (Felzenszwalb and Huttenlocher in Int. J. Comput. Vis. 61(1):55–79, 2005; Fischler and Elschlager in IEEE Trans. Comput. C-22(1):67–92, 1973), even though previous approaches have not shown such generality. A principal difficulty for such a general approach is to model the appearance of body parts. The model has to be discriminative enough to enable reliable detection in cluttered scenes and general enough to capture highly variable appearance. Therefore, as the first important component of our approach, we propose a discriminative appearance model based on densely sampled local descriptors and AdaBoost classifiers. Secondly, we interpret the normalized margin of each classifier as likelihood in a generative model and compute marginal posteriors for each part using belief propagation. Thirdly, non-Gaussian relationships between parts are represented as Gaussians in the coordinate system of the joint between the parts. Additionally, in order to cope with shortcomings of tree-based pictorial structures models, we augment our model with additional repulsive factors in order to discourage overcounting of image evidence. We demonstrate that the combination of these components within the pictorial structures framework results in a generic model that yields state-of-the-art performance for several datasets on a variety of tasks: people detection, upper body pose estimation, and full body pose estimation.
Kernel Density Topic Models: Visual Topics Without Visual Words
K. Rematas, M. Fritz and T. Tuytelaars
NIPS 2012 Workshop Modern Nonparametric Methods in Machine Learning, 2012
K. Rematas, M. Fritz and T. Tuytelaars
NIPS 2012 Workshop Modern Nonparametric Methods in Machine Learning, 2012
Semi-supervised Learning for Image Classification
S. Ebert
PhD Thesis, Universität des Saarlandes, 2012
S. Ebert
PhD Thesis, Universität des Saarlandes, 2012
Abstract
Object class recognition is an active topic in computer vision still
presenting many challenges. In most approaches, this task is addressed
by supervised learning algorithms that need a large quantity of labels
to perform well. This leads either to small datasets (< 10,000 images)
that capture only a subset of the real-world class distribution (but
with a controlled and verified labeling procedure), or to large datasets
that are more representative but also add more label noise. Therefore,
semi-supervised learning is a promising direction. It requires only
few labels while simultaneously making use of the vast amount of
images available today. We address object class recognition with
semi-supervised learning. These algorithms depend on the underlying
structure given by the data, the image description, and the similarity
measure, and the quality of the labels. This insight leads to the
main research questions of this thesis: Is the structure given by
labeled and unlabeled data more important than the algorithm itself?
Can we improve this neighborhood structure by a better similarity
metric or with more representative unlabeled data? Is there a connection
between the quality of labels and the overall performance and how
can we get more representative labels? We answer all these questions,
i.e., we provide an extensive evaluation, we propose several graph
improvements, and we introduce a novel active learning framework
to get more representative labels.
2011
South by South-east or Sitting at the Desk: Can Orientation be a Place?
U. Blanke, R. Rehner and B. Schiele
15th Annual International Symposium on Wearable Computers (ISWC 2011), 2011
U. Blanke, R. Rehner and B. Schiele
15th Annual International Symposium on Wearable Computers (ISWC 2011), 2011
Abstract
Location is a key information for context-aware systems. While coarse-grained
indoor location estimates may be obtained quite easily (e.g. based on WiFi or
GSM), finer-grained estimates typically require additional infrastructure (e.g.
ultrasound). This work explores an approach to estimate significant places,
e.g., at the fridge, with no additional setup or infrastructure. We use a
pocket-based inertial measurement sensor, which can be found in many recent
phones. We analyze how the spatial layout such as geographic orientation of
buildings, arrangement and type of furniture can serve as the basis to estimate
typical places in a daily scenario. Initial experiments reveal that our
approach can detect fine-grained locations without relying on any
infrastructure or additional devices.
Recovering Intrinsic Images with a Global Sparsity Prior on Reflectance
P. Gehler, C. Rother, M. Kiefel, L. Zhang and B. Schölkopf
Advances in Neural Information Processing Systems 24 (NIPS 2011), 2011
P. Gehler, C. Rother, M. Kiefel, L. Zhang and B. Schölkopf
Advances in Neural Information Processing Systems 24 (NIPS 2011), 2011
Abstract
We address the challenging task of decoupling material properties from lighting
properties given a single image. In the last two decades virtually all works
have concentrated on exploiting edge information to address this problem. We
take a different route by introducing a new prior on reflectance, that models
reflectance values as being drawn from a sparse set of basis colors. This
results in a Random Field model with global, latent variables (basis colors)
and pixel-accurate output reflectance values. We show that without edge
information high-quality results can be achieved, that are on par with methods
exploiting this source of information. Finally, we are able to improve on
state-of-the-art results by integrating edge information into our model. We
believe that our new approach is an excellent starting point for future
developments in this field.
Revisiting 3D Geometric Models for Accurate Object Shape and Pose
M. Z. Zia, M. Stark, B. Schiele and K. Schindler
IEEE International Conference on Computer Vision (ICCV 3dRR 2011), 2011
M. Z. Zia, M. Stark, B. Schiele and K. Schindler
IEEE International Conference on Computer Vision (ICCV 3dRR 2011), 2011
Abstract
Geometric 3D reasoning has received renewed attention recently, in the context
of visual scene understanding. The level of geometric detail, however, is
typically limited to qualitative or coarse-grained quantitative
representations. This is linked to the fact that today's object class detectors
are tuned towards robust 2D matching rather than accurate 3D pose estimation,
encouraged by 2D bounding box-based benchmarks such as Pascal VOC. In this
paper, we therefore revisit ideas from the early days of computer vision,
namely, 3D geometric object class representations for recognition. These
representations can recover geometrically far more accurate object hypotheses
than just 2D bounding boxes, including relative 3D positions of object parts.
In combination with recent robust techniques for shape description and
inference, our approach outperforms state-of-the-art results in 3D pose
estimation, while at the same time improving 2D localization. In a series of
experiments, we analyze our approach in detail, and demonstrate novel
applications enabled by our geometric object class representation, such as
fine-grained categorization of cars according to their 3D geometry and
ultra-wide baseline matching.
The Benefits of Dense Stereo for Pedestrian Detection
C. G. Keller, M. Enzweiler, M. Rohrbach, D. F. Llorca, C. Schnörr and D. M. Gavrila
IEEE Transactions on Intelligent Transportation Systems, Volume 12, Number 4, 2011
C. G. Keller, M. Enzweiler, M. Rohrbach, D. F. Llorca, C. Schnörr and D. M. Gavrila
IEEE Transactions on Intelligent Transportation Systems, Volume 12, Number 4, 2011
Abstract
This paper presents a novel pedestrian detection system for intelligent
vehicles. We propose the use of dense stereo for both the generation of regions
of interest and pedestrian classification. Dense stereo allows the dynamic
estimation of camera parameters and the road profile, which, in turn, provides
strong scene constraints on possible pedestrian locations. For classification,
we extract spatial features (gradient orientation histograms) directly from
dense depth and intensity images. Both modalities are represented in terms of
individual feature spaces, in which discriminative classifiers (linear support
vector machines) are learned. We refrain from the construction of a joint
feature space but instead employ a fusion of depth and intensity on the
classifier level. Our experiments involve challenging image data captured in
complex urban environments (i.e., undulating roads and speed bumps). Our
results show a performance improvement by up to a factor of 7.5 at the
classification level and up to a factor of 5 at the tracking level (reduction
in false alarms at constant detection rates) over a system with static scene
constraints and intensity-only classification.
Recognition of Hearing Needs From Body and Eye Movements to Improve Hearing Instruments
B. Tessendorf, A. Bulling, D. Roggen, T. Stiefmeier, M. Feilner, P. Derleth and G. Tröster
Pervasive Computing, 2011
B. Tessendorf, A. Bulling, D. Roggen, T. Stiefmeier, M. Feilner, P. Derleth and G. Tröster
Pervasive Computing, 2011
Abstract
Hearing instruments (HIs) have emerged as true pervasive computers
as they continuously adapt the hearing program to the user\textquoterights
context. However, current HIs are not able to distinguish different
hearing needs in the same acoustic environment. In this work, we
explore how information derived from body and eye movements can be
used to improve the recognition of such hearing needs. We conduct
an experiment to provoke an acoustic environment in which different
hearing needs arise: active conversation and working while colleagues
are having a conversation in a noisy office environment. We record
body movements on nine body locations, eye movements using electrooculography
(EOG), and sound using commercial HIs for eleven participants. Using
a support vector machine (SVM) classifier and person-independent
training we improve the accuracy of 77% based on sound to an accuracy
of 92% using body movements. With a view to a future implementation
into a HI we then perform a detailed analysis of the sensors attached
to the head. We achieve the best accuracy of 86% using eye movements
compared to 84% for head movements. Our work demonstrates the potential
of additional sensor modalities for future HIs and motivates to investigate
the wider applicability of this approach on further hearing situations
and needs.
Learning Output Kernels with Block Coordinate Descent
F. Dinuzzo, C. S. Ong, P. Gehler and G. Pillonetto
Proceedings of the 28th Internationl Conference on Machine Learning (ICML 2011), 2011
F. Dinuzzo, C. S. Ong, P. Gehler and G. Pillonetto
Proceedings of the 28th Internationl Conference on Machine Learning (ICML 2011), 2011
Abstract
We propose a method to learn simultaneously a vector-valued function and a
kernel between its components. The obtained kernel can be used both to improve
learning performance and to reveal structures in the output space which may be
important in their own right. Our method is based on the solution of a suitable
regularization problem over a reproducing kernel Hilbert space of vector-valued
functions. Although the regularized risk functional is non-convex, we show that
it is invex, implying that all local minimizers are global minimizers. We
derive a block-wise coordinate descent method that efficiently exploits the
structure of the objective functional. Then, we empirically demonstrate that
the proposed method can improve classification accuracy. Finally, we provide a
visual interpretation of the learned kernel matrix for some well known
datasets.
Branch&Rank: Non-linear Object Detection
A. Lehmann, P. Gehler and L. Van Gool
Proceedings of the British Machine Vision Conference 2011 (BMVC 2011), 2011
A. Lehmann, P. Gehler and L. Van Gool
Proceedings of the British Machine Vision Conference 2011 (BMVC 2011), 2011
Abstract
Branch&rank is an object detection scheme that overcomes the inherent
limitation of branch&bound: this method works with arbitrary (classifier)
functions whereas tight bounds exist only for simple functions. Objects are
usually detected with less than 100 classifier evaluation, which paves the way
for using strong (and thus costly) classifiers: We utilize non-linear SVMs with
RBF- 2 kernels without a cascade-like approximation. Our approach features
three key components: a ranking function that operates on sets of hypotheses
and a grouping of these into different tasks. Detection efficiency results from
adaptively sub-dividing the object search space into decreasingly smaller sets.
This is inherited from branch&bound, while the ranking function supersedes a
tight bound which is often unavailable (except for too simple function
classes). The grouping makes the system effective: it separates image
classification from object recognition, yet combines them in a single,
structured SVM formulation. A novel aspect of branch&rank is that a better
ranking function is expected to decrease the number of classifier calls during
detection. We demonstrate the algorithmic properties using the VOC'07 dataset.
2010
Back to the Future: Learning Shape Models from 3D CAD Data
M. Stark, M. Goesele and B. Schiele
21st British Machine Vision Conference (BMVC 2010), 2010
M. Stark, M. Goesele and B. Schiele
21st British Machine Vision Conference (BMVC 2010), 2010
Abstract
Recognizing 3D objects from arbitrary view points is one of the
most fundamental problems in computer vision. A major challenge lies
in the transition between the 3D geometry of objects and 2D
representations that can be robustly matched to natural images. Most
approaches thus rely on 2D natural images either as the sole source of
training data for building an implicit 3D representation, or by
enriching 3D models with natural image features.
In this paper, we go back to the ideas from the early days of computer
vision, by using 3D object models as the only source of information for
building a multi-view object class detector. In particular, we use
these models for learning 2D shape that can be robustly matched to 2D
natural images. Our experiments confirm the validity of our approach,
which outperforms current state-of-the-art techniques on a multi-view
detection data set.
All for one or one for all? – Combining Heterogeneous Features for Activity Spotting
U. Blanke, M. Kreil, B. Schiele, P. Lukowicz, B. Sick and T. Gruber
2010 8th IEEE International Conference on Pervasive Computing and Communications Workshops : PerCom Workshops 2010 : 7th IEEE International Workshop on Context Modeling and Reasoning (CoMoRea 2010), 2010
U. Blanke, M. Kreil, B. Schiele, P. Lukowicz, B. Sick and T. Gruber
2010 8th IEEE International Conference on Pervasive Computing and Communications Workshops : PerCom Workshops 2010 : 7th IEEE International Workshop on Context Modeling and Reasoning (CoMoRea 2010), 2010
Size Matters: Metric Visual Search Constraints from Monocular Metadata
M. Fritz, K. Saenko and T. Darrell
Advances in Neural Information Processing Systems 23 (NIPS 2010), 2010
M. Fritz, K. Saenko and T. Darrell
Advances in Neural Information Processing Systems 23 (NIPS 2010), 2010
Extracting Structures in Image Collections for Object Recognition
S. Ebert, D. Larlus and B. Schiele
Computer Vision - ECCV 2010, 2010
S. Ebert, D. Larlus and B. Schiele
Computer Vision - ECCV 2010, 2010
Abstract
Many computer vision methods rely on annotated image databases without taking
advantage of the increasing number of unlabeled images available. This paper
explores an alternative approach involving unsupervised structure discovery and
semi-supervised learning (SSL) in image collections. Focusing on object
classes, the first part of the paper contributes with an extensive evaluation of
state-of-the-art image representations underlining the decisive influence of the
local neighborhood structure, its direct consequences on SSL results, and the
importance of developing powerful object representations. In a second part, we
propose and explore promising directions to improve results by looking at the
local topology between images and feature combination strategies.
Combining Language Sources and Robust Semantic Relatedness for Attribute-Based Knowledge Transfer
M. Rohrbach, M. Stark, G. Szarvas and B. Schiele
First International Workshop on Parts and Attributes in Conjunction with ECCV 2010, 2010
M. Rohrbach, M. Stark, G. Szarvas and B. Schiele
First International Workshop on Parts and Attributes in Conjunction with ECCV 2010, 2010
Abstract
Knowledge transfer between object classes has been identified as an important
tool for scalable recognition. However, determining which knowledge to transfer
where remains a key challenge. While most approaches employ varying levels of
human supervision, we follow the idea of mining linguistic knowledge bases to
automatically infer transferable knowledge. In contrast to previous work, we
explicitly aim to design robust semantic relatedness measures and to combine
different language sources for attribute-based knowledge transfer. On the
challenging Animals with Attributes (AwA) data set, we report largely improved
attribute-based zero-shot object class recognition performance that matches the
performance of human supervision.
Vision Based Victim Detection from Unmanned Aerial Vehicles
M. Andriluka, P. Schnitzspan, J. Meyer, S. Kohlbrecher, K. Petersen, O. von Stryk, S. Roth and B. Schiele
2010 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2010
M. Andriluka, P. Schnitzspan, J. Meyer, S. Kohlbrecher, K. Petersen, O. von Stryk, S. Roth and B. Schiele
2010 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2010
Abstract
Finding injured humans is one of the primary
goals of any search and rescue operation. The aim of this paper
is to address the task of automatically finding people lying on
the ground in images taken from the on-board camera of an
unmanned aerial vehicle (UAV).
In this paper we evaluate various state-of-the-art visual
people detection methods in the context of vision based victim
detection from an UAV. The top performing approaches in
this comparison are those that rely on flexible part-based
representations and discriminatively trained part detectors. We
discuss their strengths and weaknesses and demonstrate that by
combining multiple models we can increase the reliability of the
system. We also demonstrate that the detection performance
can be substantially improved by integrating the height and
pitch information provided by on-board sensors. Jointly these
improvements allow us to significantly boost the detection
performance over the current de-facto standard, which provides
a substantial step towards making autonomous victim detection
for UAVs practical.
Monocular 3D Pose Estimation and Tracking by Detection
M. Andriluka, S. Roth and B. Schiele
2010 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2010), 2010
M. Andriluka, S. Roth and B. Schiele
2010 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2010), 2010
Abstract
Automatic recovery of 3D human pose from monocular image sequences is a
challenging and important research topic with numerous applications. Although
current methods are able to recover 3D pose for a single person in controlled
environments, they are severely challenged by real-world scenarios, such as
crowded street scenes. To address this problem, we propose a three-stage
process building on a number of recent advances. The first stage obtains an
initial estimate of the 2D articulation and viewpoint of the person from single
frames. The second stage allows early data association across frames based on
tracking-by-detection. These two stages successfully accumulate the available
2D image evidence into robust estimates of 2D limb positions over short image
sequences (= tracklets). The third and final stage uses those tracklet-based
estimates as robust image observations to reliably recover 3D pose. We
demonstrate state-of-the-art performance on the HumanEva II benchmark, and also
show the applicability of our approach to articulated 3D tracking in realistic
street conditions.
What helps Where - and Why? Semantic Relatedness for Knowledge Transfer
M. Rohrbach, M. Stark, G. Szarvas, I. Gurevych and B. Schiele
2010 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2010), 2010
M. Rohrbach, M. Stark, G. Szarvas, I. Gurevych and B. Schiele
2010 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2010), 2010
Abstract
Remarkable performance has been reported to recognize single object classes.
Scalability to large numbers of classes however remains an important challenge
for today's recognition methods. Several authors have promoted knowledge
transfer between classes as a key ingredient to address this challenge.
However, in previous work the decision which knowledge to transfer has required
either manual supervision or at least a few training examples limiting the
scalability of these approaches. In this work we explicitly address the
question of how to automatically decide which information to transfer between
classes without the need of any human intervention. For this we tap into
linguistic knowledge bases to provide the semantic link between sources (what)
and targets (where) of knowledge transfer. We provide a rigorous experimental
evaluation of different knowledge bases and state-of-the-art techniques from
Natural Language Processing which goes far beyond the limited use of language
in related work. We also give insights into the applicability (why) of
different knowledge sources and similarity measures for knowledge transfer.
Automatic Discovery of Meaningful Object Parts with Latent CRFs
P. Schnitzspan, S. Roth and B. Schiele
2010 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2010), 2010
P. Schnitzspan, S. Roth and B. Schiele
2010 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2010), 2010
Abstract
Object recognition is challenging due to high intra-class variability caused,
e.g., by articulation, viewpoint changes, and partial occlusion. Successful
methods need to strike a balance between being flexible enough to model such
variation and discriminative enough to detect objects in cluttered, real world
scenes. Motivated by these challenges we propose a latent conditional random
field (CRF) based on a flexible assembly of parts. By modeling part labels as
hidden nodes and developing an EM algorithm for learning from class labels
alone, this new approach enables the automatic discovery of semantically
meaningful object part representations. To increase the flexibility and
expressiveness of the model, we learn the pairwise structure of the underlying
graphical model at the level of object part interactions. Efficient
gradient-based techniques are used to estimate the structure of the domain of
interest and carried forward to the multi-label or object part case. Our
experiments illustrate the meaningfulness of the discovered parts and
demonstrate state-of-the-art performance of the approach.
2004