Mohamed Omran (PhD Student)

Abstract
Humans are at the centre of a significant amount of research in computer vision.
Endowing machines with the ability to perceive people from visual data is an immense
scientific challenge with a high degree of direct practical relevance. Success in automatic
perception can be measured at different levels of abstraction, and this will depend on
which intelligent behaviour we are trying to replicate: the ability to localise persons in
an image or in the environment, understanding how persons are moving at the skeleton
and at the surface level, interpreting their interactions with the environment including
with other people, and perhaps even anticipating future actions. In this thesis we tackle
different sub-problems of the broad research area referred to as "looking at people",
aiming to perceive humans in images at different levels of granularity.
We start with bounding box-level pedestrian detection: We present a retrospective
analysis of methods published in the decade preceding our work, identifying various
strands of research that have advanced the state of the art. With quantitative exper-
iments, we demonstrate the critical role of developing better feature representations
and having the right training distribution. We then contribute two methods based
on the insights derived from our analysis: one that combines the strongest aspects of
past detectors and another that focuses purely on learning representations. The latter
method outperforms more complicated approaches, especially those based on hand-
crafted features. We conclude our work on pedestrian detection with a forward-looking
analysis that maps out potential avenues for future research.
We then turn to pixel-level methods: Perceiving humans requires us to both separate
them precisely from the background and identify their surroundings. To this end, we
introduce Cityscapes, a large-scale dataset for street scene understanding. This has since
established itself as a go-to benchmark for segmentation and detection. We additionally
develop methods that relax the requirement for expensive pixel-level annotations, focusing
on the task of boundary detection, i.e. identifying the outlines of relevant objects and
surfaces. Next, we make the jump from pixels to 3D surfaces, from localising and
labelling to fine-grained spatial understanding. We contribute a method for recovering
3D human shape and pose, which marries the advantages of learning-based and model-
based approaches.
We conclude the thesis with a detailed discussion of benchmarking practices in
computer vision. Among other things, we argue that the design of future datasets
should be driven by the general goal of combinatorial robustness besides task-specific
considerations.

Encouraged by the recent progress in pedestrian detection, we investigate the gap between current state-of-the-art methods

and the “perfect single frame detector”. We enable our analysis by creating a human baseline for pedestrian detection (over the Caltech

pedestrian dataset). After manually clustering the frequent errors of a top detector, we characterise both localisation and background-

versus-foreground errors.

To address localisation errors we study the impact of training annotation noise on the detector performance, and show that we can

improve results even with a small portion of sanitised training data. To address background/foreground discrimination, we study convnets

for pedestrian detection, and discuss which factors affect their performance.

Other than our in-depth analysis, we report top performance on the Caltech pedestrian dataset, and provide a new sanitised set of

training and test annotations.

State-of-the-art learning based boundary detection methods require extensive

training data. Since labelling object boundaries is one of the most expensive

types of annotations, there is a need to relax the requirement to carefully

annotate images to make both the training more affordable and to extend the

amount of training data. In this paper we propose a technique to generate

weakly supervised annotations and show that bounding box annotations alone

suffice to reach high-quality object boundaries without using any

object-specific boundary annotations. With the proposed weak supervision

techniques we achieve the top performance on the object boundary detection

task, outperforming by a large margin the current fully supervised

state-of-the-art methods.