Andreas Bulling (Professor)

Automatic detection of emergent leaders in small groups from nonverbal
behaviour is a growing research topic in social signal processing but existing
methods were evaluated on single datasets -- an unrealistic assumption for
real-world applications in which systems are required to also work in settings
unseen at training time. It therefore remains unclear whether current methods
for emergent leadership detection generalise to similar but new settings and to
which extent. To overcome this limitation, we are the first to study a
cross-dataset evaluation setting for the emergent leadership detection task. We
provide evaluations for within- and cross-dataset prediction using two current
datasets (PAVIS and MPIIGroupInteraction), as well as an investigation on the
robustness of commonly used feature channels (visual focus of attention, body
pose, facial action units, speaking activity) and online prediction in the
cross-dataset setting. Our evaluations show that using pose and eye contact
based features, cross-dataset prediction is possible with an accuracy of 0.68,
as such providing another important piece of the puzzle towards emergent
leadership detection in the real world.

Internal thought refers to the process of directing attention away from a
primary visual task to internal cognitive processing. Internal thought is a
pervasive mental activity and closely related to primary task performance. As
such, automatic detection of internal thought has significant potential for
user modelling in intelligent interfaces, particularly for e-learning
applications. Despite the close link between the eyes and the human mind, only
a few studies have investigated vergence behaviour during internal thought and
none has studied moment-to-moment detection of internal thought from gaze.
While prior studies relied on long-term data analysis and required a large
number of gaze characteristics, we describe a novel method that is
computationally light-weight and that only requires eye vergence information
that is readily available from binocular eye trackers. We further propose a
novel paradigm to obtain ground truth internal thought annotations that
exploits human blur perception. We evaluate our method for three increasingly
challenging detection tasks: (1) during a controlled math-solving task, (2)
during natural viewing of lecture videos, and (3) during daily activities, such
as coding, browsing, and reading. Results from these evaluations demonstrate
the performance and robustness of vergence-based detection of internal thought
and, as such, open up new directions for research on interfaces that adapt to
shifts of mental attention.

Recent methods to automatically calibrate stationary eye trackers were shown
to effectively reduce inherent calibration distortion. However, these methods
require additional information, such as mouse clicks or on-screen content. We
propose the first method that only requires users' eye movements to reduce
calibration distortion in the background while users naturally look at an
interface. Our method exploits that calibration distortion makes straight
saccade trajectories appear curved between the saccadic start and end points.
We show that this curving effect is systematic and the result of distorted gaze
projection plane. To mitigate calibration distortion, our method undistorts
this plane by straightening saccade trajectories using image warping. We show
that this approach improves over the common six-point calibration and is
promising for reducing distortion. As such, it provides a non-intrusive
solution to alleviating accuracy decrease of eye tracker during long-term use.

As first-person cameras in head-mounted displays become increasingly

prevalent, so does the problem of infringing user and bystander privacy. To

address this challenge, we present PrivacEye, a proof-of-concept system that

detects privacysensitive everyday situations and automatically enables and

disables the first-person camera using a mechanical shutter. To close the

shutter, PrivacEye detects sensitive situations from first-person camera videos

using an end-to-end deep-learning model. To open the shutter without visual

input, PrivacEye uses a separate, smaller eye camera to detect changes in

users' eye movements to gauge changes in the "privacy level" of the current

situation. We evaluate PrivacEye on a dataset of first-person videos recorded

in the daily life of 17 participants that they annotated with privacy

sensitivity levels. We discuss the strengths and weaknesses of our

proof-of-concept system based on a quantitative technical evaluation as well as

qualitative insights from semi-structured interviews.

Previous work focused on predicting visual search targets from human

fixations but, in the real world, a specific target is often not known, e.g.

when searching for a present for a friend. In this work we instead study the

problem of predicting the mental picture, i.e. only an abstract idea instead of

a specific target. This task is significantly more challenging given that

mental pictures of the same target category can vary widely depending on

personal biases, and given that characteristic target attributes can often not

be verbalised explicitly. We instead propose to use gaze information as

implicit information on users' mental picture and present a novel gaze pooling

layer to seamlessly integrate semantic and localized fixation information into

a deep image representation. We show that we can robustly predict both the

mental picture's category as well as attributes on a novel dataset containing

fixation data of 14 users searching for targets on a subset of the DeepFahion

dataset. Our results have important implications for future search interfaces

and suggest deep gaze pooling as a general-purpose approach for gaze-supported

computer vision systems.

What does human gaze reveal about a users' intents and to which extend can

these intents be inferred or even visualized? Gaze was proposed as an implicit

source of information to predict the target of visual search and, more

recently, to predict the object class and attributes of the search target. In

this work, we go one step further and investigate the feasibility of combining

recent advances in encoding human gaze information using deep convolutional

neural networks with the power of generative image models to visually decode,

i.e. create a visual representation of, the search target. Such visual decoding

is challenging for two reasons: 1) the search target only resides in the user's

mind as a subjective visual pattern, and can most often not even be described

verbally by the person, and 2) it is, as of yet, unclear if gaze fixations

contain sufficient information for this task at all. We show, for the first

time, that visual representations of search targets can indeed be decoded only

from human gaze fixations. We propose to first encode fixations into a semantic

representation and then decode this representation into an image. We evaluate

our method on a recent gaze dataset of 14 participants searching for clothing

in image collages and validate the model's predictions using two human studies.

Our results show that 62% (Chance level = 10%) of the time users were able to

select the categories of the decoded image right. In our second studies we show

the importance of a local gaze encoding for decoding visual search targets of

user

Rotations performed with the index finger and thumb involve some of the most complex motor action among common multi-touch gestures, yet little is known about the factors affecting performance and ergonomics. This note presents results from a study where the angle, direction, diameter, and position of rotations were systematically manipulated. Subjects were asked to perform the rotations as quickly as possible without losing contact with the display, and were allowed to skip rotations that were too uncomfortable. The data show surprising interaction effects among the variables, and help us identify whole categories of rotations that are slow and cumbersome for users.

Common computational methods for automated eye movement detection - i.e. the

task of detecting different types of eye movement in a continuous stream of

gaze data - are limited in that they either involve thresholding on

hand-crafted signal features, require individual detectors each only detecting

a single movement, or require pre-segmented data. We propose a novel approach

for eye movement detection that only involves learning a single detector

end-to-end, i.e. directly from the continuous gaze data stream and

simultaneously for different eye movements without any manual feature crafting

or segmentation. Our method is based on convolutional neural networks (CNN)

that recently demonstrated superior performance in a variety of tasks in

computer vision, signal processing, and machine learning. We further introduce

a novel multi-participant dataset that contains scripted and free-viewing

sequences of ground-truth annotated saccades, fixations, and smooth pursuits.

We show that our CNN-based method outperforms state-of-the-art baselines by a

large margin on this challenging dataset, thereby underlining the significant

potential of this approach for holistic, robust, and accurate eye movement

protocol analysis.

Gaze reflects how humans process visual scenes and is therefore increasingly

used in computer vision systems. Previous works demonstrated the potential of

gaze for object-centric tasks, such as object localization and recognition, but

it remains unclear if gaze can also be beneficial for scene-centric tasks, such

as image captioning. We present a new perspective on gaze-assisted image

captioning by studying the interplay between human gaze and the attention

mechanism of deep neural networks. Using a public large-scale gaze dataset, we

first assess the relationship between state-of-the-art object and scene

recognition models, bottom-up visual saliency, and human gaze. We then propose

a novel split attention model for image captioning. Our model integrates human

gaze information into an attention-based long short-term memory architecture,

and allows the algorithm to allocate attention selectively to both fixated and

non-fixated image regions. Through evaluation on the COCO/SALICON datasets we

show that our method improves image captioning performance and that gaze can

complement machine attention for semantic scene understanding tasks.

The eyes are a rich channel for non-verbal communication in

our daily interactions. We propose social gaze interaction as a game

mechanic to enhance user interactions with virtual characters. We

develop a game from the ground-up in which characters are esigned to be

reactive to the player’s gaze in social ways, such as etting annoyed

when the player seems distracted or changing their dialogue depending on

the player’s apparent focus of ttention. Results from a qualitative user

study provide insights bout how social gaze interaction is intuitive for

users, elicits deep feelings of immersion, and highlight the players’

self-consciousness of their own eye movements through their strong

reactions to the characters

Head-mounted eye tracking has significant potential for

mobile gaze-based interaction with ambient displays but current

interfaces lack information about the tracker\'s gaze estimation error.

Consequently, current interfaces do not exploit the full potential of

gaze input as the inherent estimation error can not be dealt with. The

error depends on the physical properties of the display and constantly

varies with changes in position and distance of the user to the display.

In this work we present a computational model of gaze estimation error

for head-mounted eye trackers. Our model covers the full processing

pipeline for mobile gaze estimation, namely mapping of pupil positions

to scene camera coordinates, marker-based display detection, and display

mapping. We build the model based on a series of controlled measurements

of a sample state-of-the-art monocular head-mounted eye tracker. Results

show that our model can predict gaze estimation error with a root mean

squared error of 17.99~px ($1.96^\\circ$).

Mobile gaze-based interaction with multiple displays may

occur from arbitrary positions and orientations. However, maintaining

high gaze estimation accuracy still represents a significant challenge.

To address this, we present GazeProjector, a system that combines

accurate point-of-gaze estimation with natural feature tracking on

displays to determine the mobile eye tracker’s position relative to a

display. The detected eye positions are transformed onto that display

allowing for gaze-based interaction. This allows for seamless gaze

estimation and interaction on (1) multiple displays of arbitrary sizes,

(2) independently of the user’s position and orientation to the display.

In a user study with 12 participants we compared GazeProjector to

existing well- established methods such as visual on-screen markers and

a state-of-the-art motion capture system. Our results show that our

approach is robust to varying head poses, orientations, and distances to

the display, while still providing high gaze estimation accuracy across

multiple displays without re-calibration. The system represents an

important step towards the vision of pervasive gaze-based interfaces.

An increasing number of works explore collaborative human-computer systems in

which human gaze is used to enhance computer vision systems. For object

detection these efforts were so far restricted to late integration approaches

that have inherent limitations, such as increased precision without increase in

recall. We propose an early integration approach in a deformable part model,

which constitutes a joint formulation over gaze and visual data. We show that

our GazeDPM method improves over the state-of-the-art DPM baseline by 4% and a

recent method for gaze-supported object detection by 3% on the public POET

dataset. Our approach additionally provides introspection of the learnt models,

can reveal salient image structures, and allows us to investigate the interplay

between gaze attracting and repelling areas, the importance of view-specific

models, as well as viewers' personal biases in gaze patterns. We finally study

important practical aspects of our approach, such as the impact of using

saliency maps instead of real fixations, the impact of the number of fixations,

as well as robustness to gaze estimation error.

We present labelled pupils in the wild (LPW), a novel dataset of 66

high-quality, high-speed eye region videos for the development and evaluation

of pupil detection algorithms. The videos in our dataset were recorded from 22

participants in everyday locations at about 95 FPS using a state-of-the-art

dark-pupil head-mounted eye tracker. They cover people with different

ethnicities, a diverse set of everyday indoor and outdoor illumination

environments, as well as natural gaze direction distributions. The dataset also

includes participants wearing glasses, contact lenses, as well as make-up. We

benchmark five state-of-the-art pupil detection algorithms on our dataset with

respect to robustness and accuracy. We further study the influence of image

resolution, vision aids, as well as recording location (indoor, outdoor) on

pupil detection performance. Our evaluations provide valuable insights into the

general pupil detection problem and allow us to identify key challenges for

robust pupil detection on head-mounted eye trackers.

Although gaze is an attractive modality for pervasive

interaction, real-world implementation of eye-based interfaces poses

significant challenges. In particular, user calibration is tedious and

time consuming. Pursuits is an innovative interaction technique that

enables truly spontaneous interaction with eye-based interfaces. A user

can simply walk up to the screen and readily interact with moving

targets. Instead of being based on gaze location, Pursuits correlates

eye pursuit movements with objects dynamically moving on the interface.

Automatic annotation of life logging data is challenging. In this

work we present EyeContext, a system to infer high-level contextual

cues from human visual behaviour. We conduct a user study to record

eye movements of four participants over a full day of their daily

life, totalling 42.5 hours of eye movement data. Participants were

asked to self-annotate four non-mutually exclusive cues: social (interacting

with somebody vs. no interaction), cognitive (concentrated work vs.

leisure), physical (physically active vs. not active), and spatial

(inside vs. outside a building). We evaluate a proof-of-concept EyeContext

system that combines encoding of eye movements into strings and a

spectrum string kernel support vector machine (SVM) classifier. Using

person-dependent training, we obtain a top performance of 85.3%

precision (98.0% recall) for recognising social interactions. Our

results demonstrate the large information content available in long-term

human visual behaviour and opens up new venues for research on eye-based

behavioural monitoring and life logging.

Eye-based interaction has commonly been based on estimation of eye

gaze direction, to locate objects for interaction. We introduce Pursuits,

a novel and very different eye tracking method that instead is based

on following the trajectory of eye movement and comparing this with

trajectories of objects in the field of view. Because the eyes naturally

follow the trajectory of moving objects of interest, our method is

able to detect what the user is looking at, by matching eye movement

and object movement. We illustrate Pursuits with three applications

that demonstrate how the method facilitates natural interaction with

moving targets.

Previous work has validated the eyes and mobile input as a viable

approach for pointing at, and selecting out of reach objects. This

work presents Eye Pull, Eye Push, a novel interaction concept for

content transfer between public and personal devices using gaze and

touch. We present three techniques that enable this interaction:

Eye Cut & Paste, Eye Drag & Drop, and Eye Summon & Cast. We outline

and discuss several scenarios in which these techniques can be used.

In a user study we found that participants responded well to the

visual feedback provided by Eye Drag & Drop during object movement.

In contrast, we found that although Eye Summon & Cast significantly

improved performance, participants had difficulty coordinating their

hands and eyes during interaction.

Garments made of smart textiles have an enormous potential for embedding sensors in close proximity to the body in an unobtrusive and comfortable manner. Combined with signal processing and pattern recognition technologies, complex high-level information about human behaviors or situations can be inferred from the sensor data. The goal of this chapter is to introduce the reader to the design of activity-aware systems that use body-worn sensors, such as those that can be made available through smart textiles. We start this chapter by emphasizing recent trends towards ‘}wearable{’ sensing and computing and we present several examples of activity-aware applications. Then we outline the role that smart textiles can play in activity-aware applications, but also the challenges that they pose. We conclude by discussing the design process followed to devise activity-aware systems: the choice of sensors, the available data processing methods, and the evaluation techniques. We discuss recent data processing methods that address the challenges resulting from the use of smart textiles.

Research on human activity recognition has traditionally focused on

discriminating between different activities, i.e. to predict \textquoteleft}{\textquoteleft}which{\textquoteright}{\textquoteright}

activity was performed at a specific point in time. The quality of

executing an activity, the {\textquoteleft}{\textquoteleft}how (well){\textquoteright}{\textquoteright,

has only received little attention so far, even though it potentially

provides useful information for a large variety of applications,

such as sports training. In this work we first define quality of

execution and investigate three aspects that pertain to qualitative

activity recognition: the problem of specifying correct execution,

the automatic and robust detection of execution mistakes, and how

to provide feedback on the quality of execution to the user. We illustrate

our approach on the example problem of qualitatively assessing and

providing feedback on weight lifting exercises. In two user studies

we try out a sensor- and a model-based approach to qualitative activity

recognition. Our results underline the potential of model-based assessment

and the positive impact of real-time user feedback on the quality

of execution.

Eye gaze is a compelling interaction modality but requires a user

calibration before interaction can commence. State of the art procedures

require the user to fixate on a succession of calibration markers,

a task that is often experienced as difficult and tedious. We present

a novel approach, pursuit calibration, that instead uses moving targets

for calibration. Users naturally perform smooth pursuit eye movements

when they follow a moving target, and we use correlation of eye and

target movement to detect the users attention and to sample data

for calibration. Because the method knows when the users is attending

to a target, the calibration can be performed implicitly, which enables

more flexible design of the calibration task. We demonstrate this

in application examples and user studies, and show that pursuit calibration

is tolerant to interruption, can blend naturally with applications,

and is able to calibrate users without their awareness.

We are very happy to present the proceedings of the 4th Augmented

Human International Conference (Augmented Human 2013). Augmented

Human 2013 focuses on augmenting human capabilities through technology

for increased well-being and enjoyable human experience. The conference

is in cooperation with ACM SIGCHI, with its proceedings to be archived

in ACM\textquoteright}s Digital Library. With technological advances,

computing has progressively moved beyond the desktop into new physical

and social contexts. As physical artifacts gain new computational

behaviors, they become reprogrammable, customizable, repurposable,

and interoperable in rich ecologies and diverse contexts. They also

become more complex, and require intense design effort in order to

be functional, usable, and enjoyable. Designing such systems requires

interdisciplinary thinking. Their creation must not only encompass

software, electronics, and mechanics, but also the system{\textquoterights

physical form and behavior, its social and physical milieu, and beyond.

Hearing instruments (HIs) have emerged as true pervasive computers

as they continuously adapt the hearing program to the user\textquoterights

context. However, current HIs are not able to distinguish different

hearing needs in the same acoustic environment. In this work, we

explore how information derived from body and eye movements can be

used to improve the recognition of such hearing needs. We conduct

an experiment to provoke an acoustic environment in which different

hearing needs arise: active conversation and working while colleagues

are having a conversation in a noisy office environment. We record

body movements on nine body locations, eye movements using electrooculography

(EOG), and sound using commercial HIs for eleven participants. Using

a support vector machine (SVM) classifier and person-independent

training we improve the accuracy of 77% based on sound to an accuracy

of 92% using body movements. With a view to a future implementation

into a HI we then perform a detailed analysis of the sensors attached

to the head. We achieve the best accuracy of 86% using eye movements

compared to 84% for head movements. Our work demonstrates the potential

of additional sensor modalities for future HIs and motivates to investigate

the wider applicability of this approach on further hearing situations

and needs.