PhD Thesis Publications of the International Max Planck Research School for Computer Science

Deep learning is increasingly relevant in our daily lives, as it simplifies tedious tasks and enhances quality of life across various domains such as entertainment, learning, automatic assistance, and autonomous driving. However, the demand for more data to train models for emerging tasks is increasing dramatically. Deep learning models heavily depend on the quality and quantity of data, necessitating high-quality labeled datasets. Yet, each task requires different types of annotations for training and evaluation, posing challenges in obtaining comprehensive supervision. The acquisition of annotations is not only resource-intensive in terms of time and cost but also introduces biases, such as granularity in classification, where distinctions like specific breeds versus generic categories may arise. Furthermore, the dynamic nature of the world causes the challenge that previously annotated data becomes potentially irrelevant, and new categories and rare occurrences continually emerge, making it impossible to label every aspect of the world.
Therefore, this thesis aims to explore various supervision scenarios to mitigate the need for full supervision and reduce data acquisition costs. Specifically, we investigate learning without labels, referred to as self-supervised and unsupervised methods, to better understand video and image representations. To learn from data without labels, we leverage injected priors such as motion speed, direction, action order in videos, or semantic information granularity to obtain powerful data representations. Further, we study scenarios involving reduced supervision levels. To reduce annotation costs, first, we propose to omit precise annotations for one modality in multimodal learning, namely in text-video and image-video settings, and transfer available knowledge to large copora of video data. Second, we study semi-supervised learning scenarios, where only a subset of annotated data alongside unlabeled data is available, and propose to revisit regularization constraints and improve generalization to unlabeled data. Additionally, we address scenarios where parts of available data is inherently limited due to privacy and security reasons or naturally rare events, which not only restrict annotations but also limit the overall data volume. For these scenarios, we propose methods that carefully balance between previously obtained knowledge and incoming limited data by introducing a calibration method or combining a space reservation technique with orthogonality constraints. Finally, we explore multimodal and unimodal open-world scenarios where the model is asked to generalize beyond the given set of object or action classes. Specifically, we propose a new challenging setting on multimodal egocentric videos and propose an adaptation method for vision-language models to generalize on egocentric domain. Moreover, we study unimodal image recognition in an open-set setting and propose to disentangle open-set detection and image classification tasks that effectively improve generalization in different settings.
In summary, this thesis investigates challenges arising when full supervision for training models is not available. We develop methods to understand learning dynamics and the role of biases in data, while also proposing novel setups to advance training with less supervision.

Knowledge extraction from text is a key task in natural language processing, which involves many sub-tasks, such as taxonomy induction, named entity recognition and typing, relation extraction, knowledge canonicalization and so on. By constructing structured knowledge from natural language text, knowledge extraction becomes a key asset for search engines, question answering and other downstream applications. However, current knowledge extraction methods mostly focus on prominent real-world entities with Wikipedia and mainstream news articles as sources. The constructed knowledge bases, therefore, lack information about long-tail domains, with fiction and fantasy as archetypes. Fiction and fantasy are core parts of our human culture, spanning from literature to movies, TV series, comics and video games. With thousands of fictional universes which have been created, knowledge from fictional domains are subject of search-engine queries - by fans as well as cultural analysts. Unlike the real-world domain, knowledge extraction on such specific domains like fiction and fantasy has to tackle several key challenges: - Training data: Sources for fictional domains mostly come from books and fan-built content, which is sparse and noisy, and contains difficult structures of texts, such as dialogues and quotes. Training data for key tasks such as taxonomy induction, named entity typing or relation extraction are also not available. - Domain characteristics and diversity: Fictional universes can be highly sophisticated, containing entities, social structures and sometimes languages that are completely different from the real world. State-of-the-art methods for knowledge extraction make assumptions on entity-class, subclass and entity-entity relations that are often invalid for fictional domains. With different genres of fictional domains, another requirement is to transfer models across domains. - Long fictional texts: While state-of-the-art models have limitations on the input sequence length, it is essential to develop methods that are able to deal with very long texts (e.g. entire books), to capture multiple contexts and leverage widely spread cues. This dissertation addresses the above challenges, by developing new methodologies that advance the state of the art on knowledge extraction in fictional domains. - The first contribution is a method, called TiFi, for constructing type systems (taxonomy induction) for fictional domains. By tapping noisy fan-built content from online communities such as Wikia, TiFi induces taxonomies through three main steps: category cleaning, edge cleaning and top-level construction. Exploiting a variety of features from the original input, TiFi is able to construct taxonomies for a diverse range of fictional domains with high precision. - The second contribution is a comprehensive approach, called ENTYFI, for named entity recognition and typing in long fictional texts. Built on 205 automatically induced high-quality type systems for popular fictional domains, ENTYFI exploits the overlap and reuse of these fictional domains on unseen texts. By combining different typing modules with a consolidation stage, ENTYFI is able to do fine-grained entity typing in long fictional texts with high precision and recall. - The third contribution is an end-to-end system, called KnowFi, for extracting relations between entities in very long texts such as entire books. KnowFi leverages background knowledge from 142 popular fictional domains to identify interesting relations and to collect distant training samples. KnowFi devises a similarity-based ranking technique to reduce false positives in training samples and to select potential text passages that contain seed pairs of entities. By training a hierarchical neural network for all relations, KnowFi is able to infer relations between entity pairs across long fictional texts, and achieves gains over the best prior methods for relation extraction.

In this thesis we explore pattern mining and deep learning. Often seen as orthogonal, we show that these fields complement each other and propose to combine them to gain from each other’s strengths. We, first, show how to efficiently discover succinct and non-redundant sets of patterns that provide insight into data beyond conjunctive statements. We leverage the interpretability of such patterns to unveil how and which information flows through neural networks, as well as what characterizes their decisions. Conversely, we show how to combine continuous optimization with pattern discovery, proposing a neural network that directly encodes discrete patterns, which allows us to apply pattern mining at a scale orders of magnitude larger than previously possible. Large neural networks are, however, exceedingly expensive to train for which ‘lottery tickets’ – small, well-trainable sub-networks in randomly initialized neural networks – offer a remedy. We identify theoretical limitations of strong tickets and overcome them by equipping these tickets with the property of universal approximation. To analyze whether limitations in ticket sparsity are algorithmic or fundamental, we propose a framework to plant and hide lottery tickets. With novel ticket benchmarks we then conclude that the limitation is likely algorithmic, encouraging further developments for which our framework offers means to measure progress.

Social media communities like Reddit and Twitter allow users to express their views on
topics of their interest, and to engage with other users who may share or oppose these views.
This can lead to productive discussions towards a consensus, or to contended debates, where
disagreements frequently arise.
Prior work on such settings has primarily focused on identifying notable instances of antisocial
behavior such as hate-speech and “trolling”, which represent possible threats to the health of
a community. These, however, are exceptionally severe phenomena, and do not encompass
controversies stemming from user debates, differences of opinions, and off-topic content, all
of which can naturally come up in a discussion without going so far as to compromise its
development.
This dissertation proposes a framework for the systematic analysis of social media discussions
that take place in the presence of controversial themes, disagreements, and mixed opinions from
participating users. For this, we develop a feature-based model to describe key elements of a
discussion, such as its salient topics, the level of activity from users, the sentiments it expresses,
and the user feedback it receives.
Initially, we build our feature model to characterize adversarial discussions surrounding
political campaigns on Twitter, with a focus on the factual and sentimental nature of their
topics and the role played by different users involved. We then extend our approach to Reddit
discussions, leveraging community feedback signals to define a new notion of controversy
and to highlight conversational archetypes that arise from frequent and interesting interaction
patterns. We use our feature model to build logistic regression classifiers that can predict future
instances of controversy in Reddit communities centered on politics, world news, sports, and
personal relationships. Finally, our model also provides the basis for a comparison of different
communities in the health domain, where topics and activity vary considerably despite their
shared overall focus. In each of these cases, our framework provides insight into how user
behavior can shape a community’s individual definition of controversy and its overall identity.

As machine learning (ML) is increasingly used for decision making in scenarios that impact humans, there is a growing awareness of its potential for unfairness. A large body of recent work has focused on proposing formal notions of fairness in ML, as well as approaches to mitigate unfairness. However, there is a growing disconnect between the ML fairness literature and the needs to operationalize fairness in practice. This thesis addresses the need for responsible ML by developing new models and methods to address challenges in operationalizing fairness in practice. Specifically, it makes the following contributions. First, we tackle a key assumption in the group fairness literature that sensitive demographic attributes such as race and gender are known upfront, and can be readily used in model training to mitigate unfairness. In practice, factors like privacy and regulation often prohibit ML models from collecting or using protected attributes in decision making. To address this challenge we introduce the novel notion of computationally-identifiable errors and propose Adversarially Reweighted Learning (ARL), an optimization method that seeks to improve the worst-case performance over unobserved groups, without requiring access to the protected attributes in the dataset. Second, we argue that while group fairness notions are a desirable fairness criterion, they are fundamentally limited as they reduce fairness to an average statistic over pre-identified protected groups. In practice, automated decisions are made at an individual level, and can adversely impact individual people irrespective of the group statistic. We advance the paradigm of individual fairness by proposing iFair (individually fair representations), an optimization approach for learning a low dimensional latent representation of the data with two goals: to encode the data as well as possible, while removing any information about protected attributes in the transformed representation. Third, we advance the individual fairness paradigm, which requires that similar individuals receive similar outcomes. However, similarity metrics computed over observed feature space can be brittle, and inherently limited in their ability to accurately capture similarity between individuals. To address this, we introduce a novel notion of fairness graphs, wherein pairs of individuals can be identified as deemed similar with respect to the ML objective. We cast the problem of individual fairness into graph embedding, and propose PFR (pairwise fair representations), a method to learn a unified pairwise fair representation of the data. Fourth, we tackle the challenge that production data after model deployment is constantly evolving. As a consequence, in spite of the best efforts in training a fair model, ML systems can be prone to failure risks due to a variety of unforeseen reasons. To ensure responsible model deployment, potential failure risks need to be predicted, and mitigation actions need to be devised, for example, deferring to a human expert when uncertain or collecting additional data to address model’s blind-spots. We propose Risk Advisor, a model-agnostic meta-learner to predict potential failure risks and to give guidance on the sources of uncertainty inducing the risks, by leveraging information theoretic notions of aleatoric and epistemic uncertainty. This dissertation brings ML fairness closer to real-world applications by developing methods that address key practical challenges. Extensive experiments on a variety of real-world and synthetic datasets show that our proposed methods are viable in practice.

Deep learning is becoming increasingly relevant for many high-stakes applications such as autonomous driving or medical diagnosis where wrong decisions can have massive impact on human lives. Unfortunately, deep neural networks are typically assessed solely based on generalization, e.g., accuracy on a fixed test set. However, this is clearly insufficient for safe deployment as potential malicious actors and distribution shifts or the effects of quantization and unreliable hardware are disregarded. Thus, recent work additionally evaluates performance on potentially manipulated or corrupted inputs as well as after quantization and deployment on specialized hardware. In such settings, it is also important to obtain reasonable estimates of the model's confidence alongside its predictions. This thesis studies robustness and uncertainty estimation in deep learning along three main directions: First, we consider so-called adversarial examples, slightly perturbed inputs causing severe drops in accuracy. Second, we study weight perturbations, focusing particularly on bit errors in quantized weights. This is relevant for deploying models on special-purpose hardware for efficient inference, so-called accelerators. Finally, we address uncertainty estimation to improve robustness and provide meaningful statistical performance guarantees for safe deployment. In detail, we study the existence of adversarial examples with respect to the underlying data manifold. In this context, we also investigate adversarial training which improves robustness by augmenting training with adversarial examples at the cost of reduced accuracy. We show that regular adversarial examples leave the data manifold in an almost orthogonal direction. While we find no inherent trade-off between robustness and accuracy, this contributes to a higher sample complexity as well as severe overfitting of adversarial training. Using a novel measure of flatness in the robust loss landscape with respect to weight changes, we also show that robust overfitting is caused by converging to particularly sharp minima. In fact, we find a clear correlation between flatness and good robust generalization. Further, we study random and adversarial bit errors in quantized weights. In accelerators, random bit errors occur in the memory when reducing voltage with the goal of improving energy-efficiency. Here, we consider a robust quantization scheme, use weight clipping as regularization and perform random bit error training to improve bit error robustness, allowing considerable energy savings without requiring hardware changes. In contrast, adversarial bit errors are maliciously introduced through hardware- or software-based attacks on the memory, with severe consequences on performance. We propose a novel adversarial bit error attack to study this threat and use adversarial bit error training to improve robustness and thereby also the accelerator's security. Finally, we view robustness in the context of uncertainty estimation. By encouraging low-confidence predictions on adversarial examples, our confidence-calibrated adversarial training successfully rejects adversarial, corrupted as well as out-of-distribution examples at test time. Thereby, we are also able to improve the robustness-accuracy trade-off compared to regular adversarial training. However, even robust models do not provide any guarantee for safe deployment. To address this problem, conformal prediction allows the model to predict confidence sets with user-specified guarantee of including the true label. Unfortunately, as conformal prediction is usually applied after training, the model is trained without taking this calibration step into account. To address this limitation, we propose conformal training which allows training conformal predictors end-to-end with the underlying model. This not only improves the obtained uncertainty estimates but also enables optimizing application-specific objectives without losing the provided guarantee. Besides our work on robustness or uncertainty, we also address the problem of 3D shape completion of partially observed point clouds. Specifically, we consider an autonomous driving or robotics setting where vehicles are commonly equipped with LiDAR or depth sensors and obtaining a complete 3D representation of the environment is crucial. However, ground truth shapes that are essential for applying deep learning techniques are extremely difficult to obtain. Thus, we propose a weakly-supervised approach that can be trained on the incomplete point clouds while offering efficient inference. In summary, this thesis contributes to our understanding of robustness against both input and weight perturbations. To this end, we also develop methods to improve robustness alongside uncertainty estimation for safe deployment of deep learning methods in high-stakes applications. In the particular context of autonomous driving, we also address 3D shape completion of sparse point clouds.

Personal knowledge is a versatile resource that is valuable for a wide range of downstream applications. Background facts about users can allow chatbot assistants to produce more topical and empathic replies. In the context of recommendation and retrieval models, personal facts can be used to customize the ranking results for individual users. A Personal Knowledge Base, populated with personal facts, such as demographic information, interests and interpersonal relationships, is a unique endpoint for storing and querying personal knowledge. Such knowledge bases are easily interpretable and can provide users with full control over their own personal knowledge, including revising stored facts and managing access by downstream services for personalization purposes. To alleviate users from extensive manual effort to build such personal knowledge base, we can leverage automated extraction methods applied to the textual content of the users, such as dialogue transcripts or social media posts. Mainstream extraction methods specialize on well-structured data, such as biographical texts or encyclopedic articles, which are rare for most people. In turn, conversational data is abundant but challenging to process and requires specialized methods for extraction of personal facts. In this dissertation we address the acquisition of personal knowledge from conversational data. We propose several novel deep learning models for inferring speakers’ personal attributes: • Demographic attributes, age, gender, profession and family status, are inferred by HAMs - hierarchical neural classifiers with attention mechanism. Trained HAMs can be transferred between different types of conversational data and provide interpretable predictions. • Long-tailed personal attributes, hobby and profession, are predicted with CHARM - a zero-shot learning model, overcoming the lack of labeled training samples for rare attribute values. By linking conversational utterances to external sources, CHARM is able to predict attribute values which it never saw during training. • Interpersonal relationships are inferred with PRIDE - a hierarchical transformer-based model. To accurately predict fine-grained relationships, PRIDE leverages personal traits of the speakers and the style of conversational utterances. Experiments with various conversational texts, including Reddit discussions and movie scripts, demonstrate the viability of our methods and their superior performance compared to state-of-the-art baselines.

We study distributed systems, with a particular focus on graph problems and fault
tolerance.
Fault-tolerance in a microprocessor or even System-on-Chip can be improved by using
a fault-tolerant pulse propagation design. The existing design TRIX achieves this goal
by being a distributed system consisting of very simple nodes. We show that even in
the typical mode of operation without faults, TRIX performs significantly better than a
regular wire or clock tree: Statistical evaluation of our simulated experiments show that
we achieve a skew with standard deviation of O(log log H), where H is the height of the
TRIX grid.
The distance-r generalization of classic graph problems can give us insights on how
distance affects hardness of a problem. For the distance-r dominating set problem, we
present both an algorithmic upper and unconditional lower bound for any graph class
with certain high-girth and sparseness criteria. In particular, our algorithm achieves a
O(r · f(r))-approximation in time O(r), where f is the expansion function, which correlates
with density. For constant r, this implies a constant approximation factor, in constant
time. We also show that no algorithm can achieve a (2r + 1 − δ)-approximation for any
δ > 0 in time O(r), not even on the class of cycles of girth at least 5r. Furthermore, we
extend the algorithm to related graph cover problems and even to a different execution
model.
Furthermore, we investigate the problem of packet forwarding, which addresses the
question of how and when best to forward packets in a distributed system. These packets
are injected by an adversary. We build on the existing algorithm OED to handle more
than a single destination. In particular, we show that buffers of size O(log n) are sufficient
for this algorithm, in contrast to O(n) for the naive approach.

Knowledge Graphs (KGs) have applications in many domains such as Finance, Manufacturing, and Healthcare. While recent efforts have created large KGs, their content is far from complete and sometimes includes invalid statements. Therefore, it is crucial to refine the constructed KGs to enhance their coverage and accuracy via KG completion and KG validation. It is also vital to provide human-comprehensible explanations for such refinements, so that humans have trust in the KG quality. Enabling KG exploration, by search and browsing, is also essential for users to understand the KG value and limitations towards down-stream applications. However, the large size of KGs makes KG exploration very challenging. While the type taxonomy of KGs is a useful asset along these lines, it remains insufficient for deep exploration. In this dissertation we tackle the aforementioned challenges of KG refinement and KG exploration by combining logical reasoning over the KG with other techniques such as KG embedding models and text mining. Through such combination, we introduce methods that provide human-understandable output. Concretely, we introduce methods to tackle KG incompleteness by learning exception-aware rules over the existing KG. Learned rules are then used in inferring missing links in the KG accurately. Furthermore, we propose a framework for constructing human-comprehensible explanations for candidate facts from both KG and text. Extracted explanations are used to insure the validity of KG facts. Finally, to facilitate KG exploration, we introduce a method that combines KG embeddings with rule mining to compute informative entity clusters with explanations.

Our increasing reliance on complex algorithms for recommendations calls for models and methods for explainable, scrutable, and trustworthy AI. While explainability is required for understanding the relationships between model inputs and outputs, a scrutable system allows us to modify its behavior as desired. These properties help bridge the gap between our expectations and the algorithm’s behavior and accordingly boost our trust in AI. Aiming to cope with information overload, recommender systems play a crucial role in filtering content (such as products, news, songs, and movies) and shaping a personalized experience for their users. Consequently, there has been a growing demand from the information consumers to receive proper explanations for their personalized recommendations. These explanations aim at helping users understand why certain items are recommended to them and how their previous inputs to the system relate to the generation of such recommendations. Besides, in the event of receiving undesirable content, explanations could possibly contain valuable information as to how the system’s behavior can be modified accordingly. In this thesis, we present our contributions towards explainability and scrutability of recommender systems: • We introduce a user-centric framework, FAIRY, for discovering and ranking post-hoc explanations for the social feeds generated by black-box platforms. These explanations reveal relationships between users’ profiles and their feed items and are extracted from the local interaction graphs of users. FAIRY employs a learning-to-rank (LTR) method to score candidate explanations based on their relevance and surprisal. • We propose a method, PRINCE, to facilitate provider-side explainability in graph-based recommender systems that use personalized PageRank at their core. PRINCE explanations are comprehensible for users, because they present subsets of the user’s prior actions responsible for the received recommendations. PRINCE operates in a counterfactual setup and builds on a polynomial-time algorithm for finding the smallest counterfactual explanations. • We propose a human-in-the-loop framework, ELIXIR, for enhancing scrutability and subsequently the recommendation models by leveraging user feedback on explanations. ELIXIR enables recommender systems to collect user feedback on pairs of recommendations and explanations. The feedback is incorporated into the model by imposing a soft constraint for learning user-specific item representations. We evaluate all proposed models and methods with real user studies and demonstrate their benefits at achieving explainability and scrutability in recommender systems.

Most of the images one finds in the media, such as on the Internet or in textbooks and magazines, contain humans as the main point of attention. Thus, there is an inherent necessity for industry, society, and private persons to be able to thoroughly analyze and synthesize the human-related content in these images. One aspect of this analysis and subject of this thesis is to infer the 3D pose and surface deformation, using only visual information, which is also known as human performance capture. Human performance capture enables the tracking of virtual characters from real-world observations, and this is key for visual effects, games, VR, and AR, to name just a few application areas. However, traditional capture methods usually rely on expensive multi-view (marker-based) systems that are prohibitively expensive for the vast majority of people, or they use depth sensors, which are still not as common as single color cameras. Recently, some approaches have attempted to solve the task by assuming only a single RGB image is given. Nonetheless, they can either not track the dense deforming geometry of the human, such as the clothing layers, or they are far from real time, which is indispensable for many applications. To overcome these shortcomings, this thesis proposes two monocular human performance capture methods, which for the first time allow the real-time capture of the dense deforming geometry as well as an unseen 3D accuracy for pose and surface deformations. At the technical core, this work introduces novel GPU-based and data-parallel optimization strategies in conjunction with other algorithmic design choices that are all geared towards real-time performance at high accuracy. Moreover, this thesis presents a new weakly supervised multiview training strategy combined with a fully differentiable character representation that shows superior 3D accuracy. However, there is more to human-related Computer Vision than only the analysis of people in images. It is equally important to synthesize new images of humans in unseen poses and also from camera viewpoints that have not been observed in the real world. Such tools are essential for the movie industry because they, for example, allow the synthesis of photo-realistic virtual worlds with real-looking humans or of contents that are too dangerous for actors to perform on set. But also video conferencing and telepresence applications can benefit from photo-real 3D characters, as they can enhance the immersive experience of these applications. Here, the traditional Computer Graphics pipeline for rendering photo-realistic images involves many tedious and time-consuming steps that require expert knowledge and are far from real time. Traditional rendering involves character rigging and skinning, the modeling of the surface appearance properties, and physically based ray tracing. Recent learning-based methods attempt to simplify the traditional rendering pipeline and instead learn the rendering function from data resulting in methods that are easier accessible to non-experts. However, most of them model the synthesis task entirely in image space such that 3D consistency cannot be achieved, and/or they fail to model motion- and view-dependent appearance effects. To this end, this thesis presents a method and ongoing work on character synthesis, which allow the synthesis of controllable photoreal characters that achieve motion- and view-dependent appearance effects as well as 3D consistency and which run in real time. This is technically achieved by a novel coarse-to-fine geometric character representation for efficient synthesis, which can be solely supervised on multi-view imagery. Furthermore, this work shows how such a geometric representation can be combined with an implicit surface representation to boost synthesis and geometric quality.

Graph data nowadays easily become so large that it is infeasible to study the underlying structures manually. Thus, computational methods are needed to uncover large-scale structural information. In this thesis, we present methods to understand and summarise large networks.
We propose the hyperbolic community model to describe groups of more densely connected nodes within networks using very intuitive parameters. The model accounts for a
frequent connectivity pattern in real data: a few community members are highly interconnected; most members mainly have ties to this core. Our model fits real data much better than previously-proposed models. Our corresponding random graph generator, HyGen, creates graphs with realistic intra-community structure.
Using the hyperbolic model, we conduct a large-scale study of the temporal evolution of communities on online question–answer sites. We observe that the user activity within a community is constant with respect to its size throughout its lifetime, and a small group of users is responsible for the majority of the social interactions.
We propose an approach for Boolean tensor clustering. This special tensor factorisation is restricted to binary data and assumes that one of the tensor directions has only non-overlapping factors. These assumptions – valid for many real-world data, in particular time-evolving networks – enable the use of bitwise operators and lift much of the computational complexity from the task.

Text and images are the two most common data modalities found on the Internet. Understanding the synergy between text and images, that is, seamlessly analyzing information from these modalities may be trivial for humans, but is challenging for software systems. In this dissertation we study problems where deciphering text-image synergy is crucial for finding solutions. We propose methods and ideas that establish semantic connections between text and images in multimodal contents, and empirically show their effectiveness in four interconnected problems: Image Retrieval, Image Tag Refinement, Image-Text Alignment, and Image Captioning. Our promising results and observations open up interesting scopes for future research involving text-image data understanding.Text and images are the two most common data modalities found on the Internet. Understanding the synergy between text and images, that is, seamlessly analyzing information from these modalities may be trivial for humans, but is challenging for software systems. In this dissertation we study problems where deciphering text-image synergy is crucial for finding solutions. We propose methods and ideas that establish semantic connections between text and images in multimodal contents, and empirically show their effectiveness in four interconnected problems: Image Retrieval, Image Tag Refinement, Image-Text Alignment, and Image Captioning. Our promising results and observations open up interesting scopes for future research involving text-image data understanding.

Abstract
Humans are at the centre of a significant amount of research in computer vision.
Endowing machines with the ability to perceive people from visual data is an immense
scientific challenge with a high degree of direct practical relevance. Success in automatic
perception can be measured at different levels of abstraction, and this will depend on
which intelligent behaviour we are trying to replicate: the ability to localise persons in
an image or in the environment, understanding how persons are moving at the skeleton
and at the surface level, interpreting their interactions with the environment including
with other people, and perhaps even anticipating future actions. In this thesis we tackle
different sub-problems of the broad research area referred to as "looking at people",
aiming to perceive humans in images at different levels of granularity.
We start with bounding box-level pedestrian detection: We present a retrospective
analysis of methods published in the decade preceding our work, identifying various
strands of research that have advanced the state of the art. With quantitative exper-
iments, we demonstrate the critical role of developing better feature representations
and having the right training distribution. We then contribute two methods based
on the insights derived from our analysis: one that combines the strongest aspects of
past detectors and another that focuses purely on learning representations. The latter
method outperforms more complicated approaches, especially those based on hand-
crafted features. We conclude our work on pedestrian detection with a forward-looking
analysis that maps out potential avenues for future research.
We then turn to pixel-level methods: Perceiving humans requires us to both separate
them precisely from the background and identify their surroundings. To this end, we
introduce Cityscapes, a large-scale dataset for street scene understanding. This has since
established itself as a go-to benchmark for segmentation and detection. We additionally
develop methods that relax the requirement for expensive pixel-level annotations, focusing
on the task of boundary detection, i.e. identifying the outlines of relevant objects and
surfaces. Next, we make the jump from pixels to 3D surfaces, from localising and
labelling to fine-grained spatial understanding. We contribute a method for recovering
3D human shape and pose, which marries the advantages of learning-based and model-
based approaches.
We conclude the thesis with a detailed discussion of benchmarking practices in
computer vision. Among other things, we argue that the design of future datasets
should be driven by the general goal of combinatorial robustness besides task-specific
considerations.

I develop a formal framework for propositional satifisfiability with the conflict-driven clause learning (CDCL) procedure using the Isabelle/HOL proof assistant. The framework offers a convenient way to prove metatheorems and experiment with variants, including the Davis-Putnam-Logemann-Loveland procedure. The most noteworthy aspects of my work are the inclusion of rules for forget and restart and the refinement approach. I use the formalization to develop three extensions: First, an incremental solving extension of CDCL. Second, I verify an optimizing CDCL (OCDCL): Given a cost function on literals, OCDCL derives an optimal model with minimum cost. Finally, I work on model covering. Thanks to the CDCL framework I can reuse, these extensions are easier to develop. Through a chain of refinements, I connect the abstract CDCL calculus first to a more concrete calculus, then to a SAT solver expressed in a simple functional programming language, and finally to a SAT solver in an imperative language, with total correctness guarantees. The imperative version relies on the two-watched-literal data structure and other optimizations found in modern solvers. I used the Isabelle Refinement Framework to automate the most tedious refinement steps. After that, I extend this work with further optimizations like blocking literals and the use of machine words as long as possible, before switching to unbounded integers to keep completeness.

Modern deep learning has enabled amazing developments of computer vision in recent years (Hinton and Salakhutdinov, 2006; Krizhevsky et al., 2012). As a fundamental task, semantic segmentation aims to predict class labels for each pixel of images, which empowers machines perception of the visual world. In spite of recent successes of fully convolutional networks (Long etal., 2015), several challenges remain to be addressed. In this thesis, we focus on this topic, under different kinds of input formats and various types of scenes. Specifically, our study contains two aspects: (1) Data-driven neural modules for improved performance. (2) Leverage of datasets w.r.t.training systems with higher performances and better data privacy guarantees. In the first part of this thesis, we improve semantic segmentation by designing new modules which are compatible with existing architectures. First, we develop a spatio-temporal data-driven pooling, which brings additional information of data (i.e. superpixels) into neural networks, benefiting the training of neural networks as well as the inference on novel data. We investigate our approach in RGB-D videos for segmenting indoor scenes, where depth provides complementary cues to colors and our model performs particularly well. Second, we design learnable dilated convolutions, which are the extension of standard dilated convolutions, whose dilation factors (Yu and Koltun, 2016) need to be carefully determined by hand to obtain decent performance. We present a method to learn dilation factors together with filter weights of convolutions to avoid a complicated search of dilation factors. We explore extensive studies on challenging street scenes, across various baselines with different complexity as well as several datasets at varying image resolutions. In the second part, we investigate how to utilize expensive training data. First, we start from the generative modelling and study the network architectures and the learning pipeline for generating multiple examples. We aim to improve the diversity of generated examples but also to preserve the comparable quality of the examples. Second, we develop a generative model for synthesizing features of a network. With a mixture of real images and synthetic features, we are able to train a segmentation model with better generalization capability. Our approach is evaluated on different scene parsing tasks to demonstrate the effectiveness of the proposed method. Finally, we study membership inference on the semantic segmentation task. We propose the first membership inference attack system against black-box semantic segmentation models, that tries to infer if a data pair is used as training data or not. From our observations, information on training data is indeed leaking. To mitigate the leakage, we leverage our synthetic features to perform prediction obfuscations, reducing the posterior distribution gaps between a training and a testing set. Consequently, our study provides not only an approach for detecting illegal use of data, but also the foundations for a safer use of semantic segmentation models.

Answering complex natural language questions with crisp answers is crucial towards satisfying the information needs of advanced users. With the rapid growth of knowledge bases (KBs) such as Yago and Freebase, this goal has become attainable by translating questions into formal queries like SPARQL queries. Such queries can then be evaluated over knowledge bases to retrieve crisp answers. To this end, three research issues arise: (i) how to develop methods that are robust to lexical and syntactic variations in questions and can handle complex questions, (ii) how to design and curate datasets to advance research in question answering, and (iii) how to efficiently identify named entities in questions. In this dissertation, we make the following five contributions in the areas of question answering (QA) and named entity recognition (NER). For issue (i), we make the following contributions: We present QUINT, an approach for answering natural language questions over knowledge bases using automatically learned templates. Templates are an important asset for QA over KBs, simplifying the semantic parsing of input questions and generating formal queries for interpretable answers. QUINT is capable of answering both simple and compositional questions. We introduce NEQA, a framework for continuous learning for QA over KBs. NEQA starts with a small seed of training examples in the form of question-answer pairs, and improves its performance over time. NEQA combines both syntax, through template-based answering, and semantics, via a semantic similarity function. %when templates fail to do so. Moreover, it adapts to the language used after deployment by periodically retraining its underlying models. For issues (i) and (ii), we present TEQUILA, a framework for answering complex questions with explicit and implicit temporal conditions over KBs. TEQUILA is built on a rule-based framework that detects and decomposes temporal questions into simpler sub-questions that can be answered by standard KB-QA systems. TEQUILA reconciles the results of sub-questions into final answers. TEQUILA is accompanied with a dataset called TempQuestions, which consists of 1,271 temporal questions with gold-standard answers over Freebase. This collection is derived by judiciously selecting time-related questions from existing QA datasets. For issue (ii), we publish ComQA, a large-scale manually-curated dataset for QA. ComQA contains questions that represent real information needs and exhibit a wide range of difficulties such as the need for temporal reasoning, comparison, and compositionality. ComQA contains paraphrase clusters of semantically-equivalent questions that can be exploited by QA systems. We harness a combination of community question-answering platforms and crowdsourcing to construct the ComQA dataset.
For issue (iii), we introduce a neural network model based on subword units for named entity recognition. The model learns word representations using a combination of characters, bytes and phonemes. While achieving comparable performance with word-level based models, our model has an order-of-magnitude smaller vocabulary size and lower memory requirements, and it handles out-of-vocabulary words.

Following a period of expedited progress in the capabilities of digital systems, the society begins to realize that systems designed to assist people in various tasks can also harm individuals and society. Mediating access to information and explicitly or implicitly ranking people in increasingly many applications, search systems have a substantial potential to contribute to such unwanted outcomes. Since they collect vast amounts of data about both searchers and search subjects, they have the potential to violate the privacy of both of these groups of users. Moreover, in applications where rankings influence people's economic livelihood outside of the platform, such as sharing economy or hiring support websites, search engines have an immense economic power over their users in that they control user exposure in ranked results. This thesis develops new models and methods broadly covering different aspects of privacy and fairness in search systems for both searchers and search subjects. Specifically, it makes the following contributions: (1) We propose a model for computing individually fair rankings where search subjects get exposure proportional to their relevance. The exposure is amortized over time using constrained optimization to overcome searcher attention biases while preserving ranking utility. (2) We propose a model for computing sensitive search exposure where each subject gets to know the sensitive queries that lead to her profile in the top-k search results. The problem of finding exposing queries is technically modeled as reverse nearest neighbor search, followed by a weekly-supervised learning to rank model ordering the queries by privacy-sensitivity. (3) We propose a model for quantifying privacy risks from textual data in online communities. The method builds on a topic model where each topic is annotated by a crowdsourced sensitivity score, and privacy risks are associated with a user's relevance to sensitive topics. We propose relevance measures capturing different dimensions of user interest in a topic and show how they correlate with human risk perceptions. (4) We propose a model for privacy-preserving personalized search where search queries of different users are split and merged into synthetic profiles. The model mediates the privacy-utility trade-off by keeping semantically coherent fragments of search histories within individual profiles, while trying to minimize the similarity of any of the synthetic profiles to the original user profiles. The models are evaluated using information retrieval techniques and user studies over a variety of datasets, ranging from query logs, through social media and community question answering postings, to item listings from sharing economy platforms.

Although all the cells in an organism posses the same genome, the regulatory mechanisms lead to highly specific cell types. Elucidating these regulatory mechanisms is a great challenge in systems biology research. Nonetheless, it is known that a large fraction of our genome is comprised of regulatory elements, the precise mechanisms by which different combinations of regulatory elements are involved in controlling gene expression and cell identity are poorly understood. This thesis describes algorithms and approaches for modeling and analysis of different modes of gene regulation. We present POSTIT a novel algorithm for modeling and inferring transcript isoform regulation from transcriptomics and epigenomics data. POSTIT uses multi-task learning with structured-sparsity inducing regularizer to share the regulatory information between isoforms of a gene, which is shown to lead to accurate isoform expression prediction and inference of regulators. Furthermore, it can use isoform expression level and annotation as informative priors for gene expression prediction. Hence, it constitute a novel accurate approach applicable to gene or transcript isoform centric analysis using expression data. In an application to microRNA (miRNA) target prioritization, we demonstrate that it out-competes classical gene centric methods. Moreover, pinpoints important transcription factors and miRNAs that regulate differentially expressed isoforms in any biological system. Competing endogenous RNA (ceRNA) interactions mediated by miRNAs were postulated as an important cellular regulatory network, in which cross-talk between different transcripts involves competition for joint regulators. We developed a novel statistical method, called SPONGE, for large-scale inference of ceRNA networks. In this framework, we designed an efficient empirical p-value computation approach, by sampling from derived null models, which addresses important confounding factors such as sample size, number of involved regulators and strength of correlation. In an application to a large pan-cancer dataset with 31 cancers we discovered protein-coding and non-coding RNAs that are generic ceRNAs in cancer. Finally, we present an integrative analysis of miRNA and protein-based posttranscriptional regulation. We postulate a competitive regulation of the RNAbinding protein IMP2 with miRNAs binding the same RNAs using expression and RNA binding data. This function of IMP2 is relevant in the contribution to disease in the context of adult cellular metabolism. As a summary, in this thesis we have presented a number of different novel approaches for inference and the integrative analysis of regulatory networks that we believe will find wide applicability in the biological sciences.

The treatment of infections with HIV or HCV is challenging. Thus,
novel drugs and new computational approaches that support the
selection of therapies are required. This work presents methods that
support therapy selection as well as methods that advance novel
antiviral treatments.
geno2pheno[ngs-freq] identifies drug resistance from HIV-1
or HCV samples that were subjected to next-generation sequencing
by interpreting their sequences either via support vector machines
or a rules-based approach. geno2pheno[coreceptor-hiv2] determines the coreceptor that is used for viral cell entry by analyzing a
segment of the HIV-2 surface protein with a support vector machine.
openPrimeR is capable of finding optimal combinations of primers
for multiplex polymerase chain reaction by solving a set cover problem and accessing a new logistic regression model for determining
amplification events arising from polymerase chain reaction.
geno2pheno[ngs-freq] and geno2pheno[coreceptorhiv2] enable the personalization of antiviral treatments and support
clinical decision making. The application of openPrimeR on human
immunoglobulin sequences has resulted in novel primer sets that
improve the isolation of broadly neutralizing antibodies against
HIV-1. The methods that were developed in this work thus constitute
important contributions towards improving the prevention and
treatment of viral infectious diseases.

Epigenetics is the field of biology that investigates heritable factors regulating gene expression without being directly encoded in the genome of an organism. The human genome is densely packed inside a cell's nucleus in the form of chromatin. Certain constituents of chromatin play a vital role as epigenetic factors in the dynamic regulation of gene expression. Epigenetic changes on the chromatin level are thus an integral part of the mechanisms governing the development of the functionally diverse cell types in multicellular species such as human. Studying these mechanisms is not only important to understand the biology of healthy cells, but also necessary to comprehend the epigenetic component in the formation of many complex diseases. Modern wet lab technology enables scientists to probe the epigenome with high throughput and in extensive detail. The fast generation of epigenetic datasets burdens computational researchers with the challenge of rapidly performing elaborate analyses without compromising on the scientific reproducibility of the reported findings. To facilitate reproducible computational research in epigenomics, this thesis proposes a task-oriented metadata model, relying on web technology and supported by database engineering, that aims at consistent and human-readable documentation of standardized computational workflows. The suggested approach features, e.g., computational validation of metadata records, automatic error detection, and progress monitoring of multi-step analyses, and was successfully field-tested as part of a large epigenome research consortium. This work leaves aside theoretical considerations, and intentionally emphasizes the realistic need of providing scientists with tools that assist them in performing reproducible research. Irrespective of the technological progress, the dynamic and cell-type specific nature of the epigenome commonly requires restricting the number of analyzed samples due to resource limitations. The second project of this thesis introduces the software tool SCIDDO, which has been developed for the differential chromatin analysis of cellular samples with potentially limited availability. By combining statistics, algorithmics, and best practices for robust software development, SCIDDO can quickly identify biologically meaningful regions of differential chromatin marking between cell types. We demonstrate SCIDDO's usefulness in an exemplary study in which we identify regions that establish a link between chromatin and gene expression changes. SCIDDO's quantitative approach to differential chromatin analysis is user-customizable, providing the necessary flexibility to adapt SCIDDO to specific research tasks. Given the functional diversity of cell types and the dynamics of the epigenome in response to environmental changes, it is hardly realistic to map the complete epigenome even for a single organism like human or mouse. For non-model organisms, e.g., cow, pig, or dog, epigenome data is particularly scarce. The third project of this thesis investigates to what extent bioinformatics methods can compensate for the comparatively little effort that is invested in charting the epigenome of non-model species. This study implements a large integrative analysis pipeline, including state-of-the-art machine learning, to transfer chromatin data for predictive modeling between 13 species. The evidence presented here indicates that a partial regulatory epigenetic signal is stably retained even over millions of years of evolutionary distance between the considered species. This finding suggests complementary and cost-effective ways for bioinformatics to contribute to comparative epigenome analysis across species boundaries.

Search systems help users locate relevant information in the form of text documents for keyword queries. Using text alone, it is often difficult to satisfy the user's information need. To discern the user's intent behind queries, we turn to semantic annotations (e.g., named entities and temporal expressions) that natural language processing tools can now deliver with great accuracy. This thesis develops methods and an infrastructure that leverage semantic annotations to efficiently and effectively search large document collections. This thesis makes contributions in three areas: indexing, querying, and mining of semantically annotated document collections. First, we describe an indexing infrastructure for semantically annotated document collections. The indexing infrastructure can support knowledge-centric tasks such as information extraction, relationship extraction, question answering, fact spotting and semantic search at scale across millions of documents. Second, we propose methods for exploring large document collections by suggesting semantic aspects for queries. These semantic aspects are generated by considering annotations in the form of temporal expressions, geographic locations, and other named entities. The generated aspects help guide the user to relevant documents without the need to read their contents. Third and finally, we present methods that can generate events, structured tables, and insightful visualizations from semantically annotated document collections.

There is a wealth of schema-free tables on the web. The text accompanying these tables explains and qualifies the numerical quantities given in the tables. Despite this ubiquity of tabular data, there is little research that harnesses this wealth of data by semantically understanding the information that is conveyed rather ambiguously in these tables. This information can be disambiguated only by the help of the accompanying text. In the process of understanding quantity mentions in tables and text, we are faced with the following challenges; First, there is no comprehensive knowledge base for anchoring quantity mentions. Second, tables are created ad-hoc without a standard schema and with ambiguous header names; also table cells usually contain abbreviations. Third, quantities can be written in multiple forms and units of measures. Fourth, the text usually refers to the quantities in tables using aggregation, approximation, and different scales. In this thesis, we target these challenges through the following contributions: - We present the Quantity Knowledge Base (QKB), a knowledge base for representing Quantity mentions. We construct the QKB by importing information from Freebase, Wikipedia, and other online sources. - We propose Equity: a system for automatically canonicalizing header names and cell values onto concepts, classes, entities, and uniquely represented quantities registered in a knowledge base. We devise a probabilistic graphical model that captures coherence dependencies between cells in tables and candidate items in the space of concepts, entities, and quantities. Then, we cast the inference problem into an efficient algorithm based on random walks over weighted graphs. baselines. - We introduce the quantity alignment problem: computing bidirectional links between textual mentions of quantities and the corresponding table cells. We propose BriQ: a system for computing such alignments. BriQ copes with the specific challenges of approximate quantities, aggregated quantities, and calculated quantities. - We design ExQuisiTe: a web application that identifies mentions of quantities in text and tables, aligns quantity mentions in the text with related quantity mentions in tables, and generates salient suggestions for extractive text summarization systems.

We look at some graph problems related to covering, partition, and connectivity. First, we study the problems of covering and partitioning edges with bicliques, especially from the viewpoint of parameterized complexity. For the partition problem, we develop much more efficient algorithms than the ones previously known. In contrast, for the cover problem, our lower bounds show that the known algorithms are probably optimal. Next, we move on to graph coloring, which is probably the most extensively studied partition problem in graphs. Hadwiger’s conjecture is a long-standing open problem related to vertex coloring. We prove the conjecture for a special class of graphs, namely squares of 2-trees, and show that square graphs are important in connection with Hadwiger’s conjecture. Then, we study a coloring problem that has been emerging recently, called rainbow coloring. This problem lies in the intersection of coloring and connectivity. We study different variants of rainbow coloring and present bounds and complexity results on them. Finally, we move on to another parameter related to connectivity called spanning tree congestion (STC). We give tight bounds for STC in general graphs and random graphs. While proving the results on

Matrix factorizations are an important tool in data mining, and they have been used extensively for finding latent patterns in the data. They often allow to separate structure from noise, as well as to considerably reduce the dimensionality of the input matrix. While classical matrix decomposition methods, such as nonnegative matrix factorization (NMF) and singular value decomposition (SVD), proved to be very useful in data analysis, they are limited by the underlying algebraic structure. NMF, in particular, tends to break patterns into smaller bits, often mixing them with each other. This happens because overlapping patterns interfere with each other, making it harder to tell them apart. In this thesis we study matrix factorization over algebraic structures known as dioids, which are characterized by the lack of additive inverse (“negative numbers”) and the idempotency of addition (a + a = a). Using dioids makes it easier to separate overlapping features, and, in particular, it allows to better deal with the above mentioned pattern breaking problem. We consider different types of dioids, that range from continuous (subtropical and tropical algebras) to discrete (Boolean algebra). Among these, the Boolean algebra is perhaps the most well known, and there exist methods that allow one to obtain high quality Boolean matrix factorizations in terms of the reconstruction error. In this work, however, a different objective function is used – the description length of the data, which enables us to obtain compact and highly interpretable results. The tropical and subtropical algebras, on the other hand, are much less known in the data mining field. While they find applications in areas such as job scheduling and discrete event systems, they are virtually unknown in the context of data analysis. We will use them to obtain idempotent nonnegative factorizations that are similar to NMF, but are better at separating the most prominent features of the data.

Synthesizing novel views from image data is a widely investigated topic in both computer graphics and computer vision, and has many applications like stereo or multi-view rendering for virtual reality, light field reconstruction, and image post-processing. While image-based approaches have the advantage of reduced computational load compared to classical model-based rendering, efficiency is still a major concern. This thesis demonstrates how concepts and tools from artificial intelligence can be used to increase the efficiency of image-based view synthesis algorithms. In particular it is shown how machine learning can help to generate point patterns useful for a variety of computer graphics tasks, how path planning can guide image warping, how sparsity-enforcing optimization can lead to significant speedups in interactive distribution effect rendering, and how probabilistic inference can be used to perform real-time 2D-to-3D conversion.

Graph decomposition has always been a very important concept in machine learning and computer vision. Many tasks like image and mesh segmentation, community detection in social networks, as well as object tracking and human pose estimation can be formulated as a graph decomposition problem. The multicut problem in particular is a popular model to optimize for a decomposition of a given graph. Its main advantage is that no prior knowledge about the number of components or their sizes is required. However, it has several limitations, which we address in this thesis: Firstly, the multicut problem allows to specify only cost or reward for putting two direct neighbours into distinct components. This limits the expressibility of the cost function. We introduce special edges into the graph that allow to define cost or reward for putting any two vertices into distinct components, while preserving the original set of feasible solutions. We show that this considerably improves the quality of image and mesh segmentations. Second, multicut is notorious to be NP-hard for general graphs, that limits its applications to small super-pixel graphs. We define and implement two primal feasible heuristics to solve the problem. They do not provide any guarantees on the runtime or quality of solutions, but in practice show good convergence behaviour. We perform an extensive comparison on multiple graphs of different sizes and properties. Third, we extend the multicut framework by introducing node labels, so that we can jointly optimize for graph decomposition and nodes classification by means of exactly the same optimization algorithm, thus eliminating the need to hand-tune optimizers for a particular task. To prove its universality we applied it to diverse computer vision tasks, including human pose estimation, multiple object tracking, and instance-aware semantic segmentation. We show that we can improve the results over the prior art using exactly the same data as in the original works. Finally, we use employ multicuts in two applications: 1) a client-server tool for interactive video segmentation: After the pre-processing of the video a user draws strokes on several frames and a time-coherent segmentation of the entire video is performed on-the-fly. 2) we formulate a method for simultaneous segmentation and tracking of living cells in microscopy data. This task is challenging as cells split and our algorithm accounts for this, creating parental hierarchies. We also present results on multiple model fitting. We find models in data heavily corrupted by noise by finding components defining these models using higher order multicuts. We introduce an interesting extension that allows our optimization to pick better hyperparameters for each discovered model. In summary, this thesis extends the multicut problem in different directions, proposes algorithms for optimization, and applies it to novel data and settings.

With the development of chromosome conformation capture-based techniques, we now know that chromatin is packed in three-dimensional (3D) space inside the cell nucleus. Changes in the 3D chromatin architecture have already been implicated in diseases such as cancer. Thus, a better understanding of this 3D conformation is of interest to help enhance our comprehension of the complex, multipronged regulatory mechanisms of the genome. The work described in this dissertation largely focuses on development and application of interpretable machine learning methods for prediction and analysis of long-range genomic interactions output from chromatin interaction experiments. In the first part, we demonstrate that the genetic sequence information at the ge- nomic loci is predictive of the long-range interactions of a particular locus of interest (LoI). For example, the genetic sequence information at and around enhancers can help predict whether it interacts with a promoter region of interest. This is achieved by building string kernel-based support vector classifiers together with two novel, in- tuitive visualization methods. These models suggest a potential general role of short tandem repeat motifs in the 3D genome organization. But, the insights gained out of these models are still coarse-grained. To this end, we devised a machine learning method, called CoMIK for Conformal Multi-Instance Kernels, capable of providing more fine-grained insights. When comparing sequences of variable length in the su- pervised learning setting, CoMIK can not only identify the features important for classification but also locate them within the sequence. Such precise identification of important segments of the whole sequence can help in gaining de novo insights into any role played by the intervening chromatin towards long-range interactions. Although CoMIK primarily uses only genetic sequence information, it can also si- multaneously utilize other information modalities such as the numerous functional genomics data if available. The second part describes our pipeline, pHDee, for easy manipulation of large amounts of 3D genomics data. We used the pipeline for analyzing HiChIP experimen- tal data for studying the 3D architectural changes in Ewing sarcoma (EWS) which is a rare cancer affecting adolescents. In particular, HiChIP data for two experimen- tal conditions, doxycycline-treated and untreated, and for primary tumor samples is analyzed. We demonstrate that pHDee facilitates processing and easy integration of large amounts of 3D genomics data analysis together with other data-intensive bioinformatics analyses.

Despite being a vast resource of valuable information, the Web has been polluted by the spread of false claims. Increasing hoaxes, fake news, and misleading information on the Web have given rise to many fact-checking websites that manually assess these doubtful claims. However, the rapid speed and large scale of misinformation spread have become the bottleneck for manual verification. This calls for credibility assessment tools that can automate this verification process. Prior works in this domain make strong assumptions about the structure of the claims and the communities where they are made. Most importantly, black-box techniques proposed in prior works lack the ability to explain why a certain statement is deemed credible or not. To address these limitations, this dissertation proposes a general framework for automated credibility assessment that does not make any assumption about the structure or origin of the claims. Specifically, we propose a feature-based model, which automatically retrieves relevant articles about the given claim and assesses its credibility by capturing the mutual interaction between the language style of the relevant articles, their stance towards the claim, and the trustworthiness of the underlying web sources. We further enhance our credibility assessment approach and propose a neural-network-based model. Unlike the feature-based model, this model does not rely on feature engineering and external lexicons. Both our models make their assessments interpretable by extracting explainable evidence from judiciously selected web sources. We utilize our models and develop a Web interface, CredEye, which enables users to automatically assess the credibility of a textual claim and dissect into the assessment by browsing through judiciously and automatically selected evidence snippets. In addition, we study the problem of stance classification and propose a neural-network-based model for predicting the stance of diverse user perspectives regarding the controversial claims. Given a controversial claim and a user comment, our stance classification model predicts whether the user comment is supporting or opposing the claim.

Technologies for motion and performance capture of real actors have enabled the creation of realisticlooking virtual humans through detail and deformation transfer at the cost of extensive manual work and sophisticated in-studio marker-based systems. This thesis pushes the boundaries of performance capture by proposing automatic algorithms for robust 3D skeleton and detailed surface tracking in less constrained multi-view outdoor scenarios. Contributions include new multi-layered human body representations designed for effective model-based time-consistent reconstruction in complex dynamic environments with varying illumination, from a set of vision cameras. We design dense surface refinement approaches to enable smooth silhouette-free model-to-image alignment, as well as coarse-to-fine tracking techniques to enable joint estimation of skeleton motion and finescale surface deformations in complicated scenarios. High-quality results attained on challenging application scenarios confirm the contributions and show great potential for the automatic creation of personalized 3D virtual humans.

Visual search is an important task, and it is part of daily human life. Thus, it has been a long-standing goal in Computer Vision to develop methods aiming at analysing human search intent and preferences. As the target of the search only exists in mind of the person, search intent prediction remains challenging for machine perception. In this thesis, we focus on advancing techniques for search target and preference prediction from implicit human cues. First, we propose a search target inference algorithm from human fixation data recorded during visual search. In contrast to previous work that has focused on individual instances as a search target in a closed world, we propose the first approach to predict the search target in open-world settings by learning the compatibility between observed fixations and potential search targets. Second, we further broaden the scope of search target prediction to categorical classes, such as object categories and attributes. However, state of the art models for categorical recognition, in general, require large amounts of training data, which is prohibitive for gaze data. To address this challenge, we propose a novel Gaze Pooling Layer that integrates gaze information into CNN-based architectures as an attention mechanism – incorporating both spatial and temporal aspects of human gaze behaviour. Third, we go one step further and investigate the feasibility of combining our gaze embedding approach, with the power of generative image models to visually decode, i.e. create a visual representation of, the search target. Forth, for the first time, we studied the effect of body shape on people preferences of outfits. We propose a novel and robust multi-photo approach to estimate the body shapes of each user and build a conditional model of clothing categories given body-shape. We demonstrate that in real-world data, clothing categories and body-shapes are correlated. We show that our approach estimates a realistic looking body shape that captures a user’s weight group and body shape type, even from a single image of a clothed person. However, an accurate depiction of the naked body is considered highly private and therefore, might not be consented by most people. First, we studied the perception of such technology via a user study. Then, in the last part of this thesis, we ask if the automatic extraction of such information can be effectively evaded. In summary, this thesis addresses several different tasks that aims to enable the vision system to analyse human search intent and preferences in real-world scenarios. In particular, the thesis proposes several novel ideas and models in visual search target prediction from human fixation data, for the first time studied the correlation between shape and clothing categories opening a new direction in clothing recommendation systems, and introduces a new topic in privacy and computer vision, aimed at preventing automatic 3D shape extraction from images.

The Internet infrastructure relies on the correct functioning of the basic underlying protocols, which were designed for functionality. Security and privacy have been added post hoc, mostly by applying cryptographic means to different layers of communication. In the absence of accountability, as a fundamental property, the Internet infrastructure does not have a built-in ability to associate an action with the responsible entity, neither to detect or prevent misbehavior. In this thesis, we study accountability from a few different perspectives. First, we study the need of having accountability in anonymous communication networks as a mechanism that provides repudiation for the proxy nodes by tracing back selected outbound traffic in a provable manner. Second, we design a framework that provides a foundation to support the enforcement of the right to be forgotten law in a scalable and automated manner. The framework provides a technical mean for the users to prove their eligibility for content removal from the search results. Third, we analyze the Internet infrastructure determining potential security risks and threats imposed by dependencies among the entities on the Internet. Finally, we evaluate the feasibility of using hop count filtering as a mechanism for mitigating Distributed Reflective Denial-of-Service attacks, and conceptually show that it cannot work to prevent these attacks.

Eye tracking and gaze-based human-computer interfaces have become a practical modality in desktop settings, since remote eye tracking is efficient and affordable. However, remote eye tracking remains constrained to indoor, laboratory-like conditions, in which lighting and user position need to be controlled. Mobile eye tracking has the potential to overcome these limitations and to allow people to move around freely and to use eye tracking on a daily basis during their everyday routine. However, mobile eye tracking currently faces two fundamental challenges that prevent it from being practically usable and that, consequently, have to be addressed before mobile eye tracking can truly be used by everyone: Mobile eye tracking needs to be advanced and made fully functional in unconstrained environments, and it needs to be made socially acceptable. Numerous sensing and analysis methods were initially developed for remote eye tracking and have been successfully applied for decades. Unfortunately, these methods are limited in terms of functionality and correctness, or even unsuitable for application in mobile eye tracking. Therefore, the majority of fundamental definitions, eye tracking methods, and gaze estimation approaches cannot be borrowed from remote eye tracking without adaptation. For example, the definitions of specific eye movements, like classical fixations, need to be extended to mobile settings where natural user and head motion are omnipresent. Corresponding analytical methods need to be adjusted or completely reimplemented based on novel approaches encoding the human gaze behaviour. Apart from these technical challenges, an entirely new, and yet under-explored, topic required for the breakthrough of mobile eye tracking as everyday technology is the overcoming of social obstacles. A first crucial key issue to defuse social objections is the building of acceptance towards mobile eye tracking. Hence, it is essential to replace the bulky appearance of current head-mounted eye trackers with an unobtrusive, appealing, and trendy design. The second high-priority theme of increasing importance for everyone is privacy and its protection, given that research and industry have not focused on or taken care of this problem at all. To establish true confidence, future devices have to find a fine balance between protecting users’ and bystanders’ privacy and attracting and convincing users of their necessity, utility, and potential with useful and beneficial features. The solution of technical challenges and social obstacles is the prerequisite for the development of a variety of novel and exciting applications in order to establish mobile eye tracking as a new paradigm, which ease our everyday life. This thesis addresses core technical challenges of mobile eye tracking that currently prevent it from being widely adopted. Specifically, this thesis proves that 3D data used for the calibration of mobile eye trackers improves gaze estimation and significantly reduces the parallax error. Further, it presents the first effective fixation detection method for head-mounted devices that is robust against the prevalence of user and gaze target motion. In order to achieve social acceptability, this thesis proposes an innovative and unobtrusive design for future mobile eye tracking devices and builds the first prototype with fully frame-embedded eye cameras combined with a calibration-free deep-trained appearance-based gaze estimation approach. To protect users’ and bystanders’ privacy in the presence of head-mounted eye trackers, this thesis presents another first-of-its-kind prototype. It is able to identify privacy-sensitive situations to automatically enable and disable the eye tracker’s first-person camera by means of a mechanical shutter, leveraging the combination of deep scene and eye movement features. Nevertheless, solving technical challenges and social obstacles alone is not sufficient to make mobile eye tracking attractive for the masses. The key to success is the development of convincingly useful, innovative, and essential applications. To extend the protection of users’ privacy on the software side as well, this thesis presents the first privacy-aware VR gaze interface using differential privacy. This method adds noise to recorded eye tracking data so that privacy-sensitive information like a user’s gender or identity is protected without impeding the utility of the data itself. In addition, the first large-scale online survey is conducted to understand users’ concerns with eye tracking. To develop and evaluate novel applications, this thesis presents the first publicly available long-term eye tracking datasets. They are used to show the unsupervised detection of users’ activities from eye movements alone using novel and efficient video-based encoding approaches as well as to propose the first proof-of-concept method to forecast users’ attentive behaviour during everyday mobile interactions from phone-integrated and body-worn sensors. This opens up possibilities for the development of a variety of novel and exciting applications. With more advanced features, accompanied by technological progress and sensor miniaturisation, eye tracking is increasingly integrated into conventional glasses as well as virtual and augmented reality (VR/AR) head-mounted displays, becoming an integral component of mobile interfaces. This thesis paves the way for the development of socially acceptable, privacy-aware, but highly functional mobile eye tracking devices and novel applications, so that mobile eye tracking can develop its full potential to become an everyday technology for everyone.

First-order logic is one of the most prominent formalisms in computer science and mathematics. Since there is no algorithm capable of solving its satisfiability problem, first-order logic is said to be undecidable. The classical decision problem is the quest for a delineation between the decidable and the undecidable parts. The results presented in this thesis shed more light on the boundary and open new perspectives on the landscape of known decidable fragments. In the first part we focus on the new concept of separateness of variables and explore its applicability to the classical decision problem and beyond. Two disjoint sets of first-order variables are separated in a given formula if none of its atoms contains variables from both sets. This notion facilitates the definition of decidable extensions of many well-known decidable first-order fragments. We demonstrate this for several prefix fragments, several guarded fragments, the two-variable fragment, and for the fluted fragment. Although the extensions exhibit the same expressive power as the respective originals, certain logical properties can be expressed much more succinctly. In two cases the succinctness gap cannot be bounded using elementary functions. This fact already hints at computationally hard satisfiability problems. Indeed, we derive non-elementary lower bounds for the separated fragment, an extension of the Bernays-Schönfinkel-Ramsey fragment (E*A*-prefix sentences). On the semantic level, separateness of quantified variables may lead to weaker dependences than we encounter in general. We investigate this property in the context of model-checking games. The focus of the second part of the thesis is on linear arithmetic with uninterpreted predicates. Two novel decidable fragments are presented, both based on the Bernays-Schönfinkel-Ramsey fragment. On the negative side, we identify several small fragments of the language for which satisfiability is undecidable.

Proteins participate in most of the important processes in cells, and their ability to perform their function ultimately depends on their three-dimensional structure. They usually act in these processes through interactions with other molecules. Because of the importance of their role, proteins are also the common target for small molecule drugs that inhibit their activity, which may include targeting protein interactions. Understanding protein interactions and how they are affected by mutations is thus crucial for combating drug resistance and aiding drug design. This dissertation combines bioinformatics studies of protein interactions at both primary sequence and structural level. We analyse protein-protein interactions through linear motifs, as well as protein-small molecule interactions, and study how mutations affect them. This is done in the context of two systems. In the first study of drug resistance mutations in the protease of the human immunodeficiency virus type 1, we successfully apply molecular dynamics simulations to estimate the effects of known resistance-associated mutations on the free binding energy, also revealing molecular mechanisms of resistance. In the second study, we analyse consensus profiles of linear motifs that mediate the recognition by the mitogen-activated protein kinases of their target proteins. We thus gain insights into the cellular processes these proteins are involved in.

Over the years, computer systems and applications have grown significantly complex while handling a plethora of private and sensitive user information. The complexity of these applications is often assisted by a set of (un)intentional bugs with both malicious and non-malicious intent leading to information leaks. Information flow control has been studied extensively as an approach to mitigate such information leaks. The technique works by enforcing the security property of non-interference using a specified set of security policies. A vast majority of existing work in this area is based on static analyses. However, some of the applications, especially on the Web, are developed using dynamic languages like JavaScript that make the static analyses techniques stale and ineffective. As a result, there has been a growing interest in recent years to develop dynamic information flow analysis techniques. In spite of the advances in the field, dynamic information flow analysis has not been at the helm of information flow security in dynamic settings like the Web; the prime reason being that the analysis techniques and the security property related to them (non-interference) either over-approximate or are too restrictive in most cases. Concretely, the analysis techniques gen- erate a lot of false positives, do not allow legitimate release of sensitive information, support only static and rigid security policies or are not general enough to be applied to real-world applications. This thesis focuses on improving the usability of dynamic information flow techniques by presenting mechanisms that can enhance the precision and permissiveness of the analyses. It begins by presenting a sound improvement and enhancement of the permissive-upgrade strategy, a strategy widely used to enforce dynamic information flow control, which improves the strategy’s permissiveness and makes it generic in applicability. The thesis, then, presents a sound and precise control scope analysis for handling complex features like unstructured control flow and exceptions in higher-order languages. Although non-interference is a desired property for enforcing information flow control, there are program instances that require legitimate release of some parts of the secret data to provide the required functionality. Towards this end, this thesis develops a sound approach to bound information leaks dynamically while allowing information release in accordance to a pre-specified budget. The thesis concludes by applying these techniques to an information flow control-enabled Web browser and explores a policy specification mechanism that allows flexible and useful information flow policies to be specified for Web applications.

Computers become increasingly complex. Current and future systems feature configurable hardware, multiple cores with different capabilities, as well as accelerators. In addition, the memory subsystem becomes diversified too. The cache hierarchy grows deeper, is augmented with scratchpads, low-latency memory, and high-bandwidth memory. The programmer alone cannot utilize this enormous potential. Compilers have to provide insight into the program behavior, or even arrange computations and data themselves. Either way, they need a more holistic view of the program. Local transformations, which treat the iteration order, computation unit, and data layout as fixed, will not be able to fully utilize a diverse system. The polyhedral model, a high-level program representation and transformation framework, has shown great success tackling various problems in the context of diverse systems. While it is widely acknowledged for its analytical powers and transformation capabilities, it is also widely assumed to be too restrictive and fragile for real-world programs. In this thesis we improve the applicability and profitability of polyhedral-model-based techniques. Our efforts guarantee a sound polyhedral representation and extend the applicability to a wider range of programs. In addition, we introduce new applications to utilize the information available in the polyhedral program representation, including standalone optimizations and techniques to derive high-level properties.

While general-purpose Knowledge Bases (KBs) have gone a long way in compiling comprehensive knowledgee about people, events, places, etc., domain-specific KBs, such as on health, are equally important, but are less explored. Consequently, a comprehensive and expressive health KB that spans all aspects of biomedical knowledge is still missing. The main goal of this thesis is to develop principled methods for building such a KB and enabling knowledge-centric applications. We address several challenges and make the following contributions: - To construct a health KB, we devise a largely automated and scalable pattern-based knowledge extraction method covering a spectrum of different text genres and distilling a wide variety of facts from different biomedical areas. - To consider higher-arity relations, crucial for proper knowledge representation in advanced domain such as health, we generalize the fact-pattern duality paradigm of previous methods. A key novelty is the integration of facts with missing arguments by extending our framework to partial patterns and facts by reasoning over the composability of partial facts. - To demonstrate the benefits of a health KB, we devise systems for entity-aware search and analytics and for entity-relationship-oriented exploration. Extensive experiments and use-case studies demonstrate the viability of the proposed approaches.

Genomics has paved a new way to comprehend life and its evolution, and also to investigate causes of diseases and their treatment. One of the important problems in genomic analyses is haplotype assembly. Constructing complete and accurate haplotypes plays an essential role in understanding population genetics and how species evolve. In this thesis, we focus on computational approaches to haplotype assembly from third generation sequencing technologies. This involves huge amounts of sequencing data, and such data contain errors due to the single molecule sequencing protocols employed. Taking advantage of combinatorial formulations helps to correct for these errors to solve the haplotyping problem. Various computational techniques such as dynamic programming, parameterized algorithms, and graph algorithms are used to solve this problem. This thesis presents several contributions concerning the area of haplotyping. First, a novel algorithm based on dynamic programming is proposed to provide approximation guarantees for phasing a single individual. Second, an integrative approach is introduced to combining multiple sequencing datasets to generating complete and accurate haplotypes. The effectiveness of this integrative approach is demonstrated on a real human genome. Third, we provide a novel efficient approach to phasing pedigrees and demonstrate its advantages in comparison to phasing a single individual. Fourth, we present a generalized graph-based framework for performing haplotype-aware de novo assembly. Specifically, this generalized framework consists of a hybrid pipeline for generating accurate and complete haplotypes from data stemming from multiple sequencing technologies, one that provides accurate reads and other that provides long reads.

Abstract
In this thesis, we deal with two packing problems: the online bin packing
and the geometric knapsack problem. In online bin packing, the aim is to pack
a given number of items of dierent size into a minimal number of containers.
The items need to be packed one by one without knowing future items. For
online bin packing in one dimension, we present a new family of algorithms
that constitutes the rst improvement over the previously best algorithm in
almost 15 years. While the algorithmic ideas are intuitive, an elaborate analysis
is required to prove its competitive ratio. We also give a lower bound for the
competitive ratio of this family of algorithms. For online bin packing in higher
dimensions, we discuss lower bounds for the competitive ratio and show that the
ideas from the one-dimensional case cannot be easily transferred to obtain better
two-dimensional algorithms.
In the geometric knapsack problem, one aims to pack a maximum weight
subset of given rectangles into one square container. For this problem, we consider
oine approximation algorithms. For geometric knapsack with square items,
we improve the running time of the best known
PTAS
and obtain an
EPTAS
.
This shows that large running times caused by some standard techniques for
geometric packing problems are not always necessary and can be improved.
Finally, we show how to use resource augmentation to compute optimal solutions
in
EPTAS
-time, thereby improving upon the known
PTAS for this case.

From autonomous driving cars to surgical robots, robotic system has enjoyed significant growth over the past decade. With the rapid development in robotics alongside the evolution in the related fields, such as computer vision and machine learning, integrating perception, anticipation and manipulation is key to the success of future robotic system. In this thesis, we explore different ways of such integration to extend the capabilities of a robotic system to take on more challenging real world tasks. On anticipation and perception, we address the recognition of ongoing activity from videos. In particular we focus on long-duration and complex activities and hence propose a new challenging dataset to facilitate the work. We introduce hierarchical labels over the activity classes and investigate the temporal accuracy-specificity trade-offs. We propose a new method based on recurrent neural networks that learns to predict over this hierarchy and realize accuracy specificity trade-offs. Our method outperforms several baselines on this new challenge. On manipulation with perception, we propose an efficient framework for programming a robot to use human tools. We first present a novel and compact model for using tools described by a tip model. Then we explore a strategy of utilizing a dual-gripper approach for manipulating tools – motivated by the absence of dexterous hands on widely available general purpose robots. Afterwards, we embed the tool use learning into a hierarchical architecture and evaluate it on a Baxter research robot. Finally, combining perception, anticipation and manipulation, we focus on a block stacking task. First we explore how to guide robot to place a single block into the scene without collapsing the existing structure. We introduce a mechanism to predict physical stability directly from visual input and evaluate it first on a synthetic data and then on real-world block stacking. Further, we introduce the target stacking task where the agent stacks blocks to reproduce a tower shown in an image. To do so, we create a synthetic block stacking environment with physics simulation in which the agent can learn block stacking end-to-end through trial and error, bypassing to explicitly model the corresponding physics knowledge. We propose a goal-parametrized GDQN model to plan with respect to the specific goal. We validate the model on both a navigation task in a classic gridworld environment and the block stacking task.

Today in this Big Data era, overwhelming amounts of textual information across different sources with a high degree of redundancy has made it hard for a consumer to retrospect on past events. A plausible solution is to link semantically similar information contained across the different sources to enforce a structure thereby providing multiple access paths to relevant information. Keeping this larger goal in view, this work uses Wikipedia and online news articles as two prominent yet disparate information sources to address the following three problems: • We address a linking problem to connect Wikipedia excerpts to news articles by casting it into an IR task. Our novel approach integrates time, geolocations, and entities with text to identify relevant documents that can be linked to a given excerpt. • We address an unsupervised extractive multi-document summarization task to generate a fixed-length event digest that facilitates efficient consumption of information contained within a large set of documents. Our novel approach proposes an ILP for global inference across text, time, geolocations, and entities associated with the event. • To estimate temporal focus of short event descriptions, we present a semi-supervised approach that leverages redundancy within a longitudinal news collection to estimate accurate probabilistic time models. Extensive experimental evaluations demonstrate the effectiveness and viability of our proposed approaches towards achieving the larger goal.

Detection of biomarker genes play a crucial role in disease detection and treatment. Bioinformatics offers a variety of approaches for identification of biomarker genes which play key roles in complex diseases. These computational approaches enhance the insight derived from experiments and reduce the efforts of biologists and experimentalists. This is essentially achieved through prioritizing a set of genes with certain attributes. In this thesis, we show that understanding the regulatory mechanisms underlying stem cells helps to identify cancer biomarkers. We got inspired by the regulatory mechanisms of the pluripotency network in mouse embryonic stem cells and formulated the problem where a set of master regulatory genes in regulatory networks is identified with two combinatorial optimization problems namely as minimum dominating set and minimum connected dominating set in weakly and strongly connected components. Then we applied the developed methods to regulatory cancer networks to identify disease-associated genes and anti-cancer drug targets in breast cancer and hepatocellular carcinoma. As not all the nodes in the solutions are critical, we developed a prioritization method to rank a set of candidate genes which are related to a certain disease based on systematic analysis of the genes that are differentially expressed in tumor and normal conditions. Moreover, we demonstrated that the topological features in regulatory networks surrounding differentially expressed genes are highly consistent in terms of using the output of several analysis tools. We compared two randomization strategies for TF-miRNA co-regulatory networks to infer significant network motifs underlying cellular identity. We showed that the edge-type conserving method surpasses the non-conserving method in terms of biological relevance and centrality overlap. We presented several web servers and software packages that are publicly available at no cost. The Cytoscape plugin of minimum connected dominating set identifies a set of key regulatory genes in a user provided regulatory network based on a heuristic approach. The ILP formulations of minimum dominating set and minimum connected dominating set return the optimal solutions for the aforementioned problems. Our source code is publicly available. The web servers TFmiR and TFmiR2 construct disease-, tissue-, process-specific networks for the sets of deregulated genes and miRNAs provided by a user. They highlight topological hotspots and offer detection of three- and four-node FFL motifs as a separate web service for both organisms mouse and human.

Machine learning is transforming the world. Its application areas span privacy
sensitive and security critical tasks such as human identification and self-driving
cars. These applications raise privacy and security related questions that are not
fully understood or answered yet: Can automatic person recognisers identify people
in photos even when their faces are blurred? How easy is it to find an adversarial
input for a self-driving car that makes it drive off the road?
This thesis contributes one of the first steps towards a better understanding of
such concerns. We observe that many privacy and security critical scenarios for
learned models involve input data manipulation: users obfuscate their identity by
blurring their faces and adversaries inject imperceptible perturbations to the input
signal. We introduce a data manipulator framework as a tool for collectively describing
and analysing privacy and security relevant scenarios involving learned models.
A data manipulator introduces a shift in data distribution for achieving privacy or
security related goals, and feeds the transformed input to the target model. This
framework provides a common perspective on the studies presented in the thesis.
We begin the studies from the user’s privacy point of view. We analyse the
efficacy of common obfuscation methods like face blurring, and show that they
are surprisingly ineffective against state of the art person recognition systems. We
then propose alternatives based on head inpainting and adversarial examples. By
studying the user privacy, we also study the dual problem: model security. In model
security perspective, a model ought to be robust and reliable against small amounts
of data manipulation. In both cases, data are manipulated with the goal of changing
the target model prediction. User privacy and model security problems can be
described with the same objective.
We then study the knowledge aspect of the data manipulation problem. The more
one knows about the target model, the more effective manipulations one can craft.
We propose a game theoretic manipulation framework to systematically represent
the knowledge level on the target model and derive privacy and security guarantees.
We then discuss ways to increase knowledge about a black-box model by only querying
it, deriving implications that are relevant to both privacy and security perspectives.

With the goal of lifting model-based guidance from the propositional setting to first-
order logic, I have developed an approximation theorem proving approach based on
counterexample-guided abstraction refinement. A given clause set is transformed
into a simplified form where satisfiability is decidable. This approximation extends
the signature and preserves unsatisfiability: if the simplified clause set is satisfi-
able, so is the original clause set. A resolution refutation generated by a decision
procedure on the simplified clause set can then either be lifted to a refutation in
the original clause set, or it guides a refinement excluding the previously found
unliftable refutation. This way the approach is refutationally complete.
The monadic shallow linear Horn fragment, which is the initial target of the
approximation, is well-known to be decidable. It was a long standing open prob-
lem how to extend the fragment to the non-Horn case, preserving decidability, that
would, e.g., enable to express non-determinism in protocols. I have now proven de-
cidability of the non-Horn monadic shallow linear fragment via ordered resolution.
I further extend the clause language with a new type of constraints, called
straight dismatching constraints. The extended clause language is motivated by an
improved refinement of the approximation-refinement framework. All needed oper-
ations on straight dismatching constraints take linear or linear logarithmic time in
the size of the constraint. Ordered resolution with straight dismatching constraints
is sound and complete and the monadic shallow linear fragment with straight dis-
matching constraints is decidable.
I have implemented my approach based on the SPASS theorem prover. On cer-
tain satisfiable problems, the implementation shows the ability to beat established
provers such as SPASS, iProver, and Vampire.

Following a period of expedited progress in the capabilities of digital systems, the society begins to realize that systems designed to assist people in various tasks can also harm individuals and society. Mediating access to information and explicitly or implicitly ranking people in increasingly many applications, search systems have a substantial potential to contribute to such unwanted outcomes. Since they collect vast amounts of data about both searchers and search subjects, they have the potential to violate the privacy of both of these groups of users. Moreover, in applications where rankings influence people's economic livelihood outside of the platform, such as sharing economy or hiring support websites, search engines have an immense economic power over their users in that they control user exposure in ranked results. This thesis develops new models and methods broadly covering different aspects of privacy and fairness in search systems for both searchers and search subjects. Specifically, it makes the following contributions: (1) We propose a model for computing individually fair rankings where search subjects get exposure proportional to their relevance. The exposure is amortized over time using constrained optimization to overcome searcher attention biases while preserving ranking utility. (2) We propose a model for computing sensitive search exposure where each subject gets to know the sensitive queries that lead to her profile in the top-k search results. The problem of finding exposing queries is technically modeled as reverse nearest neighbor search, followed by a weekly-supervised learning to rank model ordering the queries by privacy-sensitivity. (3) We propose a model for quantifying privacy risks from textual data in online communities. The method builds on a topic model where each topic is annotated by a crowdsourced sensitivity score, and privacy risks are associated with a user's relevance to sensitive topics. We propose relevance measures capturing different dimensions of user interest in a topic and show how they correlate with human risk perceptions. (4) We propose a model for privacy-preserving personalized search where search queries of different users are split and merged into synthetic profiles. The model mediates the privacy-utility trade-off by keeping semantically coherent fragments of search histories within individual profiles, while trying to minimize the similarity of any of the synthetic profiles to the original user profiles. The models are evaluated using information retrieval techniques and user studies over a variety of datasets, ranging from query logs, through social media and community question answering postings, to item listings from sharing economy platforms.

The ability to interconnect multiple knowledge repositories within a single framework is a key asset for various use cases such as document retrieval and question answering. However, independently created repositories are inherently heterogeneous, reflecting their diverse origins. Thus, there is a need to align concepts and entities across knowledge repositories. A limitation of prior work is the assumption of high afinity between the repositories at hand, in terms of structure and terminology. The goal of this dissertation is to develop methods for constructing and curating alignments between multi-cultural knowledge repositories. The first contribution is a system, ACROSS, for reducing the terminological gap between repositories. The second contribution is two alignment methods, LILIANA and SESAME, that cope with structural diversity. The third contribution, LAIKA, is an approach to compute alignments between dynamic repositories. Experiments with a suite ofWeb-scale knowledge repositories show high quality alignments. In addition, the application benefits of LILIANA and SESAME are demonstrated by use cases in search and exploration.

Persistent Homology is a tool to analyze and visualize the shape of data from a topological viewpoint. It computes persistence, which summarizes the evolution of topological and geometric information about metric spaces over multiple scales of distances. While computing persistence is quite efficient for low-dimensional topological features, it becomes overwhelmingly expensive for medium to high-dimensional features. In this thesis, we attack this computational problem from several different angles. We present efficient techniques to approximate the persistence of metric spaces. Three of our methods are tailored towards general point clouds in Euclidean spaces. We make use of high dimensional lattice geometry to reduce the cost of the approximations. In particular, we discover several properties of the Permutahedral lattice, whose Voronoi cell is well-known for its combinatorial properties. The last method is suitable for point clouds with low intrinsic dimension, where we exploit the structural properties of the point set to tame the complexity. In some cases, we achieve a reduction in size complexity by trading off the quality of the approximation. Two of our methods work particularly well in conjunction with dimension-reduction techniques: we arrive at the first approximation schemes whose complexities are only polynomial in the size of the point cloud, and independent of the ambient dimension. On the other hand, we provide a lower bound result: we construct a point cloud that requires super-polynomial complexity for a high-quality approximation of the persistence. Together with our approximation schemes, we show that polynomial complexity is achievable for rough approximations, but impossible for sufficiently fine approximations. For some metric spaces, the intrinsic dimension is low in small neighborhoods of the input points, but much higher for large scales of distances. We develop a concept of local intrinsic dimension to capture this property. We also present several applications of this concept, including an approximation method for persistence. This thesis is written in English.

Languages shape thoughts. This is true for human spoken languages as much as for programming languages. As computers continue to expand their dominance in almost every aspect of our lives, the need to more adequately express new concepts and domains in computer languages arise. However, to evolve our thoughts we need to evolve the languages we speek in. But what tools are there to create and upgrade the computer languages? How can we encourage developers to define their own languages quickly to best match the domains they work in? Nowadays two main approaches exists. Dedicated language tools and parser generators allows to define new standalone languages from scratch. Alternatively, one can “abuse” sufficiently flexible host languages to embed small domain- specific languages within them. Both approaches have their own respective limitations. Creating standalone languages is a major endeavor. Such languages cannot be combined easily with other languages. Embedding, on the other hand, is limited by the syntax of the host language. Embedded languages, once defined, are always present without clear distinction between them and the host language. When used extensively, it leads to one humungous conglomerate of languages, with confusing syntax and unexpected interactions. In this work we present an alternative: ManyDSL. It is a unique interpreter and compiler taking strength from both approaches, while avoiding the above weaknesses. ManyDSL features its own LL1 parser generator, breaking the limits of the syntax of the host language. The grammar description is given in the same host language as the rest of the program. Portions of the grammar can be parametrized and abstracted into functions, in order to be used in other language definitions. Languages are created on the fly during the interpretation process and may be used to parse selected fragments of the subsequent source files. Similarly to embedded languages, ManyDSL translates all custom languages to the same host language before execution. The host language uses a continuation- passing style approach with a novel, dynamic approach to staging. The staging allows for arbitrary partial evaluation, and executing code at different phases of the compilation process. This can be used to define domain-specific optimiza- tions and auxiliary computation (e.g. for verification) — all within an entirely functional approach, without any explicit use of abstract syntax trees and code transformations. With the help of ManyDSL a user is able to create new languages with distinct, easily recognizable syntax. Moreover, he is able to define and use many of such languages within a single project. Languages can be switched with a well-defined boundary, enabling their interaction in a clear and controlled way. ManyDSL is meant to be the first step towards a broader language pluralism. With it we want to encourage developers to design and use languages that best suit their needs. We believe that over time, with the help of grammar libraries, creating new languages will become accessible to every programmer.

This doctoral thesis aims towards distributed natural computing inspired by the slime mold Physarum polycephalum. The vein networks formed by this organism presumably support efficient transport of protoplasmic fluid. Devising models which capture the natural efficiency of the organism and form a suitable basis for the development of natural computing algorithms is an interesting and challenging goal. We start working towards this goal by designing and executing wet-lab experi- ments geared towards producing a large number of images of the vein networks of P. polycephalum. Next, we turn the depicted vein networks into graphs using our own custom software called Nefi. This enables a detailed numerical study, yielding a catalogue of characterizing observables spanning a wide array of different graph properties. To share our results and data, i.e. raw experimental data, graphs and analysis results, we introduce a dedicated repository revolving around slime mold data, the Smgr. The purpose of this repository is to promote data reuse and to foster a practice of increased data sharing. Finally we present a model based on interacting electronic circuits including current controlled voltage sources, which mimics the emergent flow patterns observed in live P. polycephalum. The model is simple, distributed and robust to changes in the underlying network topology. Thus it constitutes a promising basis for the development of distributed natural computing algorithms.

The evolution of search from keywords to entities has necessitated the efficient harvesting and management of entity-centric information for constructing knowledge bases catering to various applications such as semantic search, question answering, and information retrieval. The vast amounts of natural language texts available across diverse domains on the Web provide rich sources for discovering facts about named entities such as people, places, and organizations.

A key challenge, in this regard, entails the need for precise identification and disambiguation of entities across documents for extraction of attributes/relations and their proper representation in knowledge bases. Additionally, the applicability of such repositories not only involves the quality and accuracy of the stored information, but also storage management and query processing efficiency. This dissertation aims to tackle the above problems by presenting efficient approaches for entity-centric knowledge
acquisition from texts and its representation in knowledge repositories.

This dissertation presents a robust approach for identifying text phrases pertaining to the same named entity across huge corpora, and their disambiguation to canonical entities present in a knowledge base, by using enriched semantic contexts and link validation encapsulated in a hierarchical clustering framework. This work further presents language and consistency features for classification models to compute the credibility of obtained textual facts, ensuring quality of the extracted information. Finally, an encoding algorithm, using frequent term detection and improved data locality, to represent entities for enhanced knowledge base storage and query performance is presented.

We study three problems. The first is the phenomenon of metastability in digital circuits. This is a state of bistable storage elements, such as registers, that is neither logical 0 nor 1 and breaks the abstraction of Boolean logic. We propose a time- and value-discrete model for metastability in digital circuits and show that it reflects relevant physical properties. Further, we propose the fundamentally new approach of using logical masking to perform meaningful computations despite the presence of metastable upsets and analyze what functions can be computed in our model. Additionally, we show that circuits with masking registers grow computationally more powerful with each available clock cycle. The second topic are parallel algorithms, based on an algebraic abstraction of the Moore-Bellman-Ford algorithm, for solving various distance problems. Our focus are distance approximations that obey the triangle inequality while at the same time achieving polylogarithmic depth and low work. Finally, we study the continuous Terrain Guarding Problem. We show that it has a rational discretization with a quadratic number of guard candidates, establish its membership in NP and the existence of a PTAS, and present an efficient implementation of a solver.

Two-dimensional, conventional images are gradually losing their hegemony, leaving room for novel formats. Among these, 8 bit images give place to high dynamic range (HDR) image formats, allowing to improve colour gamut and visibility of details in dark and bright areas leading to a more immersive viewing experience. It opens wide opportunities for post-processing, which can be useful for artistic rendering, enhancement of viewing experience or medical applications. Simultaneously, light-field scene representation as well is gaining importance, propelled by the recent reappearance of virtual reality, the improvement of both acquisition techniques, and computational and storage capabilities. Light-field data as well allows to achieve a broad range of effects in post-production: among others, it enables a change of a camera position, an aperture or a focal length. It facilitates object insertions and simplifies visual effects workflow by integrating 3D nature of visual effects with 3D nature of light fields. Content generation is one of the stumbling blocks in these realms. Sensor limitations of a conventional camera do not allow to capture wide dynamic range. This especially is the case for mobile devices, where small sensors are optimised for capturing in high-resolution. The “HDR mode” often encountered on such devices, relies on techniques called “exposure fusion” and allows to partially overcome the limited range of a sensor. The HDR video at the same time remains a challenging problem. We suggest a solution for an HDR video capturing on a mobile device. We analyse dynamic range of motion regions, the regions which are the most prone to reconstruction artefacts, and suggest a real-time exposure selection algorithm. Further, an HDR content visualization task often requires an input to be in absolute values. We address this problem by presenting a calibration algorithm that can be applied to existent imagery and does not require any additional measurement hardware. Finally, as light fields use becomes more common, a key challenge is the ability to edit or modify the appearance of the objects in the light field. To this end, we propose a multidimensional filtering approach in which the specular highlights are filtered in the spatial and angular domains to target a desired increase of the material roughness.

Knowledge Bases are one of the key components of Natural Language Understanding systems. For example, DBpedia, YAGO, and Wikidata capture and organize knowledge about named entities and relations between them, which is often crucial for tasks like Question Answering and Named Entity Disambiguation. While Knowledge Bases have good coverage of prominent entities, they are often limited with respect to relations. The goal of this thesis is to bridge this gap and automatically create lexicons of textual representations of relations, namely relational phrases. The lexicons should contain information about paraphrases, hierarchy, as well as semantic types of arguments of relational phrases. The thesis makes three main contributions. The first contribution addresses disambiguating relational phrases by aligning them with the WordNet dictionary. Moreover, the alignment allows imposing the WordNet hierarchy on the relational phrases. The second contribution proposes a method for graph construction of relations using Probabilistic Graphical Models. In addition, we apply this model to relation paraphrasing. The third contribution presents a method for constructing a lexicon of relational paraphrases with fine-grained semantic typing of arguments. This method is based on information from a multilingual parallel corpus.

Graph is a vital abstract data type that has profound significance in several applications. Because of its versitality, graphs have been adapted into several different forms and one such adaption with many practical applications is the “Labeled Graph”, where vertices and edges are labeled. An enormous research effort has been invested in to the task of managing and querying graphs, yet a lot challenges are left unsolved. In this thesis, we advance the state-of-the-art for the following query models, and propose a distributed solution to process them in an efficient and scalable manner.
• Set Reachability. We formalize and investigate a generalization of the basic notion of reachability, called set reachability. Set reachability deals with finding all reachable pairs for a given source and target sets. We present a non-iterative distributed solution that takes only a single round of communication for any set reachability query. This is achieved by precomputation, replication, and indexing of partial reachabilities among the boundary vertices.
• Basic Graph Patterns (BGP). Supported by majority of query languages, BGP queries are a common mode of querying knowledge graphs, biological datasets, etc. We present a novel distributed architecture that relies on the concepts of asynchronous executions, join-ahead pruning, and a multi-threaded query processing framework to process BGP queries in an efficient and scalable manner.
• Generalized Graph Patterns (GGP). These queries combine the semantics of pattern matching and navigational queries, and are popular in scenarios where the schema of an underlying graph is either unknown or partially known. We present a distributed solution with bimodal indexing layout that individually support efficient processing of BGP queries and navigational queries. Furthermore, we design a unified query optimizer and a processor to efficiently process GGP queries and also in a scalable manner.
To this end, we propose a prototype distributed engine, coined “TriAD” (Triple Asynchronous and Distributed) that supports all the aforementioned query models. We also provide a detailed empirical evaluation of TriAD in comparison to several state-of-the-art systems over multiple real-world and synthetic datasets.

Markov Decision Processes (MDPs) constitute a mathematical framework for modelling systems featuring both probabilistic and nondeterministic behaviour. They are widely used to solve sequential decision making problems and applied successfully in operations research, arti?cial intelligence, and stochastic control theory, and have been extended conservatively to the model of probabilistic automata in the context of concurrent probabilistic systems. However, when modeling a physical system they suffer from several limitations. One of the most important is the inherent loss of precision that is introduced by measurement errors and discretization artifacts which necessarily happen due to incomplete knowledge about the system behavior. As a result, the true probability distribution for transitions is in most cases an uncertain value, determined by either external parameters or con?dence intervals. Interval Markov decision processes (IMDPs) generalize classical MDPs by having interval-valued transition probabilities. They provide a powerful modelling tool for probabilistic systems with an additional variation or uncertainty that re?ects the absence of precise knowledge concerning transition probabilities. In this dissertation, we focus on decision algorithms for modelling and performance evaluation of such probabilistic systems leveraging techniques from mathematical optimization. From a modelling viewpoint, we address probabilistic bisimulations to reduce the size of the system models while preserving the logical properties they satisfy. We also discuss the key ingredients to construct systems by composing them out of smaller components running in parallel. Furthermore, we introduce a novel stochastic model, Uncertain weighted Markov Decision Processes (UwMDPs), so as to capture quantities like preferences or priorities in a nondeterministic scenario with uncertainties. This model is close to the model of IMDPs but more convenient to work with in the context of bisimulation minimization. From a performance evaluation perspective, we consider the problem of multi-objective robust strategy synthesis for IMDPs, where the aim is to ?nd a robust strategy that guarantees the satisfaction of multiple properties at the same time in face of the transition probability uncertainty. In this respect, we discuss the computational complexity of the problem and present a value iteration-based decision algorithm to approximate the Pareto set of achievable optimal points. Moreover, we consider the problem of computing maximal/minimal reward-bounded reachability probabilities on UwMDPs, for which we present an ef?cient algorithm running in pseudo-polynomial time. We demonstrate the practical effectiveness of our proposed approaches by applying them to a collection of real-world case studies using several prototypical tools.

Visual object detection has seen substantial improvements during the last years due to the possibilities enabled by deep learning. While research on image classification provides continuous progress on how to learn image representations and classifiers jointly, object detection research focuses on identifying how to properly use deep learning technology to effectively localise objects. In this thesis, we analyse and improve different aspects of the commonly used detection pipeline. We analyse ten years of research on pedestrian detection and find that improvement of feature representations was the driving factor. Motivated by this finding, we adapt an end-to-end learned detector architecture from general object detection to pedestrian detection. Our deep network outperforms all previous neural networks for pedestrian detection by a large margin, even without using additional training data. After substantial improvements on pedestrian detection in recent years, we investigate the gap between human performance and state-of-the-art pedestrian detectors. We find that pedestrian detectors still have a long way to go before they reach human performance, and we diagnose failure modes of several top performing detectors, giving direction to future research. As a side-effect we publish new, better localised annotations for the Caltech pedestrian benchmark. We analyse detection proposals as a preprocessing step for object detectors. We establish different metrics and compare a wide range of methods according to these metrics. By examining the relationship between localisation of proposals and final object detection performance, we define and experimentally verify a metric that can be used as a proxy for detector performance. Furthermore, we address a structural weakness of virtually all object detection pipelines: non-maximum suppression. We analyse why it is necessary and what the shortcomings of the most common approach are. To address these problems, we present work to overcome these shortcomings and to replace typical non-maximum suppression with a learnable alternative. The introduced paradigm paves the way to true end-to-end learning of object detectors without any post-processing. In summary, this thesis provides analyses of recent pedestrian detectors and detection proposals, improves pedestrian detection by employing deep neural networks, and presents a viable alternative to traditional non-maximum suppression.

An information retrieval (IR) system assists people in consuming huge amount of data, where the evaluation and the construction of such systems are important. However, there exist two difficulties: the overwhelmingly large number of query-document pairs to judge, making IR evaluation a manually laborious task; and the complicated patterns to model due to the non-symmetric, heterogeneous relationships between a query-document pair, where different interaction patterns such as term dependency and proximity have been demonstrated to be useful, yet are non-trivial for a single IR model to encode. In this thesis we attempt to address both difficulties from the perspectives of IR evaluation and of the retrieval model respectively, by reducing the manual cost with automatic methods, by investigating the usage of crowdsourcing in collecting preference judgments, and by proposing novel neural retrieval models. In particular, to address the large number of query-document pairs in IR evaluation, a low-cost selective labeling method is proposed to pick out a small subset of representative documents for manual judgments in favor of the follow-up prediction for the remaining query-document pairs; furthermore, a language-model based cascade measure framework is developed to evaluate the novelty and diversity, utilizing the content of the labeled documents to mitigate incomplete labels. In addition, we also attempt to make the preference judgments practically usable by empirically investigating different properties of the judgments when collected via crowdsourcing; and by proposing a novel judgment mechanism, making a compromise between the judgment quality and the number of judgments. Finally, to model different complicated patterns in a single retrieval model, inspired by the recent advances in deep learning, we develop novel neural IR models to incorporate different patterns like term dependency, query proximity, density of relevance, and query coverage in a single model. We demonstrate their superior performances through evaluations on different datasets.

In this thesis, we investigate a certain type of local similarities between geometric shapes. We analyze the surface of a shape and find all points that are contained inside identical, spherical neighborhoods of a radius r. This allows us to decompose surfaces into canonical sets of building blocks, which we call microtiles. We show that the microtiles of a given object can be used to describe a complete family of related shapes. Each of these shapes is locally similar to the original, meaning that it contains identical r-neighborhoods, but can have completely different global structure. This allows for using r-microtiling for inverse modeling of shape variations and we develop a method for shape decomposi
tion into rigid, 3D manufacturable building blocks that can be used to physically assemble shape collections. We obtain a small set of constructor pieces that are well suited for manufacturing and assembly by a novel method for tiling grammar simplification: We consider the connection between microtiles and noncontext-free tiling grammars and optimize a graph-based representation, finding a good balance between expressiveness, simplicity and ease of assembly. By changing the objective function, we can re-purpose the grammar simplification method for mesh compression. The microtiles of a model encode its geometrically redundant parts, which can be used for creating shape representations with minimal memory footprints. Altogether, with this work we attempt to give insights into how rigid partial symmetries can be efficiently computed and used in the context of inverse modeling of shape families, shape understanding, and compression.

Much progress has been made in image and video segmentation
over the last years. To a large extent, the success can be attributed to
the strong appearance models completely learned from data, in particular
using deep learning methods. However,to perform best these methods require
large representative datasets for training with expensive pixel-level
annotations, which in case of videos are prohibitive to obtain. Therefore,
there is a need to relax this constraint and to consider alternative forms
of supervision, which are easier and cheaper to collect. In this thesis,
we aim to develop algorithms for learning to segment in images and videos
with different levels of supervision.
First, we develop approaches for training convolutional networks with weaker
forms of supervision, such as bounding boxes or image labels, for object
boundary estimation and semantic/instance labelling tasks. We propose to
generate pixel-level approximate groundtruth from these weaker forms of
annotations to train a network, which allows to achieve high-quality
results comparable to the full supervision quality without any
modifications of the network architecture or the training procedure.
Second, we address the problem of the excessive computational and memory
costs inherent to solving video segmentation via graphs. We propose
approaches to improve the runtime and memory efficiency as well as the
output segmentation quality by learning from the available training data
the best representation of the graph. In particular, we contribute with
learning must-link constraints, the topology and edge weights of the graph
as well as enhancing the graph nodes - superpixels - themselves.
Third, we tackle the task of pixel-level object tracking and address the
problem of the limited amount of densely annotated video data for training
convolutional networks. We introduce an architecture which allows training
with static images only and propose an elaborate data synthesis scheme
which creates a large number of training examples close to the target
domain from the given first frame mask. With the proposed techniques we
show that densely annotated consequent video data is not necessary to
achieve high-quality temporally coherent video segmentationresults.
In summary, this thesis advances the state of the art in weakly supervised
image segmentation, graph-based video segmentation and pixel-level object
tracking and contributes with the new ways of training convolutional
networks with a limited amount of pixel-level annotated training data.

Modern image classification methods are based on supervised learning algorithms that require labeled training data. However, only a limited amount of annotated data may be available in certain applications due to scarcity of the data itself or high costs associated with human annotation. Introduction of additional information and structural constraints can help improve the performance of a learning algorithm. In this thesis, we study the framework of learning using privileged information and demonstrate its relation to learning with instance weights. We also consider multitask feature learning and develop an efficient dual optimization scheme that is particularly well suited to problems with high dimensional image descriptors. Scaling annotation to a large number of image categories leads to the problem of class ambiguity where clear distinction between the classes is no longer possible. Many real world images are naturally multilabel yet the existing annotation might only contain a single label. In this thesis, we propose and analyze a number of loss functions that allow for a certain tolerance in top k predictions of a learner. Our results indicate consistent improvements over the standard loss functions that put more penalty on the first incorrect prediction compared to the proposed losses. All proposed learning methods are complemented with efficient optimization schemes that are based on stochastic dual coordinate ascent for convex problems and on gradient descent for nonconvex formulations.

Computer Vision has undergone major changes over the recent five years. Here, we investigate if the performance of such architectures generalizes to more complex tasks that require a more holistic approach to scene comprehension. The presented work focuses on learning spatial and multi-modal representations, and the foundations of a Visual Turing Test, where the scene understanding is tested by a series of questions about its content. In our studies, we propose DAQUAR, the first ‘question answering about real-world images’ dataset together with methods, termed a symbolic-based and a neural-based visual question answering architectures, that address the problem. The symbolic-based method relies on a semantic parser, a database of visual facts, and a bayesian formulation that accounts for various interpretations of the visual scene. The neural-based method is an end-to-end architecture composed of a question encoder, image encoder, multimodal embedding, and answer decoder. This architecture has proven to be effective in capturing language-based biases. It also becomes the standard component of other visual question answering architectures. Along with the methods, we also investigate various evaluation metrics that embraces uncertainty in word's meaning, and various interpretations of the scene and the question.

One of the major hurdles preventing the full exploitation of information from online communities is the widespread concern regarding the quality and credibility of user-contributed content. Prior works in this domain operate on a static snapshot of the community, making strong assumptions about the structure of the data (e.g., relational tables), or consider only shallow features for text classification. To address the above limitations, we propose probabilistic graphical models that can leverage the joint interplay between multiple factors in online communities --- like user interactions, community dynamics, and textual content --- to automatically assess the credibility of user-contributed online content, and the expertise of users and their evolution with user-interpretable explanation. To this end, we devise new models based on Conditional Random Fields for different settings like incorporating partial expert knowledge for semi-supervised learning, and handling discrete labels as well as numeric ratings for fine-grained analysis. This enables applications such as extracting reliable side-effects of drugs from user-contributed posts in healthforums, and identifying credible content in news communities. Online communities are dynamic, as users join and leave, adapt to evolving trends, and mature over time. To capture this dynamics, we propose generative models based on Hidden Markov Model, Latent Dirichlet Allocation, and Brownian Motion to trace the continuous evolution of user expertise and their language model over time. This allows us to identify expert users and credible content jointly over time, improving state-of-the-art recommender systems by explicitly considering the maturity of users. This also enables applications such as identifying helpful product reviews, and detecting fake and anomalous reviews with limited information.

Although virtually all cells in an organism share the same genome, regulatory mechanisms give rise to hundreds of different, highly specialized cell types. Understanding these mechanisms has been in the limelight of epigenomic research. It is now evident that cellular identity is inscribed in the epigenome of each individual cell. Nonetheless, the precise mechanisms by which different epigenomic marks are involved in regulating gene expression are just beginning to be unraveled. Furthermore, epigenomic patterns are highly dynamic and subject to environmental influences. Any given cell type is defined by cell populations exhibiting epigenetic heterogeneity at different levels. Characterizing this heterogeneity is paramount in understanding the regulatory role of the epigenome. Different epigenomic marks can be profiled using high-throughput sequencing, and global initiatives have started to provide a comprehensive picture of the human epigenome by assaying a multitude of marks across a broad panel of cell types and conditions. In particular, DNA methylation has been extensively studied for its gene-regulatory role in health and disease. This thesis describes computational methods and pipelines for the analysis of DNA methylation data. It provides concepts for addressing bioinformatic challenges such as the processing of large, epigenome-wide datasets and integrating multiple levels of information in an interpretable manner. We developed RnBeads, an R package that facilitates comprehensive, interpretable analysis of large-scale DNA methylation datasets at the level of single CpGs or genomic regions of interest. With the epiRepeatR pipeline, we introduced additional tools for studying global patterns of epigenomic marks in transposons and other repetitive regions of the genome. Blood-cell differentiation represents a useful model for studying trajectories of cellular differentiation. We developed and applied bioinformatic methods to dissect the DNA methylation landscape of the hematopoietic system. Here, we provide a broad outline of cell-type-specific DNA methylation signatures and phenotypic diversity reflected in the epigenomes of human mature blood cells. We also describe the DNA methylation dynamics in the process of immune memory formation in T helper cells. Moreover, we portrayed epigenetic fingerprints of defined progenitor cell types and derived computational models that were capable of accurately inferring cell identity. We used these models in order to characterize heterogeneity in progenitor cell populations, to identify DNA methylation signatures of hematopoietic differentiation and to infer the epigenomic similarities of blood cell types. Finally, by interpreting DNA methylation patterns in leukemia and derived pluripotent cells, we started to discern how epigenomic patterns are altered in disease and explored how reprogramming of these patterns could potentially be used to restore a non-malignant state. In summary, this work showcases novel methods and computational tools for the identification and interpretation of epigenetic signatures of cell identity. It provides a detailed view on the epigenomic landscape spanned by DNA methylation patterns in hematopoietic cells that enhances our understanding of epigenetic regulation in cell differentiation and disease.

This dissertation introduces a range of new methods to produce images of virtual scenes in a matter of milliseconds. Imposing as few constraints as possible on the set of scenes that can be handled, e.g., regarding geometric changes over time or lighting conditions, precludes pre-computations and makes this a particularly difficult problem. We first present a general approach, called deep screen space, using which a variety of light transport aspects can be simulated within the aforementioned setting. This approach is then further extended to additionally handle scenes containing participating media like clouds. We also show how to improve the correctness of deep screen space and related algorithms by accounting for mutual visibility of points in a scene. After that, we take a completely different point of view on image generation using a learning-based approach to approximate a rendering function. We show that neural networks can hallucinate shading effects which otherwise have to be computed using costly analytic computations. Finally, we contribute a holistic framework to deal with phosphorescent materials in computer graphics, covering all aspects from acquisition of real materials, to easy editing, to image synthesis.

Information and knowledge extraction from natural language text is a key asset for question answering, semantic search, automatic summarization, and other machine reading applications. There are many sub-tasks involved such as named entity recognition, named entity disambiguation, co-reference resolution, relation extraction, event detection, discourse parsing, and others. Solving these tasks is challenging as natural language text is unstructured, noisy, and ambiguous. Key challenges, which focus on identifying and linking named entities, as well as discovering relations between them, include: • High NERD Quality. Named entity recognition and disambiguation, NERD for short, are preformed first in the extraction pipeline. Their results may affect other downstream tasks. • Coverage vs. Quality of Relation Extraction. Model-based information extraction methods achieve high extraction quality at low coverage, whereas open information extraction methods capture relational phrases between entities. However, the latter degrades in quality by non-canonicalized and noisy output. These limitations need to be overcome. • On-the-fly Knowledge Acquisition. Real-world applications such as question answering, monitoring content streams, etc. demand on-the-fly knowledge acquisition. Building such an end-to-end system is challenging because it requires high throughput, high extraction quality, and high coverage. This dissertation addresses the above challenges, developing new methods to advance the state of the art. The first contribution is a robust model for joint inference between entity recognition and disambiguation. The second contribution is a novel model for relation extraction and entity disambiguation on Wikipediastyle text. The third contribution is an end-to-end system for constructing querydriven, on-the-fly knowledge bases.

Generating natural language descriptions for visual data links computer vision and computational linguistics. Being able to generate a concise and human-readable description of a video is a step towards visual understanding. At the same time, grounding natural language in visual data provides disambiguation for the linguistic concepts, necessary for many applications. This thesis focuses on both directions and tackles three specific problems. First, we develop recognition approaches to understand video of complex cooking activities. We propose an approach to generate coherent multi-sentence descriptions for our videos. Furthermore, we tackle the new task of describing videos at variable level of detail. Second, we present a large-scale dataset of movies and aligned professional descriptions. We propose an approach, which learns from videos and sentences to describe movie clips relying on robust recognition of visual semantic concepts. Third, we propose an approach to ground textual phrases in images with little or no localization supervision, which we further improve by introducing Multimodal Compact Bilinear Pooling for combining language and vision representations. Finally, we jointly address the task of describing videos and grounding the described people. To summarize, this thesis advances the state-of-the-art in automatic video description and visual grounding and also contributes large datasets for studying the intersection of computer vision and computational linguistics.

Entity recognition and disambiguation (ERD) for the biomedical domain are notoriously difficult problems due to the variety of entities and their often long names in many variations. Existing works focus heavily on the molecular level in two ways. First, they target scientific literature as the input text genre. Second, they target single, highly specialized entity types such as chemicals, genes, and proteins. However, a wealth of biomedical information is also buried in the vast universe of Web content. In order to fully utilize all the information available, there is a need to tap into Web content as an additional input. Moreover, there is a need to cater for other entity types such as symptoms and risk factors since Web content focuses on consumer health. The goal of this thesis is to investigate ERD methods that are applicable to all entity types in scientific literature as well as Web content. In addition, we focus on under-explored aspects of the biomedical ERD problems -- scalability, long noun phrases, and out-of-knowledge base (OOKB) entities. This thesis makes four main contributions, all of which leverage knowledge in UMLS (Unified Medical Language System), the largest and most authoritative knowledge base (KB) of the biomedical domain. The first contribution is a fast dictionary lookup method for entity recognition that maximizes throughput while balancing the loss of precision and recall. The second contribution is a semantic type classification method targeting common words in long noun phrases. We develop a custom set of semantic types to capture word usages; besides biomedical usage, these types also cope with non-biomedical usage and the case of generic, non-informative usage. The third contribution is a fast heuristics method for entity disambiguation in MEDLINE abstracts, again maximizing throughput but this time maintaining accuracy. The fourth contribution is a corpus-driven entity disambiguation method that addresses OOKB entities. The method first captures the entities expressed in a corpus as latent representations that comprise in-KB and OOKB entities alike before performing entity disambiguation.

he extremely fast advances in wet-lab techniques lead to an exponential growth of heterogeneous and unstructured biological data, posing a great challenge to data integration in nowadays system biology. The traditional clustering approach, although widely used to divide the data into groups sharing common features, is less powerful in the analysis of heterogeneous data from n different sources (n _ 2). The co-clustering approach has been widely used for combined analyses of multiple networks to address the challenge of heterogeneity. In this thesis, novel methods for the co-clustering of large scale heterogeneous data sets are presented in the software package n-CluE: one exact algorithm and two heuristic algorithms based on the model of bi-/n-cluster editing by modeling the input as n-partite graphs and solving the clustering problem with various strategies. In the first part of the thesis, the complexity and the fixed-parameter tractability of the extended bicluster editing model with relaxed constraints are investigated, namely the ?-bicluster editing model and its NP-hardness is proven. Based on the results of this analysis, three strategies within the n-CluE software package are then established and discussed, together with the evaluations on performances and the systematic comparisons against other algorithms of the same type in solving bi-/n-cluster editing problem. To demonstrate the practical impact, three real-world analyses using n-CluE are performed, including (a) prediction of novel genotype-phenotype associations by clustering the data from Genome-Wide Association Studies; (b) comparison between n-CluE and eight other biclustering tools on GEO Omnibus microarray data sets; (c) drug repositioning predictions by co-clustering on drug, gene and disease networks. The outstanding performance of n-CluE in the real-world applications shows its strength and flexibility in integrating heterogeneous data and extracting biological relevant information in bioinformatic analyses.

Rule-based configuration systems are being successfully used in industry, such as DOPLER at Siemens. Those systems make complex domain knowledge available to users and let them derive valid, customized products out of large sets of components. However, maintenance of such systems remains a challenge. Formal models are a prerequisite for the use of automated methods of analysis. This thesis deals with the formalization of rule-based configuration. We develop two logics whose transition semantics are suited for expressing the way systems like DOPLER operate. This is due to the existence of two types of transitions, namely user and rule transitions, and a fixpoint mechanism that determines their dynamic relationship. The first logic, PIDL, models propositional systems, while the second logic, PIDL+, additionally considers arithmetic constraints. They allow the formulation and automated verification of relevant properties of rule- based configuration systems.

People are often a central element of visual scenes, particularly in real-world street scenes. Thus it has been a long-standing goal in Computer Vision to develop methods aiming at analyzing humans in visual data. Due to the complexity of real-world scenes, visual understanding of people remains challenging for machine perception. In this thesis we focus on advancing the techniques for people detection and tracking in crowded street scenes. We also propose new models for human pose estimation and motion segmentation in realistic images and videos. First, we propose detection models that are jointly trained to detect single person as well as pairs of people under varying degrees of occlusion. The learning algorithm of our joint detector facilitates a tight integration of tracking and detection, because it is designed to address common failure cases during tracking due to long-term inter-object occlusions. Second, we propose novel multi person tracking models that formulate tracking as a graph partitioning problem. Our models jointly cluster detection hypotheses in space and time, eliminating the need for a heuristic non-maximum suppression. Furthermore, for crowded scenes, our tracking model encodes long-range person re-identification information into the detection clustering process in a unified and rigorous manner. Third, we explore the visual tracking task in different granularity. We present a tracking model that simultaneously clusters object bounding boxes and pixel level trajectories over time. This approach provides a rich understanding of the motion of objects in the scene. Last, we extend our tracking model for the multi person pose estimation task. We introduce a joint subset partitioning and labelling model where we simultaneously estimate the poses of all the people in the scene. In summary, this thesis addresses a number of diverse tasks that aim to enable vision systems to analyze people in realistic images and videos. In particular, the thesis proposes several novel ideas and rigorous mathematical formulations, pushes the boundary of state-of-the-arts and results in superior performance.

Proof assistants are becoming widespread for formalization of theories both in computer science and mathematics. They provide rich logics with powerful type systems and machine-checked proofs which increase the confidence in the correctness in complicated and detailed proofs.
However, they incur a significant overhead compared to pen-and-paper proofs.
This thesis describes work on bridging the gap between high-order proof assistants and first-order automated theorem provers by extending the capabilities of the automated theorem provers to provide features usually found in proof assistants.
My first contribution is the development and implementation of a first-order superposition calculus with a polymorphic type system that supports type classes and the accompanying refutational completeness proof for that calculus. The inclusion of the type system into the superposition calculus and solvers completely removes the type encoding overhead when encoding problems from many proof assistants.
My second contribution is the development of SupInd, an extension of the typed superposition calculus that supports data types and structural induction over those data types. It includes heuristics that guide the induction and conjecture strengthening techniques, which can be applied independently of the underlying calculus.
I have implemented the contributions in a tool called Pirate. The evaluations of both contributions show promising results.

Skin has been proposed as a large, always-available, and easy to access input surface for mobile computing. However, it is fundamentally different than prior rigid devices: skin is elastic, highly curved, and provides tactile sensation. This thesis advances the understanding of skin as an input surface and contributes novel skin-worn devices and their interaction techniques. We present the findings from an elicitation study on how and where people interact on their skin. The findings show that participants use various body locations for on-skin interaction. Moreover, they show that skin allows for expressive interaction using multi-touch input and skin-specific modalities. We contribute three skin-worn device classes and their interaction techniques to enable expressive on-skin interactions: iSkin investigates multi-touch and pressure input on various body locations. SkinMarks supports touch, squeeze, and bend sensing with co-located visual output. The devices' conformality to skin enables interaction on highly challenging body locations. Finally, ExpressSkin investigates expressive interaction techniques using fluid combinations of high-resolution pressure, shear, and squeeze input. Taken together, this thesis contributes towards expressive on-skin interaction with multi-touch and skin-specific input modalities on various body locations.

Nowadays, access to digital information has become ubiquitous, while three-dimensional visual representation is becoming indispensable to knowledge understanding and information retrieval. Three-dimensional digitization plays a natural role in bridging connections between the real and virtual world, which prompt the huge demand for massive three-dimensional digital content. But reducing the effort required for three-dimensional modeling has been a practical problem, and long standing challenge in compute graphics and related fields.
In this thesis, we propose several techniques for lightening up the content creation process, which have the common theme of being structure-aware, \ie maintaining global relations among the parts of shape. We are especially interested in formulating our algorithms such that they make use of symmetry structures, because of their concise yet highly abstract principles are universally applicable to most regular patterns.
We introduce our work from three different aspects in this thesis. First, we characterized spaces of symmetry preserving deformations, and developed a method to explore this space in real-time, which significantly simplified the generation of symmetry preserving shape variants. Second, we empirically studied three-dimensional offset statistics, and developed a fully automatic retargeting application, which is based on verified sparsity. Finally, we made step forward in solving the approximate three-dimensional partial symmetry detection problem, using a novel co-occurrence analysis method, which could serve as the foundation to high-level applications.

A distributed hash table (DHT) is a peer-to-peer network that offers the function of a classic hash table, but where different key-value pairs are stored at different nodes on the network. Like a classic hash table, the main function provided by a DHT is key lookup, which retrieves the value stored at a given key.
Examples of DHT protocols include Chord, Pastry, Kademlia and Tapestry.
Such DHT protocols certain correctness and performance guarantees, but formal verification typically discovers border cases that violate those guarantees. In his PhD thesis, Tianxiang Lu reported correctness problems in published versions of Pastry and developed a model called LuPastry, for which he provided a partial proof of correct delivery of lookup messages assuming no node failure, mechanized in the TLA+ Proof System. In analyzing Lu's proof, I discovered that it contained unproven assumptions, and found counterexamples to several of these assumptions. The contribution of this thesis is threefold. First, I present LuPastry+, a revised TLA+ specification of LuPastry. Aside from needed bug fixes, LuPastry+ contains new definitions that make the specification more modular and significantly improve proof automation. Second, I present a complete TLA+ proof of correct delivery for LuPastry+. Third, I prove that the final step of the node join process of LuPastry/LuPastry+ is not necessary to achieve consistency. In particular, I develop a new specification with a simpler node join process, which I denote by Simplified LuPastry+, and prove correct delivery of lookup messages for this new specification. The proof of correctness of Simplified LuPastry+ is written by reusing the proof for LuPastry+, which represents a success story in proof reuse, especially for proofs of this size.
Each of the two proofs amounts to over 32,000 proof steps; to my knowledge, they are currently the largest proofs written in the TLA+ language, and---together with Lu's proof---the only examples of applying full theorem proving for the verification of DHT protocols

Post-desktop user interfaces, such as smartphones, tablets, interactive tabletops, public displays and mid-air interfaces, already are a ubiquitous part of everyday human life, or have the potential to be. One of the key features of these interfaces is the reduced number or even absence of input movement constraints imposed by a device form-factor. This freedom is advantageous for users, allowing them to interact with computers using more natural limb movements; however, it is a source of 4 issues for research and design of post-desktop interfaces which make traditional analysis methods inefficient: the new movement space is orders of magnitude larger than the one analyzed for traditional desktops; the existing knowledge on post-desktop input methods is sparse and sporadic; the movement space is non-uniform with respect to performance; and traditional methods are ineffective or inefficient in tackling physical ergonomics pitfalls in post-desktop interfaces. These issues lead to the research problem of efficient assessment, analysis and design methods for high-throughput ergonomic post-desktop interfaces.

To solve this research problem and support researchers and designers, this thesis proposes efficient experiment- and model-based assessment methods for post-desktop user interfaces. We achieve this through the following contributions:
- adopt optical motion capture and biomechanical simulation for HCI experiments as a versatile source of both performance and ergonomics data describing an input method;
- identify applicability limits of the method for a range of HCI tasks;
- validate the method outputs against ground truth recordings in typical HCI setting;
- demonstrate the added value of the method in analysis of performance and ergonomics of touchscreen devices; and
- summarize performance and ergonomics of a movement space through a clustering of physiological data.

The proposed method successfully deals with the 4 above-mentioned issues of post-desktop input. The efficiency of the methods makes it possible to effectively tackle the issue of large post-desktop movement spaces both at early design stages (through a generic model of a movement space) as well as at later design stages (through user studies). The method provides rich data on physical ergonomics (joint angles and moments, muscle forces and activations, energy expenditure and fatigue), making it possible to solve the issue of ergonomics pitfalls. Additionally, the method provides performance data (speed, accuracy and throughput) which can be related to the physiological data to solve the issue of non-uniformity of movement space. In our adaptation the method does not require experimenters to have specialized expertise, thus making it accessible to a wide range of researchers and designers and contributing towards the solution of the issue of post-desktop knowledge sparsity.

Markov automata constitute an expressive continuous-time compositional modelling formalism, featuring stochastic timing and nondeterministic as well as probabilistic branching, all supported in one model. They span as special cases, the models of discrete and continuous-time Markov chains, as well as interactive Markov chains and probabilistic automata. Moreover, they might be equipped with reward and resource structures in order to be used for analysing quantitative aspects of systems, like performance metrics, energy consumption, repair and maintenance costs. Due to their expressive nature, they serve as semantic backbones of engineering frameworks, control applications and safety critical systems. The Architecture Analysis and Design Language (AADL), Dynamic Fault Trees (DFT) and Generalised Stochastic Petri Nets (GSPN) are just some examples. Their expressiveness thus far prevents them from efficient analysis by stochastic solvers and probabilistic model checkers. A major problem context of this thesis lies in their analysis under some budget constraints, i.e. when only a finite budget of resources can be spent by the model.
We study mathematical foundations of Markov automata since these are essential for the analysis addressed in this thesis. This includes, in particular, understanding their measurability and establishing their probability measure. Furthermore, we address the analysis of Markov automata in the presence of both reward acquisition and resource consumption within a finite budget of resources. More specifically, we put the problem of computing the optimal expected resource-bounded reward in our focus. In our general setting, we support transient, instantaneous and final reward collection as well as transient resource consumption. Our general formulation of the problem encompasses in particular the optimal time-bound reward and reachability as well as resource-bounded reachability. We develop a sound theory together with a stable approximation scheme with a strict error bound to solve the problem in an efficient way. We report on an implementation of our approach in a supporting tool and also demonstrate its effectiveness and usability over an extensive collection of industrial and academic case studies.

Virtual and Augmented Reality applications typically rely on both stereoscopic presentation and involve intensive object and observer motion.
A combination of high dynamic range and stereoscopic capabilities become popular for consumer displays, and is a desirable functionality of head mounted displays to come.
The thesis is focused on complex interactions between all these visual cues on digital displays.

The first part investigates challenges of the stereoscopic 3D and motion combination.
We consider an interaction between the continuous motion presented as discrete frames.
Then, we discuss a disparity processing for accurate reproduction of objects moving in the depth direction.
Finally, we investigate the depth perception as a function of motion parallax and eye fixation changes by means of saccadic motion.

The second part focuses on the role of high dynamic range imaging for stereoscopic displays.
We go beyond the current display capabilities by considering the full perceivable luminance range and we simulate the real world experience in such adaptation conditions.
In particular, we address the problems of disparity retargeting across such wide luminance ranges and reflective/refractive surface rendering.

The core of our research methodology is perceptual modeling supported by our own experimental studies to overcome limitations of current display technologies and improve the viewer experience by enhancing perceived depth, reducing visual artifacts or improving viewing comfort.

Complex computer systems play an important role in every part of everyday life
and their correctness is often vital to human safety. In light of the recent
advances in the area of formal methods and the increasing availability and
maturity of tools and techniques, the use of verification techniques to show
that a system satisfies a specified property is about to become an integral
part of the development process. To minimize the development costs, formal
methods must be applied as early as possible, before the entire system is fully
developed, or even at the stage when only its specification is available. The
goal of synthesis is to automatically construct an implementation guaranteed to
fulfill the provided specification, and, if no implementation exists, to report
that the given requirements cannot be realized. When synthesizing an individual
component within a system and its external environment, the synthesis procedure
must take into account the component�s interface and deliver implementations
that comply with it. For example, what a component can observe about its
environment may be restricted by imprecise sensors or inaccessible
communication channels. In addition, sufficiently precise models of a
component�s environment are typically infinite-state, for example due to
modeling real time or unbounded communication buffers. This thesis presents
novel synthesis methods that respect the given interface limitations of the
synthesized system components and are applicable to infinite-state models. The
studied computational model is that of infinite-state two-player games under
incomplete information. The contributions are structured into three parts,
corresponding to a classification of such games according to the interface
between the synthesized component and its environment.
In the first part, we obtain decidability results for a class of game
structures where the player corresponding to the synthesized component has a
given finite set of possible observations and a finite set of possible actions.
A prominent type of systems for which the interface of a component naturally
defines a finite set of observations are Lossy Channel Systems. We provide
symbolic game solving and strategy synthesis algorithms for lossy channel games
under incomplete information with safety and reachability winning conditions.
Our second contribution is a counterexample-guided abstraction refinement
scheme for solving infinite-state under incomplete information in which the
actions available to the component are still finitely many, but no finite set
of possible observations is given. This situation is common, for example, in
the synthesis of mutex protocols or robot controllers. In this setting, the
observations correspond to observation predicates, which are logical formulas,
and their computation is an integral part of our synthesis procedure. The
resulting game solving method is applicable to games that are out of the scope
of other available techniques. Last we study systems in which, in addition to
the possibly infinite set of observation predicates, the component can choose
between infinitely many possible actions. Timed games under incomplete
information are a fundamental class of games for which this is the case. We
extend the abstraction-refinement procedure to develop the first systematic
method for the synthesis of observation predicates for timed control.
Automatically refining the set of candidate observations based on
counterexamples demonstrates better potential than brute-force enumeration of
observation sets, in particular for systems where fine granularity of the
observations is necessary.

Probabilistic databases store, query, and manage large amounts of uncertain information. This thesis advances the state-of-the-art in probabilistic databases in three different ways:
1. We present a closed and complete data model for temporal probabilistic databases and analyze its complexity. Queries are posed via temporal deduction rules which induce lineage formulas capturing both time and uncertainty.
2. We devise a methodology for computing the top-k most probable query answers. It is based on first-order lineage formulas representing sets of answer candidates. Theoretically derived probability bounds on these formulas enable pruning low-probability answers.
3. We introduce the problem of learning tuple probabilities which allows updating and cleaning of probabilistic databases. We study its complexity, characterize its solutions, cast it into an optimization problem, and devise an approximation algorithm based on stochastic gradient descent.
All of the above contributions support consistency constraints and are evaluated experimentally.

The idea of using multiple choices to improve allocation schemes is now well
understood and is often illustrated by the following example. Suppose n balls
are allocated to n bins with each ball choosing a bin independently and
uniformly at random. The \emphmaximum load}, or the number of balls in the
most loaded bin, will then be approximately \log n \over \log \log n with
high probability. Suppose now the balls are allocated sequentially by placing a
ball in the least loaded bin among the k≥ 2 bins chosen independently and
uniformly at random. Azar, Broder, Karlin, and Upfal showed that in this
scenario, the maximum load drops to {\log \log n \over \log k} +\Theta(1),
with high probability, which is an exponential improvement over the previous
case.
In this thesis we investigate multiple choice allocations from a slightly
different perspective. Instead of minimizing the maximum load, we fix the bin
capacities and focus on maximizing the number of balls that can be allocated
without overloading any bin. In the process that we consider we have m=\lfloor
cn \rfloor balls and n bins. Each ball chooses k bins independently and
uniformly at random. \emph{Is it possible to assign each ball to one of its
choices such that the no bin receives more than ℓ balls?} For all k≥ 3
and ℓ≥ 2 we give a critical value, c_{k,ℓ}^*, such that when
cc_{k,ℓ}^* this is not the case.
In case such an allocation exists, \emph{how quickly can we find it?} Previous
work on total allocation time for case k≥ 3 and ℓ=1 has analyzed a
\emph{breadth first strategy} which is shown to be linear only in expectation.
We give a simple and efficient algorithm which we also call \emph{local search
allocation}(LSA) to find an allocation for all k≥ 3 and ℓ=1. Provided
the number of balls are below (but arbitrarily close to) the theoretical
achievable load threshold, we give a \emph{linear bound for the total
allocation time that holds with high probability.
We demonstrate, through simulations, an order of magnitude improvement for
total and maximum allocation times when compared to the state of the art method.
Our results find applications in many areas including hashing, load balancing,
data management, orientability of random hypergraphs and maximum matchings in a
special class of bipartite graphs.

Ambiguity, complexity, and diversity in natural language textual expressions
are major hindrances to automated knowledge extraction. As a result
state-of-the-art methods for extracting entities and relationships from
unstructured data make incorrect extractions or produce noise. With the advent
of human computing, computationally hard tasks have been addressed through
human inputs. While text-based knowledge acquisition can benefit from this
approach, humans alone cannot bear the burden of extracting knowledge from the
vast textual resources that exist today. Even making payments for crowdsourced
acquisition can quickly become prohibitively expensive.
In this thesis we present principled methods that effectively garner human
computing inputs for improving the extraction of knowledge-base facts from
natural language texts. Our methods complement automatic extraction techniques
with human computing to reap the benefits of both while overcoming each other�s
limitations. We present the architecture and implementation of HIGGINS, a
system that combines an information extraction (IE) engine with a human
computing (HC) engine to produce high quality facts. The IE engine combines
statistics derived from large Web corpora with semantic resources like WordNet
and ConceptNet to construct a large dictionary of entity and relational
phrases. It employs specifically designed statistical language models for
phrase relatedness to come up with questions and relevant candidate answers
that are presented to human workers. Through extensive experiments we establish
the superiority of this approach in extracting relation-centric facts from
text. In our experiments we extract facts about fictitious characters in
narrative text, where the issues of diversity and complexity in expressing
relations are far more pronounced. Finally, we also demonstrate how interesting
human computing games can be designed for knowledge acquisition tasks.

There have been numerous efforts recently to digitize previously published
content and preserving born-digital content leading to the widespread growth of
large text repositories.
Web archives are such continuously growing text collections which contain
versions
of documents spanning over long time periods. Web archives present many
opportunities for historical, cultural and political analyses. Consequently
there is a growing need for tools which can efficiently access and search them.
In this work, we are interested in indexing methods for supporting text-search
workloads over web archives like time-travel queries and phrase queries. To
this end we make the following contributions:
Time-travel queries are keyword queries with a temporal predicate, e.g., mpii
saarland @ [06/2009], which return versions of documents in the past. We
introduce
a novel index organization strategy, called index sharding, for efficiently
supporting time-travel queries without incurring additional index-size blowup.
We also propose index-maintenance approaches which scale to such continuously
growing collections. We develop query-optimization techniques for time-travel
queries called partition selection which maximizes recall at any given
query-execution stage. We propose indexing methods to support phrase queries,
e.g., to be or not to be that is the question. We index multi-word sequences
and devise novel queryoptimization methods over the indexed sequences to
efficiently answer phrase queries. We demonstrate the superior performance of
our approaches over existing methods by extensive experimentation on real-world
web archives.

Rational cryptography has recently emerged as a very promising field of
research by combining notions and techniques from cryptography and game theory,
because it offers an alternative to the rather inexible traditional
cryptographic model. In contrast to the classical view of cryptography where
protocol participants are considered either honest or arbitrarily malicious,
rational cryptography models participants as rational players that try to
maximize their benefit and thus deviate from the protocol only if they gain an
advantage by doing so.
The main research goals for rational cryptography are the design of more ecient
protocols when players adhere to a rational model, the design and
implementation of automated proofs for rational security notions and the study
of the intrinsic connections between game theoretic and cryptographic notions.
In this thesis, we address all these issues.
First we present the mathematical model and the design for a new rational file
sharing protocol which we call RatFish. Next, we develop a general method for
automated verification for rational cryptographic protocols and we show how to
apply our technique in order to automatically derive the rational security
property for RatFish.
Finally, we study the intrinsic connections between game theory and
cryptography by defining a new game theoretic notion, which we call game
universal implementation, and by showing its equivalence with the notion of
weak stand-alone security.

In the context of image and video editing, this thesis proposes methods for
modifying the semantic content of a recorded scene. Two different editing
problems are approached: First, the removal of ghosting artifacts from high
dynamic range (HDR) images recovered from exposure sequences, and second, the
removal of objects from video sequences recorded with and without camera
motion. These editings need to be performed in a way that the result looks
plausible to humans, but without having to recover detailed models about the
content of the scene, e.g. its geometry, reflectance, or illumination. The
proposed editing methods add new key ingredients, such as camera noise models
and global optimization frameworks, that help achieving results that surpass
the capabilities of state-of-the-art methods. Using these ingredients, each
proposed method defines local visual properties that approximate well the
specific editing requirements of each task. These properties are then encoded
into a energy function that, when globally minimized, produces the required
editing results. The optimization of such energy functions corresponds to
Bayesian inference problems that are solved efficiently using graph cuts. The
proposed methods are demonstrated to outperform other state-of-the-art methods.
Furthermore, they are demonstrated to work well on complex real-world scenarios
that have not been previously addressed in the literature, i.e., highly
cluttered scenes for HDR deghosting, and highly dynamic scenes and
unconstrained camera motion for object removal from videos.

The processing of human motion data constitutes an important strand of research
with many applications
in computer animation, sport science and medicine. Currently, there exist
various systems for recording
human motion data that employ sensors of different modalities such as optical,
inertial and
depth sensors. Each of these sensor modalities have intrinsic advantages and
disadvantages that
make them suitable for capturing specific aspects of human motions as, for
example, the overall
course of a motion, the shape of the human body, or the kinematic properties of
motions.
In this thesis, we contribute with algorithms that exploit the respective
strengths of these
different modalities for comparing, classifying, and tracking human motion in
various scenarios.
First, we show how our proposed techniques can be employed, \textite.\,g.,
for real-time motion reconstruction
using efficient cross-modal retrieval techniques. Then, we discuss a practical
application of inertial sensors-based
features to the classification of trampoline motions. As a further contribution,
we elaborate on estimating the human body shape from depth data with
applications to personalized motion tracking.
Finally, we introduce methods to stabilize a depth tracker in challenging
situations such as in presence of occlusions.
Here, we exploit the availability of complementary inertial-based sensor
information.

Modern computers are not random access machines (RAMs).
They have a memory hierarchy, multiple cores, and a virtual memory.
We address the computational cost of the address translation in the virtual
memory and difficulties in design of parallel algorithms on modern many-core
machines.

Starting point for our work on virtual memory is the observation that the
analysis of some simple algorithms (random scan of an array, binary search,
heapsort) in either the RAM model or the EM model (external memory model) does
not correctly predict growth rates of actual running times.
We propose the VAT model (virtual address translation) to account for the cost
of address translations and analyze the algorithms mentioned above and others
in the model.
The predictions agree with the measurements.
We also analyze the VAT-cost of cache-oblivious algorithms.

In the second part of the paper we present a case study of the design of an
efficient 2D convex hull algorithm for GPUs.
The algorithm is based on \emphthe ultimate planar convex hull algorithm} of
Kirkpatrick and Seidel, and it has been referred to as \emph{the first
successful implementation of the QuickHull algorithm on the GPU by Gao et al.
in their 2012 paper on the 3D convex hull.
Our motivation for work on modern many-core machines is the general belief of
the engineering community that the theory does not produce applicable results,
and that the theoretical researchers are not aware of the difficulties that
arise while adapting algorithms for practical use.
We concentrate on showing how the high degree of parallelism available on GPUs
can be applied to problems that do not readily decompose into many independent
tasks.

This thesis describes the research results and contributions that have been
achieved during the author�s doctoral work. It is divided into two independent
parts, each of which is devoted to a particular research aspect.
The first part covers the true-to-detail creation of digital pieces of art,
so-called
relief sculptures, from given 3D models. The main goal is to limit the depth of
the
contained objects with respect to a certain perspective without compromising the
initial three-dimensional impression. Here, the preservation of significant
features
and especially their sharpness is crucial. Therefore, it is necessary to
overemphasize fine surface details to ensure their perceptibility in the more
complanate relief.Our developments are aimed at amending the flexibility and
user-friendliness during the generation process. The main focus is on providing
real-time solutions with intuitive usability that make it possible to create
precise, lifelike andaesthetic results. These goals are reached by a GPU
implementation, the use of efficient filtering techniques, and the replacement
of user defined parameters by adaptive values. Our methods are capable of
processing dynamic scenes and allow the generation of seamless artistic reliefs
which can be composed of multiple elements.
The second part addresses the analysis of repetitive structures, so-called
symmetries, within very large data sets. The automatic recognition of components
and their patterns is a complex correspondence problem which has numerous
applications ranging from information visualization over compression to
automatic
scene understanding. Recent algorithms reach their limits with a growing amount
of data, since their runtimes rise quadratically. Our aim is to make even
massive
data sets manageable. Therefore, it is necessary to abstract features and to
develop a suitable, low-dimensional descriptor which ensures an efficient,
robust, and purposive search. A simple inspection of the proximity within the
descriptor space helps to significantly reduce the number of necessary pairwise
comparisons. Our method scales quasi-linearly and allows a rapid analysis of
data sets which could not be handled by prior approaches because of their size.

Sentiments are positive and negative emotions, evaluations and stances. This
dissertation focuses on learning based systems for automatic analysis of
sentiments and comparisons in natural language text. The proposed approach
consists of three contributions:

1. Bag-of-opinions model: For predicting document-level polarity and intensity,
we proposed the bag-of-opinions model by modeling each document as a bag of
sentiments, which can explore the syntactic structures of sentiment-bearing
phrases for improved rating prediction of online reviews.
2. Multi-experts model: Due to the sparsity of manually-labeled training data,
we designed the multi-experts model for sentence-level analysis of sentiment
polarity and intensity by fully exploiting any available sentiment indicators,
such as phrase-level predictors and sentence similarity measures.
3. LSSVMrae model: To understand the sentiments regarding entities, we proposed
LSSVMrae model for extracting sentiments and comparisons of entities at both
sentence and subsentential level.

Different granularity of analysis leads to different model complexity, the
finer the more complex. All proposed models aim to minimize the use of
hand-labeled data by maximizing the use of the freely available resources.
These models explore also different feature representations to capture the
compositional semantics inherent in sentiment-bearing expressions. Our
experimental results on real-world data showed that all models significantly
outperform the state-of-the-art methods on the respective tasks.

The drastic increase in production of multimedia content has emphasized the
research concerning its organization and retrieval. In this thesis, we address
the problem of music retrieval when a set of images is given as input query,
i.e., the problem of soundtrack recommendation for images. The task at hand is
to recommend appropriate music to be played during the presentation of a given
set of query images. To tackle this problem, we formulate a hypothesis that the
knowledge appropriate for the task is contained in publicly available
contemporary movies. Our approach, Picasso, employs similarity search
techniques inside the image and music domains, harvesting movies to form a link
between the domains. To achieve a fair and unbiased comparison between
different soundtrack recommendation approaches, we proposed an evaluation
benchmark. The evaluation results are reported for Picasso and the baseline
approach, using the proposed benchmark. We further address two efficiency
aspects that arise from the Picasso approach. First, we investigate the problem
of processing top-K queries with set-defined selections and propose an index
structure that aims at minimizing the query answering latency. Second, we
address the problem of similarity search in high-dimensional spaces and propose
two enhancements to the Locality Sensitive Hashing (LSH) scheme. We also
investigate the prospects of a distributed similarity search algorithm based on
LSH using the MapReduce framework. Finally, we give an overview of the
PicasSound|a smartphone application based on the Picasso approach.

3D scanning technology has matured to a point where very large scale
acquisition of high resolution geometry has become feasible. However, having
large quantities of 3D data poses new technical challenges. Many applications
of practical use require an understanding of semantics of the acquired
geometry. Consequently scene understanding plays a key role for many
applications. This thesis is concerned with two core topics: 3D object
detection and semantic alignment. We address the problem of efficiently
detecting large quantities of objects in 3D scans according to object
categories learned from sparse user annotation. Objects are modeled by a
collection of smaller sub-parts and a graph structure representing part
dependencies. The thesis introduces two novel approaches: A part-based chain
structured Markov model and a general part-based full correlation model. Both
models come with efficient detection schemes which allow for interactive
run-times.

Knowledge bases are of great importance for Web search, recommendations, and
many Information Retrieval tasks. However, maintaining them for not so popular
entities is often a bottleneck. Typically, such entities have limited textual
coverage and only a few ontological facts. Moreover, these entities are not
well populated with multimodal data, such as images, videos, or audio
recordings.
The goals in this thesis are (1) to populate a given knowledge base with
multimodal data about entities, such as images or audio recordings, and (2) to
ease the task of maintaining and expanding the textual knowledge about a given
entity, by recommending valuable text excerpts to the contributors of knowledge
bases.
The thesis makes three main contributions. The first two contributions
concentrate on finding images of named entities with high precision, high
recall, and high visual diversity. Our main focus are less popular entities,
for which the image search engines fail to retrieve good results. Our methods
utilize background knowledge about the entity, such as ontological facts or a
short description, and a visual-based image similarity to rank and diversify a
set of candidate images.
Our third contribution is an approach for extracting text contents related to a
given entity. It leverages a language-model-based similarity between a short
description of the entity and the text sources, and solves a budget-constraint
optimization program without any assumptions on the text structure. Moreover,
our approach is also able to reliably extract entity related audio excerpts
from news podcasts. We derive the time boundaries from the usually very noisy
audio transcriptions.

\chapterAbstract}

To extend the traditional knowledge base with temporal dimension, this thesis
offers methods and tools for harvesting temporal facts from both
semi-structured and textual sources.
Our contributions are briefly summarized as follows.

\begin{enumerate}
\item{\bf Timely YAGO:} A temporal knowledge base called Timely YAGO
(T-YAGO) which extends YAGO with temporal attributes is built. We define a
simple RDF-style data model to support temporal knowledge.

\item{\bf PRAVDA:} To be able to harvest as many temporal facts from
free-text as possible, we develop a system PRAVDA.
It utilizes a graph-based semi-supervised learning algorithm to extract fact
observations, which are further cleaned up by an Integer Linear Program based
constraint solver.
We also attempt to harvest spatio-temporal facts to track a person's
trajectory.

\item{\bf PRAVDA-live:} A user-centric interactive knowledge harvesting
system, called PRAVDA-live, is developed for extracting facts from natural
language free-text. It is built on the framework of PRAVDA.
It supports fact extraction of user-defined relations from ad-hoc selected text
documents
and ready-to-use RDF exports.

\item{\bf T-URDF:} We present a simple and efficient representation
model for time-dependent uncertainty in combination with first-order
inference rules and recursive queries over RDF-like knowledge bases.
We adopt the common possible-worlds semantics known from probabilistic
databases and extend it towards histogram-like confidence distributions that
capture the validity of facts across time.


\end{enumerate

All of these components are fully implemented systems, which together form an
integrative architecture.
PRAVDA and PRAVDA-live aim at gathering new facts (particularly temporal facts),
and then T-URDF reconciles them.
Finally these facts are stored in a (temporal) knowledge base, called T-YAGO.
A SPARQL-like time-aware querying language, together with a visualization tool,
are designed for T-YAGO.
Temporal knowledge can also be applied for document summarization.

The US presidential race, the re-election of President Hugo Chavez, and the
economic crisis in Greece and other European countries are some of the
controversial topics being played on the news everyday. To understand the
landscape of opinions on political controversies, it would be helpful to know
which politician or other stakeholder takes which position - support or
opposition - on specific aspects of these topics. The work described in this
thesis aims to automatically derive a map of the opinions-people network from
news and other Web documents. The focus is on acquiring opinions held by
various stakeholders on politically controversial topics. This opinions-people
network serves as a knowledge-base of opinions in the form of hopinion holderi
hopinioni htopici triples.
Our system to build this knowledge-base makes use of online news sources in
order to extract opinions from text snippets. These sources come with a set of
unique challenges. For example, processing text snippets involves not just
identifying the topic and the opinion, but also attributing that opinion to a
specific
opinion holder. This requires making use of deep parsing and analyzing the
parse tree. Moreover, in order to ensure uniformity, both the topic as well the
opinion holder should be mapped to canonical strings, and the topics should
also be organized into a hierarchy. Our system relies on two main components:
i) acquiring opinions which uses a combination of techniques to extract opinions
from online news sources, and ii) organizing topics which crawls and extracts
debates from online sources, and organizes these debates in a hierarchy of
political controversial topics. We present systematic evaluations of the
different components of our system, and show their high accuracies. We also
present some of the different kinds of applications that require political
analysis. We present some
application requires political analysis such as identifying flip-floppers,
political
bias, and dissenters. Such applications can make use of the knowledge-base of
opinions.

n the presence of growing data, the need for efficient query processing under
result quality and index size control becomes more and more a challenge to
search engines. We show how to use proximity scores to make query processing
effective and efficient with focus on either of the optimization goals.
More precisely, we make the following contributions:
• We present a comprehensive comparative analysis of proximity score models and
a rigorous analysis of the potential of phrases and adapt a leading proximity
score model for XML data.
• We discuss the feasibility of all presented proximity score models for top-k
query processing and present a novel index combining a content and proximity
score that helps to accelerate top-k query processing and improves result
quality.
• We present a novel, distributed index tuning framework for term and term pair
index lists that optimizes pruning parameters by means of well-defined
optimization criteria under disk space constraints. Indexes can be tuned with
emphasis on efficiency or effectiveness: the resulting indexes yield fast
processing at high result quality.
• We show that pruned index lists processed with a merge join outperform top-k
query processing with unpruned lists at a high result quality.
• Moreover, we present a hybrid index structure for improved cold cache run
times.

Social tagging networks have become highly popular for publishing and searching
contents. Users in such networks can review, rate and comment on contents, or
annotate them with keywords (social tags) to give short but exact text
representations of even non-textual contents. In addition, there is an inherent
support for interactions and relationships among users. Thus, users naturally
form groups of friends or of common interests.

We address three research areas in our work utilising these intrinsic features
of social tagging networks.

(1) We investigate new approaches for exploiting the social knowledge of and
the relationships between users for searching and recommending relevant
contents, and integrate them in a comprehensive framework, coined SENSE, for
search in social tagging networks.

(2) To dynamically update precomputed lists of transitive friends in descending
order of their distance in user graphs of social tagging networks, we provide
an algorithm for incrementally solving the all pairs shortest distance problem
in large, disk-resident graphs and formally prove its correctness.

(3) Since users are content providers in social tagging networks, users may
keep their own data at independent, local peers that collaborate in a
distributed P2P network. We provide an algorithm for such systems to counter
cheating of peers in authority computations over social networks.

The viability of each solution is demonstrated by extensive experiments
regarding effectiveness and efficiency.

Web archives offer a rich and plentiful
source of information to researchers, analysts, and legal experts.
For this purpose, they gather Web sites as the sites change over time.
In order to keep up to high standards of data quality, Web archives have to
collect all versions of the Web sites.
Due to limited resuources and technical constraints this is not possible.
Therefore, Web archives consist of versions archived at various time points
without guarantee for mutual consistency.

This thesis presents a model for assessing the data quality in
Web archives as well as a family of crawling strategies yielding
high-quality captures. We distinguish between single-visit crawling strategies
for exploratory and visit-revisit crawling strategies for evidentiary purposes.
Single-visit strategies download every page
exactly once aiming for an ``undistorted'' capture of the ever-changing Web.
We express the quality of such the resulting capture with the ``blur'' quality
measure.
In contrast, visit-revisit strategies download every page twice. The initial
downloads of all pages form the visit phase of the crawling strategy.
The second downloads are grouped together in the revisit phase.
These two phases enable us to check which pages changed during the crawling
process.
Thus, we can identify the pages that are consistent with each other.
The quality of the visit-revisit captures is expressed by the ``coherence''
measure.

Quality-conscious strategies are based on predictions of the change behaviour of
individual pages. We model the Web site dynamics by Poisson processes
with page-specific change rates. Furthermore, we show that these rates can be
statistically predicted. Finally, we propose visualization techniques for
exploring
the quality of the resulting Web archives.

A fully functional prototype demonstrates the practical viability of our
approach.

Object class recognition is an active topic in computer vision still
presenting many challenges. In most approaches, this task is addressed
by supervised learning algorithms that need a large quantity of labels
to perform well. This leads either to small datasets (< 10,000 images)
that capture only a subset of the real-world class distribution (but
with a controlled and verified labeling procedure), or to large datasets
that are more representative but also add more label noise. Therefore,
semi-supervised learning is a promising direction. It requires only
few labels while simultaneously making use of the vast amount of
images available today. We address object class recognition with
semi-supervised learning. These algorithms depend on the underlying
structure given by the data, the image description, and the similarity
measure, and the quality of the labels. This insight leads to the
main research questions of this thesis: Is the structure given by
labeled and unlabeled data more important than the algorithm itself?
Can we improve this neighborhood structure by a better similarity
metric or with more representative unlabeled data? Is there a connection
between the quality of labels and the overall performance and how
can we get more representative labels? We answer all these questions,
i.e., we provide an extensive evaluation, we propose several graph
improvements, and we introduce a novel active learning framework
to get more representative labels.

The goal of music information retrieval (MIR) is to develop novel strategies
and techniques for organizing, exploring, accessing, and understanding music
data in an efficient manner.
The conversion of waveform-based audio data into semantically meaningful
feature representations by the use of digital signal processing techniques is
at the center of MIR and constitutes a difficult field of research because of
the complexity and diversity of music signals.
In this thesis, we introduce novel signal processing methods
that allow for extracting musically meaningful information from audio signals.
As main strategy, we exploit musical knowledge about the signals' properties to
derive feature representations that show a significant degree of robustness
against musical variations but still exhibit a high musical expressiveness. We
apply this general strategy to three different areas of MIR:
Firstly, we introduce novel techniques for extracting tempo and beat
information, where we particularly consider challenging music with changing
tempo and soft note onsets. Secondly, we present novel algorithms for the
automated segmentation and analysis of folk song field recordings, where one
has to cope with significant fluctuations in intonation and tempo as well as
recording artifacts. Thirdly, we explore a cross-version approach
to content-based music retrieval based on the query-by-example paradigm. In all
three areas, we focus on application scenarios where strong musical variations
make the extraction of musically meaningful information a challenging task.

This thesis presents a novel computational framework that allows for a robust
extraction and quantification of the Morse-Smale complex of a scalar field
given on a 2- or 3-dimensional manifold. The proposed framework is based on
Forman's discrete Morse theory, which guarantees the topological consistency of
the computed complex. Using a graph theoretical formulation of this theory, we
present an algorithmic library that computes the Morse-Smale complex
combinatorially with an optimal complexity of
O(n^2) and efficiently creates a multi-level representation of it. We explore
the discrete nature of this complex, and relate it to the smooth counterpart.
It is often necessary to estimate the feature strength of the individual
components of the Morse-Smale complex -- the critical points and separatrices.
To do so, we propose a novel output-sensitive strategy to compute the
persistence of the critical points. We also extend this wellfounded concept to
separatrices by introducing a novel measure of feature strength called
separatrix persistence. We evaluate the applicability of our methods in a wide
variety of application areas ranging from computer graphics to planetary
science to computer and electron tomography.

In this thesis we present two new type systems for verifying the security of
cryptographic protocol models expressed in a spi-calculus and, respectively, of
protocol implementations expressed in a concurrent lambda calculus. The two
type systems combine prior work on renement types with union and intersection
types and with the novel ability to reason statically about the disjointness of
types. The increased expressivity enables the analysis of important protocol
classes that were previously out of scope for the typebased analyses of
cryptographic protocols. In particular, our type systems can statically analyze
protocols that are based on zero-knowledge proofs, even in scenarios when
certain protocol participants are compromised. The analysis is scalable and
provides security proofs for an unbounded number of protocol executions. The
two type systems come
with mechanized proofs of correctness and efficient implementations.

The topic of this thesis is the analysis of the evolution of software
components. In order to track the evolution of software components, one needs
to collect the evolution information of each component. This information is
stored in the version control system (VCS) of the project�the repository of the
history of events happening throughout the project�s lifetime. By using
software archive mining techniques one can extract and leverage this
information.
The main contribution of this thesis is the introduction of evolution usage
trends
and evolution change patterns. The raw information about the occurrences of each
component is stored in the VCS of the project. By organizing it in evolution
trends and patterns, we are able to draw conclusions and issue recommendations
concerning each individual component and the project as a whole.
Evolution Trends An evolution trend is a way to track the evolution of a
software
component throughout the span of the project. The trend shows the increases
and decreases in the usage of a specific component, which can be indicative of
the quality of this component. AKTARI is a tool, presented in this thesis, that
is
based on such evolution trends and can be used by the software developers to
observe and draw conclusions about the behavior of their project.
Evolution Patterns An evolution pattern is a pattern of a frequently occurring
code
change throughout the span of the project. Those frequently occurring changes
are project-specific and are explanatory of the way the project evolves. Each
such evolution pattern contains in itself the specific way �things are done� in
the project and as such can serve for defect detection and defect prevention.
The technique of mining evolution patterns is implemented as a basis for the
LAMARCK tool, presented in this thesis.

quipping machines with knowledge, through the construction of machine-readable
knowledge bases, presents a key asset for semantic search, machine
translation, question answering, and other formidable challenges in
artificial intelligence. However, human knowledge predominantly resides
in books and other natural language text forms. This means that knowledge
bases must be extracted and synthesized from natural language text.
When the source of text is the Web, extraction methods must cope with
ambiguity, noise, scale, and updates.

The goal of this dissertation is to develop knowledge base population
methods that address the afore mentioned characteristics of Web text. The
dissertation makes three contributions. The first contribution is a method
for mining high-quality facts at scale, through distributed constraint reasoning
and a pattern representation model that is robust against noisy
patterns. The second contribution is a method for mining a large comprehensive
collection of relation types beyond those commonly found in
existing knowledge bases. The third contribution is a method for extracting
facts from dynamic Web sources such as news articles and social media
where one of the key challenges is the constant emergence of new entities.
All methods have been evaluated through experiments involving Web-scale
text collections.

This thesis provides a unifying view on the succinctness of systems: the
capability of a modeling formalism to describe the behavior of a system of
exponential size using a polynomial syntax.
The key theoretical contribution is the introduction of sequential circuit
machines as a new universal computation model that focuses on succinctness
as the central aspect. The thesis demonstrates that many well-known modeling
formalisms such as communicating state machines, linear-time temporal
logic, or timed automata exhibit an immediate connection to this
machine model. Once a (syntactic) connection is established, many complexity
bounds for structurally restricted sequential circuit machines can be
transferred to a certain formalism in a uniform manner. As a consequence,
besides a far-reaching unification of independent lines of research, we are
also able to provide matching complexity bounds for various analysis problems,
whose complexities were not known so far. For example, we establish
matching lower and upper bounds of the small witness problem and several
variants of the bounded synthesis problem for timed automata, a particularly
important succinct modeling formalism.
Also for timed automata, our complexity-theoretic analysis leads to the
identification of tractable fragments of the timed synthesis problem under
partial observability. Specifically, we identify timed controller synthesis
based on discrete or template-based controllers to be equivalent to model
checking. Based on this discovery, we develop a new model checking-based
algorithm to efficiently find feasible template instantiations.
From a more practical perspective, this thesis also studies the preservation
of succinctness in analysis algorithms using symbolic data structures.
While efficient techniques exist for specific forms of succinctness considered
in isolation, we present a general approach based on abstraction refinement
to combine off-the-shelf symbolic data structures. In particular, for handling
the combination of concurrency and quantitative timing behavior in
networks of timed automata, we report on the tool Synthia which combines
binary decision diagrams with difference bound matrices. In a comparison
with the timed model checker Uppaal and the timed game solver Uppaal-
Tiga running on standard benchmarks from the timed model checking and
synthesis domain, respectively, the experimental results clearly demonstrate
the effectiveness of our new approach.

Rendering methods based on ray tracing provide high image realism, but have
been historically regarded as offline only. This has changed in the past
decade, due to significant advances in the construction and traversal
performance of acceleration structures and the efficient use of data-parallel
processing. Today, all major graphics companies offer real-time ray tracing
solutions. The following work has contributed to this development with some key
insights.
We first address the limited support of dynamic scenes in previous work, by
proposing two new parallel-friendly construction algorithms for KD-trees and
BVHs. By approximating the cost function, we accelerate construction by up to
an order of magnitude (especially for BVHs), at the expense of only tiny
degradation to traversal performance.
For the static portions of the scene, we also address the topic of creating the
"perfect" acceleration structure. We develop a polynomial time non-greedy BVH
construction algorithm. We then modify it to produce a new type of acceleration
structure that inherits both the high performance of KD-trees and the small
size of
BVHs.
Finally, we focus on bringing real-time ray tracing to commodity desktop
computers.
We develop several new KD-tree and BVH traversal algorithms specically
tailored for the GPU. With them, we show for the first time that GPU ray tracing
is indeed feasible, and it can outperform CPU ray tracing by almost an order of
magnitude, even on large CAD models.

Correspondence problems like optic flow belong to the fundamental problems in
computer vision. Here, one aims at finding correspondences between the pixels
in two (or more) images. The correspondences are described by a displacement
vector field that is often found by minimising an energy (cost) function. In
this thesis, we present several contributions to the energy-based solution of
correspondence problems: (i) We start by developing a robust data term with a
high degree of invariance under illumination changes. Then, we design an
anisotropic smoothness term that works complementary to the data term, thereby
avoiding undesirable interference. Additionally, we propose a simple method for
determining the optimal balance between the two terms. (ii) When discretising
image derivatives that occur in our continuous models, we show that adopting
one-sided upwind discretisations from the field of hyperbolic differential
equations can be beneficial. To ensure a fast solution of the nonlinear system
of equations that arises when minimising the energy, we use the recent fast
explicit diffusion (FED) solver in an explicit gradient descent scheme. (iii)
Finally, we present a novel application of modern optic flow methods where we
align exposure series used in high dynamic range
(HDR) imaging. Furthermore, we show how the alignment information can be used in
a joint super-resolution and HDR method

The human immunodeficiency virus (HIV) is the causative agent of the acquired
immunodeficiency syndrome (AIDS) which claimed nearly $30$ million lives and is
arguably among the worst plagues in human history. With no cure or vaccine in
sight, HIV patients are treated by administration of combinations of
antiretroviral drugs. The very large number of such combinations makes the
manual search for an effective therapy practically impossible, especially in
advanced stages of the disease. Therapy selection can be supported by
statistical methods that predict the outcomes of candidate therapies. However,
these methods are based on clinical data sets that are biased in many ways. The
main sources of bias are the evolving trends of treating HIV patients, the
sparse, uneven therapy representation, the different treatment backgrounds of
the clinical samples and the differing abundances of the various
therapy-experience levels.

In this thesis we focus on the problem of devising bias-aware statistical
learning methods for HIV therapy screening -- predicting the effectiveness of
HIV combination therapies. For this purpose we develop five novel approaches
that when predicting outcomes of HIV therapies address the aforementioned
biases in the clinical data sets. Three of the approaches aim for good
prediction performance for every drug combination independent of its abundance
in the HIV clinical data set. To achieve this, they balance the sparse and
uneven therapy representation by using different routes of sharing common
knowledge among related therapies. The remaining two approaches additionally
account for the bias originating from the differing treatment histories of the
samples making up the HIV clinical data sets. For this purpose, both methods
predict the response of an HIV combination therapy by taking not only the most
recent (target) therapy but also available information from preceding therapies
into account. In this way they provide good predictions for advanced patients
in mid to late stages of HIV treatment, and for rare drug combinations.

All our methods use the time-oriented evaluation scenario, where models are
trained on data from the less recent past while their performance is evaluated
on data from the more recent past. This is the approach we adopt to account for
the evolving treatment trends in the HIV clinical practice and thus offer a
realistic model assessment.

Unsupervised machine learning aims to make predictions when labeled data is

absent, and thus, supervised machine learning cannot be applied. These

algorithms build on assumptions about how data and predictions relate to each

other. One technique for unsupervised problem settings are generative models,

which specify the set of assumptions as a probabilistic process that generates

the data.

The subject of this thesis is how to most effectively exploit input data that

has an underlying graph structure in unsupervised learning for three important

use cases. The first use case deals with localizing defective code regions in

software, given the execution graph of code lines and transitions. Citation

networks are exploited in the next use case to quantify the influence of

citations on the content of the citing publication. In the final use case,

shared tastes of friends in a social network are identified, enabling the

prediction of items from a user a particular friend of his would be interested

in.

For each use case, prediction performance is evaluated via held-out test data

that is only scarcely available in the domain. This comparison quantifies under

which circumstances each generative model best exploits the given graph

structure.

Nowadays we are rapidly moving from a mainly textual-based to a
multimedia-based Internet, for which the widely deployed IEEE 802.11 wireless
LANs can be one of the promising candidates to make them available to users
anywhere, anytime, on any device. However, it is still a challenge to support
group-oriented real-time multimedia services, such as video-on-demand, video
conferencing, distance educations, mobile entertainment services, interactive
games, etc., in wireless LANs, as the current protocols do not support
multicast, in particular they just send multicast packets in open-loop as
broadcast packets, i.e., without any possible acknowledgements or
retransmissions. In this thesis, we focus on MAC layer reliable multicast
approaches which outperform upper layer ones with both shorter delays and
higher efficiencies. Different from polling based approaches, which suffer from
long delays, low scalabilities and low efficiencies, we explore a feedback
jamming mechanism where negative acknowledgement (NACK) frames are allowed from
the non-leader receivers to destroy the acknowledgement (ACK) frame from the
single leader receiver and prompts retransmissions from the sender. Based on
the feedback jamming scheme, we propose two MAC layer multicast error
correction protocols, SEQ driven Leader Based Protocol (SEQ-LBP) and Hybrid
Leader Based Protocol (HLBP), the former is an Automatic Repeat reQuest (ARQ)
scheme while the later combines both ARQ and the packet level Forward Error
Correction (FEC). We evaluate the feedback jamming probabilities and the
performances of SEQ-LBP and HLBP based on theoretical analyses, NS-2
simulations and experiments on a real test-bed built with consumer wireless LAN
cards. Test results confirm the feasibility of the feedback jamming scheme and
the outstanding performances of the proposed protocols SEQ-LBP and HLBP, in
particular SEQ-LBP is good for small multicast groups due to its short delay,
effectiveness and simplicity while HLBP is better for large multicast groups
because of its high efficiency and high scalability with respect to the number
of receivers per group.

This thesis consists of two independent parts. In the rst part of the thesis,
we develop a Riemann-Roch theory for sublattices of the root lattice An
extending the work of Baker and Norine (Advances in Mathematics, 215(2):
766-788, 2007) and study questions that arise from this theory. Our theory is
based on the study of critical points of a certain simplicial distance function
on a lattice and establishes connections between the Riemann-Roch theory and
the Voronoi diagrams of lattices under certain simplicial distance functions.
In particular, we provide a new geometric approach for the study of the
Laplacian of graphs. As a consequence, we obtain a geometric proof of the
Riemann-Roch theorem for graphs and generalise the result to other sub-lattices
of An. Furthermore, we use the geometric approach to study the problem of
computing the rank of a divisor on a finite multigraph G to obtain an algorithm
that runs in polynomial time for a fiixed
number of vertices, in particular with running time 2O(n log n)poly(size(G))
where n is the number of vertices of G. Motivated by this theory, we study a
dimensionality reduction approach to the graph automorphism problem and we also
obtain an algorithm for the related problem of counting automorphisms of graphs
that is based on exponential sums.
In the second part of the thesis, we develop an approach, based on
complex-valued hash functions, to count cycles in graphs in the data streaming
model. Our algorithm is based on the idea of computing instances of
complex-valued random variables over the given stream and improves drastically
upon the nave sampling algorithm.

This thesis is mainly concerned with the hypergraph transversal problem, which
asks to generate all minimal transversals of a given hypergraph. While the
current best upper bound on the complexity of the problem is quasi-polynomial
in the combined input and output sizes, it is shown to be solvable in output
polynomial time for a number of hypergraph classes. We extend this polynomial
frontier to the hypergraphs induced by hyperplanes and constant-sided polytopes
in fixed dimension R^d and hypergraphs for which every minimal transversal and
hyperedge intersection is bounded. We also show the problem to be fixed
parameter tractable with respect to the minimum integer k such that the input
hypergraph is k-degenerate, and also with respect to its maximum complementary
degree. Whereas we improve the known bounds when the parameter is the maximum
degree of a hypergraph.

We also study the readability of a monotone Boolean function which is defined
as the minimum integer r such that it can be represented by an AND-OR-formula
with every variable occurrence is bounded by r. We prove that it is NP-hard to
approximate the readability of even a depth three Boolean formula. We also give
tight sublinear upper bounds on the readability of a monotone Boolean function
given in CNF (or DNF) form, parameterized by the number of terms in the CNF and
the maximum number of variables in the intersection of any constant number of
terms. For interval DNF's we give much tighter logarithmic bounds on the
readability.

Finally, we discuss an implementation of a quasi-polynomial algorithm for the
hypergraph transversal problem that runs in polynomial space. We found our
implementation to be competitive with all but one previous implementation on
various datasets.

The acquired immunodeficiency syndrome (AIDS) is one of the biggest medical
challenges in the world today. Its causative pathogen, the human
immunodeficiency virus (HIV), is responsible for millions of deaths per year.
Although about two dozen antiviral drugs are currently available, progression
of the disease can only be delayed but patients cannot be cured.
In recent years, the new class of coreceptor antagonists has been added to the
arsenal of antiretroviral drugs. These drugs block viral cell-entry by binding
to one of the receptors the virus requires for infection of a cell. However,
some HIV variants can also use another coreceptor so that coreceptor usage has
to be tested before administration of the drug.
This thesis analyzes the use of statistical learning methods to infer HIV
coreceptor usage from viral genotype. Improvements over existing methods are
achieved by using sequence information of so far not used genomic regions, next
generation sequencing technologies, and by combining different existing
prediction systems.
In addition, HIV coreceptor usage prediction is analyzed with respect to
clinical outcome in patients treated with coreceptor antagonists. The results
demonstrate that inferring HIV coreceptor usage from viral genotype can be
reliably used in daily routine.

Web archives include both archives of contents originally published on
the Web (e.g., the Internet Archive) but also archives of contents
published long ago that are now accessible on the Web (e.g., the
archive of The Times). Thanks to the increased awareness that web-born
contents are worth preserving and to improved digitization techniques,
web archives have grown in number and size. To unfold their full
potential, search techniques are needed that consider their inherent
special characteristics.

This work addresses three important problems toward this objective and
makes the following contributions:

* We present the Time-Travel Inverted indeX (TTIX) as an efficient
solution to time-travel text search in web archives, allowing users to
search only the parts of the web archive that existed at a user's time
of interest.

* To counter negative effects that terminology evolution has on the
quality of search results in web archives, we propose a novel
query-reformulation technique, so that old but highly relevant
documents are retrieved in response to today's queries.

* For temporal information needs, for which the user is best satisfied
by documents that refer to particular times, we describe a retrieval
model that integrates temporal expressions (e.g., ``in the 1990s'')
seamlessly into a language modeling approach.

Experiments for each of the proposed methods show their efficiency and
effectiveness, respectively, and demonstrate the viability of our
approach to search in web archives.

Generating realistic human shapes and motion is an important task both in the motion picture industry and in computer games. In feature films, high quality and believability are the most important characteristics. Additionally, when creating virtual doubles the generated charactes have to match as closely as possible to given real persons. In contrast, in computer games the level of realism does not need to be as high but real-time performance is essential. It is desirable to meet all these requirements with a general model of human pose and shape.
In addition, many markerless human tracking methods applied, e.g., in biomedicine or sports science can benefit greatly from the availability of such a model because most methods require a 3D model of the tracked subject as input, which can be generated on-the-fly given a suitable shape and pose model.
In this thesis, a comprehensive procedure is presented to generate different general models of human pose. A database of 3D scans spanning the space of human pose and shape variations is introduced. Then, four different approaches for transforming the database into a general model of human pose and shape are presented, which improve the current state of the art. Experiments are performed to evaluate and compare the
proposed models on real-world problems, i.e., characters are generated given semantic constraints and the underlying shape and pose of humans given 3D scans, multi-view video, or uncalibrated monocular images is estimated.

Computing global illumination (GI) in virtual scenes becomes increasingly

attractive even for real-time applications nowadays. GI delivers important cues

in the perception of 3D virtual scenes, which is important for material and

architectural design. Therefore, for photo-realistic rendering in the design

and even the game industry, GI has become indispensable. While the computer

simulation of realistic global lighting is well-studied and often considered as

solved, computing it efficiently is not. Saving computation costs is therefore

the main motivation of current research in GI. Efficient algorithms have to

take various aspects into account, such as the algorithmic complexity and

convergence, its mapping to parallel processing hardware, and the knowledge of

certain lighting properties including the capabilities of the human visual

system.

In this dissertation we exploit both low-level and high-level coherence in the

practical design of GI algorithms for a variety of target applications ranging

from high-quality production rendering to dynamic real-time rendering. We also

focus on automatic rendering-accuracy control to approximate GI in such a way

that the error is perceptually unified in the result images, thereby taking

not only into account the limitations of the human visual system but also later

video compression with an MPEG encoder. In addition, this dissertation provides

many ideas and supplementary material, which complements published work and

could be of practical relevance.

Superposition is an established decision procedure for a variety of first-order
logic theories represented by sets of clauses. A satisfiable theory, saturated
by superposition, implicitly defines a minimal Herbrand model for the theory.
This raises the question in how far superposition calculi can be employed for
reasoning about such minimal models. This is indeed often possible when
existential properties are considered. However, proving universal properties
directly leads to a modification of the minimal model's term-generated domain,
as new Skolem functions are introduced. For many applications, this is not
desired because it changes the problem.

In this thesis, I propose the first superposition calculus that can explicitly
represent existentially quantified variables and can thus compute with respect
to a given fixed domain. It does not eliminate existential variables by
Skolemization, but handles them using additional constraints with which each
clause is annotated. This calculus is sound and refutationally complete in the
limit for a fixed domain semantics. For saturated Horn theories and classes of
positive formulas, the calculus is even complete for proving properties of the
minimal model itself, going beyond the scope of known superposition-based
approaches.

The calculus is applicable to every set of clauses with equality and does not
rely on any syntactic restrictions of the input. Extensions of the calculus
lead to various new decision procedures for minimal model validity. A main
feature of these decision procedures is that even the validity of queries
containing one quantifier alternation can be decided. In particular, I prove
that the validity of any formula with at most one quantifier alternation is
decidable in models represented by a finite set of atoms and that the validity
of several classes of such formulas is decidable in models represented by
so-called disjunctions of implicit generalizations. Moreover, I show that the
decision of minimal model validity can be reduced to the superposition-based
decision of first-order validity for models of a class of predicative Horn
clauses where all function symbols are at most unary.

In this thesis, we cover three scenarios that violate common simplifying
assumptions about the nature of light transport.

We begin with the first ingredient to any 3D rendering: a geometry model. Most
3D scanners require the object-of-interest to show diffuse reflectance. The
further a material deviates from the Lambertian model, the more likely these
setups are to produce corrupted results. By placing a traditional laser
scanning setup in a participating (in particular, fluorescent) medium, we have
built a light sheet scanner that delivers robust results for a wide range of
materials, including glass.

Further investigating the phenomenon of fluorescence, we notice that, despite
its ubiquity, it has received moderate attention in computer graphics. In
particular, to date no data-driven reflectance models of fluorescent materials
have been available. To describe the wavelength-shifting reflectance of
fluorescent materials, we define the bispectral bidirectional reflectance and
reradiation distribution function (BRRDF), for which we introduce an
image-based measurement setup as well as an effcient acquisition scheme.

Finally, we envision a computer display that shows materials instead of
colours, and present a prototypical device that can exhibit anisotropic
reflectance distributions similar to com-
mon models in computer graphics.

Verification problems are often expressed in a language which mixes several
theories.
A natural question to ask is whether one can use decision procedures for
individual theories to construct a decision procedure for the union theory.
In the cases where this is possible one has a powerful method at hand to handle
complex theories effectively.

The setup considered in this thesis is that of one base theory which is
extended by one or more theories.
The question is if and when a given ground satisfiability problem in the
extended setting can be
effectively reduced to an equi-satisfiable
problem
over the base theory. A case where this reductive approach is always possible
is that of so-called \emph{local theory extensions.}

The theory of local extensions is developed and some applications concerning
monotone functions are given.
Then the theory of local theory extensions is generalized in order to deal with
data structures that
exhibit local behavior.
It will be shown that a suitable fragment of both the theory of arrays and the
theory of pointers
is local in this broader sense.
%
Finally, the case of more than one theory extension is discussed.
In particular, a \emph{modularity} result is given that under certain
circumstances the locality of each of the extensions
lifts to locality of the entire extension.

The reductive approach outlined above has become particularly relevant
in recent years due to the rise of powerful solvers for background
theories common in verification tasks. These so-called SMT-solvers
effectively handle theories such as real linear or integer arithmetic.
As part of this thesis, a program called \emph{\mbox{H-PILoT}} was
implemented which carries out reductive reasoning for local theory
extensions. H-PILoT found applications in mathematics, multiple-valued
logics, data-structures and reasoning in complex systems.

In this thesis we are studying three different problems that belong to the intersection of Game Theory and Computer Science. The first concerns the design of efficient protocols for a Contention Resolution

problem regarding selfish users who all need to transmit information over a common single–access

channel. We will provide efficient solutions for different variants of the problem, depending on the

feedback that the users can receive from the channel. The second problem concerns the Price of Stability

of a fair cost sharing Network Design problem for undirected graphs. We consider the general case for

which the best known upper bound is the Harmonic number Hn, where n is the number of players, and

the best known lower bound is 12=7 ~ 1:778. We improve the value of the previously best lower bound

to 42=23 ~ 1:8261. Furthermore, we study two and three players instances. Our upper bounds indicate

a separation between the Price of Stability on undirected graphs and that on directed graphs, where

Hn is tight. Previously, such a gap was only known for the cases where all players shared a terminal,

and for weighted players. Finally, the last problem applies Game Theory as an evaluation tool for a

computer system: we will employ the concept of Stochastic Stability from Evolutionary Game Theory

as a measure for the efficiency of different queue policies that can be employed at an Internet router.

Genome-wide sequencing projects of many different organisms produce large
numbers of sequences that are functionally characterized using experimental and
bioinformatics methods. Following the development of the first bio-ontologies,
knowledge of the functions of genes and proteins is increasingly made available
in a standardized format. This allows for devising approaches that directly
exploit functional information using semantic and functional similarity
measures. This thesis addresses different aspects of the development and
application of such similarity measures.

First, we analyze semantic and functional similarity measures and apply them for
investigating the functional space in different taxa. Second, a new software
program and a new database are described, which overcome limitations of existing
tools and simplify the utilization of similarity measures for different
applications.

Third, we delineate two applications of our functional similarity measures. We
utilize them for analyzing domain and protein interaction datasets and derive
thresholds for grouping predicted domain interactions into low- and
high-confidence subsets. We also present the new MedSim method for
prioritization of candidate disease genes, which is based on the observation
that genes and proteins contributing to similar diseases are functionally
related. We demonstrate that the MedSim method performs at least as well as more
complex state-of-the-art methods and significantly outperforms current methods
that also utilize functional annotation.

The creation of high quality animations of real-world human actors has long
been a challenging problem in computer graphics. It involves the modeling of
the shape of the virtual actors, creating their motion, and the reproduction of
very fine dynamic
details. In order to render the actor under arbitrary lighting, it is required
that reflectance properties are modeled for each point on the surface. These
steps, that are usually performed manually by professional modelers, are time
consuming and cumbersome.

In this thesis, we show that algorithmic solutions for some of the problems
that arise in the creation of high quality animation of real-world people are
possible using multi-view video data. First, we present a novel spatio-temporal
approach
to create a personalized avatar from multi-view video data of a moving person.
Thereafter, we propose two enhancements to a method that captures human shape,
motion and reflectance properties of amoving human using eightmulti-view video
streams. Afterwards we extend this work, and in order to add very fine dynamic
details to the geometric models, such as wrinkles and folds in the clothing, we
make use of the multi-view video recordings and present a statistical method
that can passively capture the fine-grain details of time-varying scene
geometry. Finally, in order to reconstruct structured shape and animation of
the subject from video, we present a dense 3D correspondence finding method
that enables spatiotemporally coherent reconstruction of surface animations
directly frommulti-view
video data.

These algorithmic solutions can be combined to constitute a complete animation
pipeline for acquisition, reconstruction and rendering of high quality virtual
actors from multi-view video data. They can also be used individually in a
system that require the solution of a specific algorithmic sub-problem. The
results demonstrate that using multi-view video data it is possible to find the
model description that enables realistic appearance of animated virtual actors
under different lighting
conditions and exhibits high quality dynamic details in the geometry.