Mateusz Malinowski (PhD Student)

Computer Vision has undergone major changes over the recent five years. Here, we investigate if the performance of such architectures generalizes to more complex tasks that require a more holistic approach to scene comprehension. The presented work focuses on learning spatial and multi-modal representations, and the foundations of a Visual Turing Test, where the scene understanding is tested by a series of questions about its content. In our studies, we propose DAQUAR, the first ‘question answering about real-world images’ dataset together with methods, termed a symbolic-based and a neural-based visual question answering architectures, that address the problem. The symbolic-based method relies on a semantic parser, a database of visual facts, and a bayesian formulation that accounts for various interpretations of the visual scene. The neural-based method is an end-to-end architecture composed of a question encoder, image encoder, multimodal embedding, and answer decoder. This architecture has proven to be effective in capturing language-based biases. It also becomes the standard component of other visual question answering architectures. Along with the methods, we also investigate various evaluation metrics that embraces uncertainty in word's meaning, and various interpretations of the scene and the question.

We are addressing an open-ended question answering task

about real-world images. With the help of currently available methods

developed in Computer Vision and Natural Language Processing, we would

like to push an architecture with a global visual representation to its

limits. In our contribution, we show how to achieve competitive

performance on VQA with global visual features (Residual Net) together

with a carefully desgined architecture.

Boundary prediction in images as well as video has been a very active topic

of research and organizing visual information into boundaries and segments is

believed to be a corner stone of visual perception. While prior work has

focused on predicting boundaries for observed frames, our work aims at

predicting boundaries of future unobserved frames. This requires our model to

learn about the fate of boundaries and extrapolate motion patterns. We

experiment on established real-world video segmentation dataset, which provides

a testbed for this new task. We show for the first time spatio-temporal

boundary extrapolation in this challenging scenario. Furthermore, we show

long-term prediction of boundaries in situations where the motion is governed

by the laws of physics. We successfully predict boundaries in a billiard

scenario without any assumptions of a strong parametric model or any object

notion. We argue that our model has with minimalistic model assumptions derived

a notion of 'intuitive physics' that can be applied to novel scenes.

Together with the development of more accurate methods in Computer Vision and

Natural Language Understanding, holistic architectures that answer on questions

about the content of real-world images have emerged. In this tutorial, we build

a neural-based approach to answer questions about images. We base our tutorial

on two datasets: (mostly on) DAQUAR, and (a bit on) VQA. With small tweaks the

models that we present here can achieve a competitive performance on both

datasets, in fact, they are among the best methods that use a combination of

LSTM with a global, full frame CNN representation of an image. We hope that

after reading this tutorial, the reader will be able to use Deep Learning

frameworks, such as Keras and introduced Kraino, to build various architectures

that will lead to a further performance improvement on this challenging task.

Progress in language and image understanding by machines has sparkled the
interest of the research community in more open-ended, holistic tasks, and
refueled an old AI dream of building intelligent machines. We discuss a few
prominent challenges that characterize such holistic tasks and argue for
"question answering about images" as a particular appealing instance of such a
holistic task. In particular, we point out that it is a version of a Turing
Test that is likely to be more robust to over-interpretations and contrast it
with tasks like grounding and generation of descriptions. Finally, we discuss
tools to measure progress in this field.

As language and visual understanding by machines progresses rapidly, we are observing an increasing interest in holistic architectures that tightly interlink both modalities in a joint learning and inference process. This trend has allowed the community to progress towards more challenging and open tasks and refueled the hope at achieving the old AI dream of building machines that could pass a turing test in open domains. In order to steadily make progress towards this goal, we realize that quantifying performance becomes increasingly difficult. Therefore we ask how we can precisely define such challenges and how we can evaluate different algorithms on this open tasks? In this paper, we summarize and discuss such challenges as well as try to give answers where appropriate options are available in the literature. We exemplify some of the solutions on a recently presented dataset of question-answering task based on real-world indoor images that establishes a visual turing challenge. Finally, we argue despite the success of unique ground-truth annotation, we likely have to step away from carefully curated dataset and rather rely on ’}social consensus{’ as the main driving force to create suitable benchmarks. Providing coverage in this inherently ambiguous output space is an emerging challenge that we face in order to make quantifiable progress in this area.

Over the last two decades we have witnessed strong progress on modeling

visual object classes, scenes and attributes that have significantly

contributed to automated image understanding. On the other hand, surprisingly

little progress has been made on incorporating a spatial representation and

reasoning in the inference process. In this work, we propose a pooling

interpretation of spatial relations and show how it improves image retrieval

and annotations tasks involving spatial language. Due to the complexity of the

spatial language, we argue for a learning-based approach that acquires a

representation of spatial relations by learning parameters of the pooling

operator. We show improvements on previous work on two datasets and two

different tasks as well as provide additional insights on a new dataset with an

explicit focus on spatial relations.

From the early HMAX model to Spatial Pyramid Matching, spatial pooling

has played an important role in visual recognition pipelines. By

aggregating local statistics, it equips the recognition pipelines

with a certain degree of robustness to translation and deformation

yet preserving spatial information. Despite of its predominance in

current recognition systems, we have seen little progress to fully

adapt the pooling strategy to the task at hand. In this paper, we

propose a flexible parameterization of the spatial pooling step and

learn the pooling regions together with the classifier. We investigate

a smoothness regularization term that in conjuncture with an efficient

learning scheme makes learning scalable. Our framework can work with

both popular pooling operators: sum-pooling and max-pooling. Finally,

we show benefits of our approach for object recognition tasks based

on visual words and higher level event recognition tasks based on

object-bank features. In both cases, we improve over the hand-crafted

spatial pooling step showing the importance of its adaptation to

the task.

Biologically inspired, from the early HMAX model to Spatial Pyramid Matching,

pooling has played an important role in visual recognition pipelines. Spatial

pooling, by grouping of local codes, equips these methods with a certain degree

of robustness to translation and deformation yet preserving important spatial

information. Despite the predominance of this approach in current recognition

systems, we have seen little progress to fully adapt the pooling strategy to

the task at hand. This paper proposes a model for learning task dependent

pooling scheme -- including previously proposed hand-crafted pooling schemes as

a particular instantiation. In our work, we investigate the role of different

regularization terms showing that the smooth regularization term is crucial to

achieve strong performance using the presented architecture. Finally, we

propose an efficient and parallel method to train the model. Our experiments

show improved performance over hand-crafted pooling schemes on the CIFAR-10 and

CIFAR-100 datasets -- in particular improving the state-of-the-art to 56.29% on

the latter.