Publications - Last Year

Although considerable progress has been made in semantic scene understanding
under clear weather, it is still a tough problem under adverse weather
conditions, such as dense fog, due to the uncertainty caused by imperfect
observations. Besides, difficulties in collecting and labeling foggy images
hinder the progress of this field. Considering the success in semantic scene
understanding under clear weather, we think it is reasonable to transfer
knowledge learned from clear images to the foggy domain. As such, the problem
becomes to bridge the domain gap between clear images and foggy images. Unlike
previous methods that mainly focus on closing the domain gap caused by fog --
defogging the foggy images or fogging the clear images, we propose to alleviate
the domain gap by considering fog influence and style variation simultaneously.
The motivation is based on our finding that the style-related gap and the
fog-related gap can be divided and closed respectively, by adding an
intermediate domain. Thus, we propose a new pipeline to cumulatively adapt
style, fog and the dual-factor (style and fog). Specifically, we devise a
unified framework to disentangle the style factor and the fog factor
separately, and then the dual-factor from images in different domains.
Furthermore, we collaborate the disentanglement of three factors with a novel
cumulative loss to thoroughly disentangle these three factors. Our method
achieves the state-of-the-art performance on three benchmarks and shows
generalization ability in rainy and snowy scenes.

Eye-tracking technology is an integral component of new display devices such
as virtual and augmented reality headsets. Applications of gaze information
range from new interaction techniques exploiting eye patterns to
gaze-contingent digital content creation. However, system latency is still a
significant issue in many of these applications because it breaks the
synchronization between the current and measured gaze positions. Consequently,
it may lead to unwanted visual artifacts and degradation of user experience. In
this work, we focus on foveated rendering applications where the quality of an
image is reduced towards the periphery for computational savings. In foveated
rendering, the presence of latency leads to delayed updates to the rendered
frame, making the quality degradation visible to the user. To address this
issue and to combat system latency, recent work proposes to use saccade landing
position prediction to extrapolate the gaze information from delayed
eye-tracking samples. While the benefits of such a strategy have already been
demonstrated, the solutions range from simple and efficient ones, which make
several assumptions about the saccadic eye movements, to more complex and
costly ones, which use machine learning techniques. Yet, it is unclear to what
extent the prediction can benefit from accounting for additional factors. This
paper presents a series of experiments investigating the importance of
different factors for saccades prediction in common virtual and augmented
reality applications. In particular, we investigate the effects of saccade
orientation in 3D space and smooth pursuit eye-motion (SPEM) and how their
influence compares to the variability across users. We also present a simple
yet efficient correction method that adapts the existing saccade prediction
methods to handle these factors without performing extensive data collection.

Video frame interpolation (VFI) enables many important applications that
might involve the temporal domain, such as slow motion playback, or the spatial
domain, such as stop motion sequences. We are focusing on the former task,
where one of the key challenges is handling high dynamic range (HDR) scenes in
the presence of complex motion. To this end, we explore possible advantages of
dual-exposure sensors that readily provide sharp short and blurry long
exposures that are spatially registered and whose ends are temporally aligned.
This way, motion blur registers temporally continuous information on the scene
motion that, combined with the sharp reference, enables more precise motion
sampling within a single camera shot. We demonstrate that this facilitates a
more complex motion reconstruction in the VFI task, as well as HDR frame
reconstruction that so far has been considered only for the originally captured
frames, not in-between interpolated frames. We design a neural network trained
in these tasks that clearly outperforms existing solutions. We also propose a
metric for scene motion complexity that provides important insights into the
performance of VFI methods at the test time.

Resolving morphological chemical phase transformations at the nanoscale is of
vital importance to many scientific and industrial applications across various
disciplines. The TXM-XANES imaging technique, by combining full field
transmission X-ray microscopy (TXM) and X-ray absorption near edge structure
(XANES), has been an emerging tool which operates by acquiring a series of
microscopy images with multi-energy X-rays and fitting to obtain the chemical
map. Its capability, however, is limited by the poor signal-to-noise ratios due
to the system errors and low exposure illuminations for fast acquisition. In
this work, by exploiting the intrinsic properties and subspace modeling of the
TXM-XANES imaging data, we introduce a simple and robust denoising approach to
improve the image quality, which enables fast and high-sensitivity chemical
imaging. Extensive experiments on both synthetic and real datasets demonstrate
the superior performance of the proposed method.

Most in-the-wild images are stored in Low Dynamic Range (LDR) form, serving
as a partial observation of the High Dynamic Range (HDR) visual world. Despite
limited dynamic range, these LDR images are often captured with different
exposures, implicitly containing information about the underlying HDR image
distribution. Inspired by this intuition, in this work we present, to the best
of our knowledge, the first method for learning a generative model of HDR
images from in-the-wild LDR image collections in a fully unsupervised manner.
The key idea is to train a generative adversarial network (GAN) to generate HDR
images which, when projected to LDR under various exposures, are
indistinguishable from real LDR images. The projection from HDR to LDR is
achieved via a camera model that captures the stochasticity in exposure and
camera response function. Experiments show that our method GlowGAN can
synthesize photorealistic HDR images in many challenging cases such as
landscapes, lightning, or windows, where previous supervised generative models
produce overexposed images. We further demonstrate the new application of
unsupervised inverse tone mapping (ITM) enabled by GlowGAN. Our ITM method does
not need HDR images or paired multi-exposure images for training, yet it
reconstructs more plausible information for overexposed regions than
state-of-the-art supervised learning models trained on such data.