High Dynamic Range Imaging and Perception Issues in Graphics

Tremendous progress in the development and accessibility of high dynamic range (HDR) technology that has happened just recently creates many interesting opportunities and challenges in graphics. There is a need of developing a complete pipeline for HDR image and video processing from acquisition, through compression and quality evaluation, to display. In this section we present our contributions to such a pipeline. In Section 0.1.1 we describe our pioneering work on the HDR video encoding. In Section 0.1.2 we outline extensions needed to upgrade advanced perception-inspired image fidelity metrics to handle HDR images. Finally, we discuss the problem of tone mapping for efficient image display on low dynamic range (LDR) devices in Section 0.1.3.

HDR Video Encoding

Investigators: Rafal Mantiuk, Grzegorz Krawczyk, and Karol Myszkowski

Due to rapid technological progress in HDR video capture and display [SHS⁺04], the efficient storage and transmission of such data is crucial for the completeness of any HDR imaging pipeline. We propose a new approach for inter-frame encoding of HDR video, which is embedded in the well-established MPEG-4 video compression standard [MKMS04]. The key component of our technique is luminance quantization that is optimized for the contrast threshold perception in the human visual system. The quantization scheme requires only 10-11 bits to encode 12 orders of magnitude of visible luminance range and does not lead to perceivable contouring artifacts. Besides video encoding, the proposed quantization provides perceptually-optimized luminance sampling for fast implementation of any global tone mapping operator (refer also to Section 0.1.3) using a lookup table. To improve the quality of synthetic video sequences, we introduce a coding scheme for discrete cosine transform (DCT) blocks with high contrast. We demonstrate the capabilities of HDR video in a player, which enables decoding, tone mapping, and applying post-processing effects in real-time. The tone mapping algorithm as well as its parameters can be changed interactively while the video is playing. Figure 0.1 shows our dynamic range data exploration tool that allows the user to view a selected range of luminance in rectangular windows displayed on top of the video. Also, we can simulate post-processing effects such as glare, night vision, and motion blur, which appear very realistic due to the usage of HDR data.

**Figure 0.1:** Sample frames from HDR video sequences: **(left)** the CAFETERIA sequence, dynamic range in the logarithmic scale [-1.9,3.6]. The background frame is clamped to a displayable range. Our dynamic range exploration tool, visible as two windows, shows a luminance range [-1.0,1.0] in the cafeteria interior and a high luminance range [1.0,3.0] outdoor. Details shown in these windows would not be visible in traditional LDR video. The source HDR panorama courtesy of Spheron, Inc. **(right)** the LIGHT sequence captured with a HDR video camera, dynamic range [0.3,4.9]. Details of the halogen bulb are well preserved despite high luminances. The visible range in the exploration tool window is [2.9,4.9].

HDR Visible Difference Predictor

Investigators: Rafal Mantiuk and Karol Myszkowski

When designing an image synthesis or processing application, it is desirable to measure the visual quality of the resulting images. To avoid tedious subjective tests, where a group of people has to assess the quality degradation, objective visual quality metrics can be used. The most successful objective metrics are based on models of the Human Visual System (HVS) and can predict such effects as a non-linear response to luminance, limited sensitivity to spatial and temporal frequencies, and visual masking [Dal93].

Most of the objective quality metrics have been designed to operate on images or video that are to be displayed on CRT or LCD displays. While this assumption seems to be clearly justified in case of LDR images, it poses problems as new applications that operate on HDR data become more common. A perceptual HDR quality metric could be used for the validation of HDR video encodings discussed in Section 0.1.1. Another application may involve steering the computation in a realistic image synthesis algorithm (refer to Section "Error Estimation and Advanced Sampling in Off-line Rendering"), where the amount of computation devoted to a particular region of the scene would depend on the visibility of potential artifacts.

We propose several modifications to Daly's Visual Difference Predicator (VDP) [Dal93]. The modifications significantly improve a prediction of perceivable differences in the full visible range of luminance [MMS04]. This extends the applicability of the original metric from a comparison of displayed images (compressed luminance) to a comparison of real word scenes of measured luminance (HDR images). The proposed metric does not rely on the global state of eye adaptation to luminance, but rather assumes local adaptation to each fragment of a scene. Such local adaptation is essential for a good prediction of contrast visibility in High-Dynamic Range (HDR) images, as a single HDR image can contain both dimly illuminated interior and strong sunlight. For such situations, the assumption of global adaptation to luminance does not hold. Figure 0.2 shows an example of the HDR VDP output in the form of the probability map of perceivable differences between a pair of HDR images.

In collaboration with Scott Daly from Sharp Laboratories of America we further improved our HDR VDP [MDMS05] by considering an optical transfer function which describes scattering of the light in the eye optics. Also, to calibrate our HDR VDP we conducted experiments using an advanced HDR display [SHS⁺04], capable of displaying the range of luminance that is close to that found in real scenes.

**Figure 0.2:** Visual Difference Predictor for High Dynamic Range Images (HDR VDP) takes as an input two images, a reference and a distorted image (upper and lower left), and produces as a result the probability of detection map (right). Red color on such a map denotes a high probability that the human observer will notice visible differences between two input images.

Tone Mapping

Investigators: Akiko Yoshida, Volker Blanz, Grzegorz Krawczyk, and Karol Myszkowski

A number of successful tone mapping operators for contrast compression have been proposed due to the need to visualize high dynamic range (HDR) images on LDR devices (refer to [DCWP02] for a recent survey). They were inspired by fields such diverse as image processing, photographic practice, and modeling of the human visual systems (HVS). The variety of approaches calls for a systematic perceptual evaluation of their performance.

We conducted a psychophysical experiment based on a direct comparison between the appearance of real-world scenes and HDR images of these scenes displayed on a LDR monitor [YBMS05]. In our experiment, HDR images were tone mapped by seven existing tone mapping operators. The primary interest of this psychophysical experiment was to assess the differences in how tone mapped images are perceived by human observers and to find out which attributes of image appearance account for these differences when tone mapped images are compared directly with their corresponding real-world scenes rather than with each other. The human subjects rated image naturalness, overall contrast, overall brightness, and detail reproduction in dark and bright image regions with respect to the corresponding real-world scene.

Also, we participated actively in developing new tone mapping operators. We introduced the adaptive logarithmic mapping method [DMAC03] which achieved the interactive performance and therefore was suitable for HDR video applications (refer to Section 0.1.1). Figure 0.3(left) shows an example image tone mapped using this method.

In more recent efforts we focused on the problem of lightness perception [KMMS04]. The key concept of our approach is to divide the scene into areas (frameworks) of consistent luminance and to map independently the source luminance in each framework relative to reference white. Frameworks are merged afterwards retaining the relation of perceived brightness between them. Additionally a perceptual effect of self luminosity, generally neglected in other methods, is taken into account. This approach produces results with realistic impression of brightness and natural colors in situations when modern tone mapping operators often fail as shown in Figure 0.3(right).

**Figure 0.3:** The TREE image from OpenEXR samples tone mapped using the adaptive logarithmic mapping method (left) and using the frameworks (right). The dark foreground in shadow and the sunny background results in high luminance contrasts in this image. Compression of such high contrasts using a global operator like the adaptive logarithmic mapping leads to the visible loss of texture details (see the grass and trees in the background) and to the high saturation of colors. Using the frameworks approach, texture details are well preserved and colors remain natural.

Bibliography

Dal93: S. Daly.
The Visible Differences Predictor: An algorithm for the assessment of image fidelity.
In A.B. Watson, editor, Digital Image and Human Vision, pages 179-206. Cambridge, MA: MIT Press, 1993.
DCWP02: K. Devlin, A. Chalmers, A. Wilkie, and W. Purgathofer.
Tone reproduction and physically based spectral rendering.
In Eurographics 2002: State of the Art Reports, pages 101-123. Eurographics, 2002.
DMAC03: Frederic Drago, Karol Myszkowski, Thomas Annen, and Norishige Chiba.
Adaptive logarithmic mapping for displaying high contrast scenes.
In Pere Brunet and Dieter W. Fellner, editors, Proc. of EUROGRAPHICS 2003, volume 22 of Computer Graphics Forum, pages 419-426, Granada, Spain, 2003. Blackwell.
KMMS04: Grzegorz Krawczyk, Rafal Mantiuk, Karol Myszkowski, and Hans-Peter Seidel.
Lightness perception inspired tone mapping.
In Heinrich Bülthoff and Holly Rushmeier, editors, Proceedings APGV 2004 : 1st Symposium on Applied Perception in Graphics and Visualization, pages 172-173, New York, USA, August 2004. ACM.
MDMS05: Rafal Mantiuk, Scott Daly, Karol Myszkowski, and Hans-Peter Seidel.
Predicting visible differences in high dynamic range images - model and its calibration.
In Human Vision and Electronic Imaging X, IS&T/SPIE's 17th Annual Symposium on Electronic Imaging (2005), volume 5666 of SPIE Proceedings Series, pages 000-012, San Jose, California USA, January 2005. SPIE.
MKMS04: Rafal Mantiuk, Grzegorz Krawczyk, Karol Myszkowski, and Hans-Peter Seidel.
Perception-motivated high dynamic range video encoding.
ACM Transactions on Graphics, 23(3):733-741, July 2004.
(Proc. ACM SIGGRAPH '04).
MMS04: Rafal Mantiuk, Karol Myszkowski, and Hans-Peter Seidel.
Visible difference predictor for high dynamic range images.
In Wil Thissen, Peter Wieringa, Maja Pantic, and Marcel Ludema, editors, 2004 IEEE International Conference on Systems, Man & Cybernetics. - Vol. 3, pages 2763-2769, Hague, Netherlands, October 2004. IEEE.
SHS⁺04: H. Seetzen, W. Heidrich, W. Stuerzlinger, G. Ward, L. Whitehead, M. Trentacoste, A. Ghosh, and A. Vorozcovs.
High dynamic range display systems.
ACM Transactions on Graphics, 23(3):757-765, 2004.
YBMS05: Akiko Yoshida, Volker Blanz, Karol Myszkowski, and Hans-Peter Seidel.
Perceptual evaluation of tone mapping operators with real-world sceness.
In Bernice Rogowitz, Thrasyvoulos Pappas, and Scott Daly, editors, Human Vision and Electronic Imaging X, IS&T/SPIE's 17th Annual Symposium on Electronic Imaging (2005), volume 5666 of SPIE Proceedings Series, San Jose, USA, January 2005. SPIE.