Coded aperture based light field acquisition

by Alexander Reuter


LCD in front of a lens.

Setting

The goal of this project was the acquisition of a light field based on an coded aperture. To achieve an artificial aperture I use an LC display in front of the lens. This LC display has usually set all pixels black except of a defined constant pattern which let the incoming light pass. The light field is the result of different pattern positions over the LCD area depending on the final light field image position. Incoming light rays pass the artificial aperture and the lens aperture successively. In an ideal setting artificial aperture and lens aperture are next to each other. Due to simplicity the LCD is put on the surface of the lens. Different aperture positions (of the LCD) lead to filtering of different incoming ray angles. The final result in the image plane is a parallax movement depending on the artificial aperture position. This is basically the light field. To obtain a final light field by using this setting I needed to regard different steps which I want to describe closer below. In the end I will circumscribe some future work which will get implemented the next weeks or in more disjunct projects.



Degenerating aspects



Soon after using the naive setting described above I realized that there are several aspects which degenerates or even inhibits the final light field acquisition. One has to keep an eye on these aspects in order to avoid a bad point of departure. These aspects are:

It is possible to reduce the amount of degeneration of each aspect in a reasonable way. I will describe ways of doing that in the next chapters.


Type of LC display

Using an LC display to block incoming light sounds very straightforward. A perfect display would approach an infinite contrast ratio, would pass and block every spectral light depending on the state of a corresponding pixel and would never change the spectrum of incident light. However real LC D's won't match these requirements. For example TFT panels used in desktop monitors do not fit well to this setting since the pixel mask is too narrow and leads to diffraction. We want to block the incoming light rays without any change of their direction. But there exist also other LC displays: Commonly used in cars or automatons. There exist three important categories: STN and FSTN/TSTN and DSTN. STN displays do not offer brightness contrast but color contrast. More precisely: The contrast is not achieved by passing and blocking the complete light, but passing green and yellow light. Green and yellow can simulate good contrast ratios in a subjective way for humans but aren't usable for this setting. DSTN displays result in real brightness contrast by using a second STN layer but have poor contrast ratios. FSTN displays reduce the complexity. They use just one STN layer but use additional filters to get rid of distortions. They achieve contrast ratios up to 18:1. In this setting I used an FSTN display.


Drawback: Black pixels let incoming light pass.



Handling low contrast ratios of LCD



A contrast ratio of 18:1 as stated before are upper limits and usually laboratory based values. The model used in this project achieves usually contrast ratios between 10:1 and 12:1. These ratios even depend on the spectrum of light which leads to color stitches in the final image. For instance this display produces a kind of red color stitch, which means, that the contrast ratio of red light is worse compared to other spectral light. This results in worse noise behavior and worse dynamic behavior in red areas. For example when having normal illuminated red areas the final light field image will be noisy there. On the other hand when having specular lights these bright areas may tilt over and the final color will be cyan. So the red channel will indicate the limits of illumination. A way to come over the bad noise behavior is to take multiple images and average over them.

Another problem is that “black” pixels still let pass incoming light in a meaningful way. This leads to a final image which still has too much information defined by the aperture of the lens. To come over with this problem one just has to subtract an image which was taken with all pixels set to black from the image that was taken with a window of “white” pixels.

Position of LCD

To obtain the light field the distance between the lens and the observed object should be as small as possible. The smaller the distance the more meaningful the parallax effect which is the basis for the light field. The depth of field is usually near the observed object. So if the LCD gets closer to the object it will positioned into focus which basically reduces the parallax effect. One can see this in the round black areas at the corners of the images. To get a perfect out of focus aperture one may be interested in opening the lens and position the LCD near the lens aperture.


Artificial aperture underlies same principles as lens aperture. Goal: Avoid noise and small DoF





Color shifting


Image obtained with all pixels set to black. Due to low contrast one can see the object clearly.

Due to color shifts produced by the LCD and different white points of light sources one is interested in a mapping to real object colors. To achieve this one may use a color checker with known spectral behavior of each checker area. These areas are designed to fill a wide field of spectral colors. First one have to take a light field image of the color checker. This image contains the acquired colors C'. We have the colors of the acquired colors and the real areas spectral colors which can translated to different RGB colors C by different color spaces. I used the sRGB color space. Now one want to find a matrix which maps from a given acquired color C' to an object color C. I assumed a 4 times 4 matrix M. Now we have following linear system: C = C' * M. Then we redistribute C' and M to obtain a matrix D containing the acquired colors and a vector N containing the desired matrix elements. This final over estimated linear system of equations can be solved and provides the final matrix which can be used to map every acquired pixel value to ideal object values.




Negative pixels

Since the whole setting has several imperfections as described before, it is usual to get some negative pixel values. The ratio between highest positive pixel value and the lowest negative pixel value depends highly on different exposure times and noise. Mostly dark areas and especially the focus based round shapes at the corners provide a high amount of these negative pixels. But also very bright pixels may explode the bounds, given to the previous mapping algorithm ( remember: One had to choose a color space. The resulting RGB values lie within a range. This range does not necessary hold for the mapped colors. ). To handle these wide distributed colors I decided to normalize them. To avoid fluctuating image brightnesses over the light field images I normalize them globally by finding the brightest and the darkest pixel value over all final light field images and normalize over them.






Image obtained with a window in LCD. Due to the bad "black" image (above) light passing the black pixels reduces the quality.


Common color checker





Environmental light

The acquisition of a light field will take very much time: Taking 5 duplicate light field images for removing noise. Each composed by a “black” and a “windowed” image. Generating a 7x7 light field results in a photo session of 5*2*49 images, so 490 images. Over this amount of time sunlight may change significantly. This is the reason for taking an image with all pixels set to black each duplicate image. But even then: A cloud can move between sun and object between the black and windowed image. This results in significantly artifacts. The best way to handle this situation is to use artificial light in office or photo laboratories.


Final light field image after subtracting, mapping, noise reduction and normalizing.




Final light field video






Vibration

Assuming the camera on a stative on wooden floor taking photos of objects in Millimeter range. Every vibration of the floor, for instance by walking around, gets amplified by the stative. When taking photos and such a vibration occurs, one will find black and white artifacts at the edges of the observed object immediately. So one will build this setting in a room with solid basement and tries to avoid the presence of people in the corresponding room.

Future work:

There are several important points not handled yet:

The LCD just produces squared windows. Using different windows will offer great advantages: Squares will lead to ringing artifact, which is not the case for circles so using them would increase the final quality. Letters can show interesting disc of confusion behaviors. Finally one can use apertures which are not zero in Fourier domain. Having zeros there is one of the biggest problems when deblurring images.

Changing from PFM files to EXR files will lead to less memory usage. This step would increase the speed due to less bandwidth usage additionally.

Enhancing light field viewer with aperture selection and focus selection will offer the user very interesting aspects of light field observation.

Finding an LCD with a higher contrast ratio or using multiple displays successively would increase the quality of the final light field images significantly.

One has to take an image from the color checker orthogonal to the image boundaries and reset the start coordinate as like as the offset coordinates in the program “sub” manually. One goal is to automate this step.