Digital Imaging: Plenoptic camera refocuses digitally

A digital handheld camera developed by researchers at Stanford University looks to all the world like an ordinary consumer camera, but has capabilities that are anything but ordinary.¹ For example, the image produced by the camera (or, more precisely, by its associated digital processor) can be reprocessed at any time to move the plane of best focus forward and back, as if refocusing the lens. More startling, the image can be processed in a slightly different way to create sharp focus everywhere-in effect, extending the depth of field way beyond what one would expect for the aperture stop (f-stop) of the lens.

The camera is a modification of the so-called “plenoptic” camera invented by John Wang and Edward Adelson at the Massachusetts Institute of technology (MIT; Cambridge, MA) in 1992.² In a plenoptic camera, an ordinary camera lens creates an image in the usual way, but rather than placing a sensor array exactly at the image plane, a microlens array is placed there instead, with each microlens becoming an image pixel. The actual image sensor-which must have a much higher pixel count than the microlens array-is placed a short distance behind the microlens array; each microlens in the array images the exit pupil of the camera lens onto a unique sensor area consisting of, perhaps, a few hundred sensor pixels.

Although the number of lenses in the microlens array defines the number of pixels in the final image, the small exit-pupil image each microlens creates on the sensor array carries a multitude of data points on that single image pixel. In fact, each of the hundreds of sensor pixels (the data points) devoted to a given image pixel carries information on a different perspective for that image pixel, and thus for the pixel’s equivalent at the object plane (which is the portion of the exterior scene on which the camera lens is focused).

Wang and Adelson intended the plenoptic camera to be a device that would capture a single image, but allow the viewer to select a subset of sensor pixels (through digital processing) that would allow the viewer to alter the perspective of the image, and thus peek around 3-D objects.

The Stanford version

The plenoptic camera created by the Stanford group differs in two ways from the original version. First, its optical path is simplified, allowing it to be integrated into an ordinary (though high-pixel-count) digital still camera. Second, additional processing approaches have been developed to allow digital refocusing and digital depth-of-focus enhancement (see figure). As with the original version, the Stanford camera allows shifts in perspective (an ability that works best for macro photography, in which the camera lens is close to a small object and relatively large in size in relation to it).

The Stanford camera includes a 4000 × 4000-pixel sensor array with a 9-µm pitch, a 296 × 296-microlens array with lenslets of 500-µm focal length and 125‑µm diameter, and either a 140-mm f/2.8 or an 80-mm f/2.0 camera lens, both with adjustable stops. The software produces a 4-D light field (two lateral dimensions arising from the image plane and two from the pupil plane). Any vignetting is normalized by dividing each image-pixel output by the fraction of rays intercepted by the sensor light field for that image pixel.

With a camera lens set to an f/4 stop, for example, the technique leads to focal depths akin to those of higher f-stops. “With N×N sensor pixels under each microlens, you can theoretically refocus from an f/4 (say) to an f/(4N),” says Ren Ng, one of the researchers. “In our case, we find empirically that you can focus from f/4 to f/22. The f/4 version collects all the light coming through the f/4 aperture, which is about 36 times more than an f/22 aperture.”

Data sets captured by the plenoptic camera could be used for entertainment; a viewer could sit at a computer and change image perspectives, focal depths, and focal positions at will. These data may also have more-serious uses-for example, allowing the capture of a transient phenomenon at large focal depth and from simultaneous differing perspectives. Ng notes that the Stanford group has printed a computer-generated hologram from one of the data sets.

REFERENCES

1. R. Ng et al., Tech. Rep. CSTR 2005-02, Stanford Computer Science; graphics.stanford.edu/papers/lfcamera.

2. T. Adelson and J.Y.A. Wang, IEEE Trans. Pattern Analysis and Machine Intell. 14(2) (February 1992).