Night Vision: Bio-inspired night vision algorithm improves dim-light video

Nocturnal insects have had millions of years to fine-tune their ability to distinguish minute movements and even colors and patterns in dark or dimly lit environments. By understanding the subtle neural and optical visual adaptations of these creatures, researchers at Lund University (Sweden) have developed a biomimetic image-processing algorithm that dramatically improves the quality of dim-light video.¹

Nocturnal vision

In addition to the photon shot noise (or quantum fluctuations that limit achievable signal-to-noise ratio when photon absorption rates are low), transducer noise and dark noise are also a factor. Transducer noise appears when photoreceptors cannot produce—as a result of biochemical processes involved in signal amplification—an equivalent electrical response for each absorbed photon, while dark noise occurs when thermal activations are incorrectly interpreted as photon-absorption events. (There can be up to 360 dark events per hour at room temperature for some nocturnal toads, although the rate is much lower in nocturnal insects.)

Despite their extreme low-light environment, nocturnal insects can distinguish colors and faint movements to avoid obstacles when flying, travel using the faint polarization patterns of Moon glow, and even navigate by the light from the Milky Way. To do this, research on a type of Central American nocturnal bee shows that several neural strategies are used by these insects to reduce noise when photons are absorbed by the photoreceptors within their eyes, including (1) special optics within the eyes that improve light collection; (2) enhancement of the neural image created in the retina; and (3) an optimized filtering of images in space and time at a level higher than the retina within the visual system.

With the first of these adaptations physically met by the compound eyes of nocturnal insects, the second and third adaptations were specifically exploited by the research team to develop an algorithm that improves dim-light video imaging.

Neural mechanisms and high-level processing

Beyond conventional video noise-reduction algorithms that use spatiotemporal weighted averaging, motion-compensating or smoothing techniques, structure-adaptive anisotropic filtering, or high-dynamic-range imaging, the Lund researchers apply a standard amplifying intensity transformation to a low-intensity input video sequence, calculate a structure tensor with its eigenvalues and eigenvectors, construct a summation kernel that removes the edges of an object within the field of view, and integrate the output intensity using tone mapping (wherein spatial and temporal filtering attenuates and smooths detail).

Essentially, the algorithm first amplifies the signal (and noise) in the dim image sequence in much the same manner that nocturnal photoreceptors do. To get rid of the noise but retain (and strengthen) the enhanced signal, the algorithm next spatially sums groups of neighboring pixels (to create larger but brighter pixels) and temporally sums the signals of each over time. The balance between the extents of summation in space and time is set locally by the speed of local image details—the greater the speed, the less the algorithm is tipped in favor of temporal summation.

In addition, because this summation occurs separately for each color channel, color information is retained. And by using a lateral inhibition routine, edges are enhanced. Summation and lateral inhibition are both hallmarks of vision in nocturnal animals. Finally, the magnitude of both the amplification and the summation is adjusted automatically according to light level, adapting (like an eye) to the visual environment.

“The algorithm works remarkably well,” says Eric Warrant, professor of Zoology at Lund University (see figure). “We envisage that it can be used in a number of applications, from medical imaging to surveillance, or to simply improve the quality of video films captured by consumer cameras in dim light.”

REFERENCE
1. Eric Warrant et al., Proc. IEEE Bioinspired Imaging special issue 102, 10, 1411–1426 (Oct. 2014).