Hyperspectral imaging’s potential for transforming the imaging field
Hyperspectral imaging (HSI) is an analytical technique based on spectroscopy, which collects thousands of images at different wavelengths for the same spatial area. While the human eye has only three color receptors in the blue, green, and red, HSI measures the continuous spectrum of light for each pixel of the scene with fine wavelength resolution not only in the visible, but also in the near-infrared (NIR). The light striking each pixel is broken down into many different spectral bands to provide more information on what is imaged. The collected data form a 3D hyperspectral cube, in which two dimensions represent the spatial extent of the scene and the third its spectral content.
HSI is part of a class of techniques commonly referred to as spectral imaging or spectral analysis, which also includes multispectral imaging (MSI). The main difference between these two imaging systems is that HSI uses continuous and contiguous ranges of wavelengths (e.g., 400 to 1100 nm in steps of 1 nm), whereas MSI uses a subset of targeted wavelengths at chosen locations (e.g., 400 to 1100 nm in steps of 20 nm). MSI sensors usually have between 3 and 10 different band measurements in each pixel, while HSI images can contain thousands of narrower, but contiguous spectral bands. As a result, HSI sensors contain much more data than MSI sensors. MSI can be used to map forested areas, for example, while hyperspectral imagery can be used to map tree species within the forest.
Although MSI can be thought of as a reduced subset of HSI, the two technologies are complementary and choosing between them comes down to application requirements. In the visible channel, for example, MSI can be used to identify the contour of fruits and spot difficult-to-see bruises on fruits such as blueberries. But if the aim is to obtain a very detailed analysis of the fat content of avocados, this can only be done with HSI technology.
The algorithms and image processing methodologies behind both MSI and HSI date back to the 1970s and the launch of Landsat 1, the first satellite of the United States’ Landsat program, which was also the first satellite to carry a multispectral scanner. This was followed by the development of NASA’s Airborne Imaging Spectrometer (AIS) and AVIRIS in the mid-1980s when the term “hyperspectral imaging” was coined by Jerry Solomon, supervisor of the Imaging Analysis Systems Group at the California Institute of Technology.1
At that time, there were no microprocessors and image processing had to be carried out in large, centralized computer centers. Since then, and particularly within the last five years, as a result of developments in microprocessing and chip-based sensors, HSI technology has expanded from Earth observation into many fields such as agriculture, industrial sorting, medical research, food quality control, and environmental monitoring.
HSI applications
Each material possesses a specific spectral signature that can be used as a “fingerprint” for its unique identification. Therefore, HSI finds a wide range of applications, thanks to its noninvasive, label-free, and nondestructive capability in recognizing the components of matter. They include:
- Medical: for augmented-reality surgery and medical diagnostics
- Machine vision: for classifying virtually all materials—from food and minerals to textiles and plastics
- Remote sensing: for classifying agricultural soils, minerals, plastic, pollution, and surveillance
- Agriculture: to detect early symptoms of diseases, water stress, and soil quality for optimal control of the growth process
- Art and heritage analysis: to tell how, when, and where an artifact was originally made or subsequently altered
- Food quality control: to identify and quantify the chemical components of a food product, providing information such as nutrition, fat percentage, sugar level, and freshness
- Sorting waste: to sort and separate plastics, textiles, metals, glass, paper, cardboard, and hazardous materials such as asbestos
- Pharmaceutical: for quality assurance of pharmaceutical products’ chemical composition
- Mining and oil: for rapid and reliable mineral analysis in drill cores and other geological samples
- Forensics: gathering evidence from a crime scene, such as blood stains and gun powder residues
Commercialization of HSI systems
The global hyperspectral imaging market is expected to grow at a compound annual growth rate (CAGR) of 8.2% between 2020 and 2027, reaching $3.52 billion by 2027, according to a recent market report.2 Asia Pacific and Europe are expected to be the fastest-growing regions, with CAGRs of 9.3% and 8.5%, respectively.
The main drivers are an increasing demand for accurate and reliable analysis of materials, the development of new technologies, such as artificial intelligence combined with the development of low-cost imaging systems and the increasing demand for remote-sensing applications. The main applications will be for medical diagnosis, machine vision for industry, environmental monitoring, and military surveillance.
Types of HSI
Hyperspectral cameras can capture information to form a 3D data cube in five ways:
Whiskbroom (point scanning) captures one single pixel at a time. The image builds as the camera scans across the sample and contains all of its spectral information. While very time-consuming during image acquisition, the method results in a very high spectral resolution.
Push-broom (line scanning) measures continuous spectra, one line of pixels at a time, and is widely applied in industrial quality control monitoring processes. Its main limitation are the high losses caused by the entrance slit of the spectrometer.
Fourier transform (FT) spectroscopy is an alternative method for measuring continuous spectra. This approach combines a monochrome imaging sensor with an interferometer and has a higher light throughput than push-broom systems.
Spectral scanning gathers the entire spatial information for a given wavelength, one wavelength at a time. While fast per image, spectral scanning produces cubes slowly due to the time required to change wavelengths.
Hyperspectral snapshot cameras capture hyperspectral video. These are fast and ideal for imaging moving objects, although they typically offer limited spectral and spatial resolutions.
European fabricators of HSI systems
Specim (Oulu, Finland) is a global supplier of HSI solutions based on push-broom technology. Their SpecimONE platform includes the industry-proven Specim FX hyperspectral camera, SpecimINSIGHT software tool for creating the classification models, and SpecimCUBE processing unit to run the models in an inline system in real time. It is a complete, easy-to-use spectral imaging platform designed to make hyperspectral technology adaptation to industrial applications easier, faster, and cheaper than ever before (see Fig. 1). The Specim FX series are the first hyperspectral cameras designed explicitly for industrial machine vision covering wavelengths from visible and NIR to the thermal range (see Fig. 2). The Specim FX50 is the only hyperspectral camera available on the market, covering the full midwave-infrared (MWIR) spectral range (2.7–5.3 μm), for example, used for detecting black plastics.
With a performance-to-cost ratio that now meets industrial requirements, Specim’s hyperspectral cameras have applications in machine vision; e.g., for sorting hazardous waste materials such as asbestos, separating plastics, textiles, metals, glass, paper, cardboard, and other materials, as well as determining the sugar level in potatoes in the food industry.
Interuniversity Microelectronics Centre (imec; Leuven, Belgium) is an international research and development organization active in the fields of nanoelectronics and digital technologies. imec specializes in advanced technologies, including spectral imaging on-chip with a strong focus on semiconductor fab, equipment, and process technology. Traditional HSI cameras contain precision optics to select and diffract the light, which makes them relatively heavy, expensive, slow, and delicate—with a need for frequent recalibration. To address these problems, imec developed a unique CMOS process for spectral imaging in which thin-film spectral filters are deposited at wafer level directly on top of image sensor pixels (see Fig. 3). The filters eliminate the complex optical camera design and can be deposited on a commercial CMOS imager, like the one in a smartphone or a model with scientific imaging capabilities.
imec’s manufacturing process enables volume production of sensors at a reasonable cost and the thin-film filter designs can be tuned for specific applications. This on-chip spectral imaging technology exists in different “flavors,” depending on the pattern of the spectral filters. For example, snapshot spectral sensors consist of a Bayer-like mosaic pattern on top of an image sensor, with spectral narrow-band filters per pixel, enabling video-rate spectral imaging ideally suited for scenes with lots of movement or for monitoring dynamic processes. The linescan hyperspectral sensors comprise a striped pattern on top of each row of pixels, enabling high-resolution HSI. Comparable to line-scanning push-broom technology, imec’s technology provides a more compact, more robust, and easier-to-use form since the two-dimensional (2D) image is split into color channels directly on the chip. Applications include medicine, machine vision, remote sensing, agriculture, art and heritage analysis, and forensics.
NIREOS (Milan, Italy) created HERA, a novel hyperspectral camera based on the FT approach, which uses interference of light rather than dispersion to measure spectra (see Fig. 4). Hyperspectral systems based on spatial or spectral scanning (e.g., push-broom scanners or filter-based cameras) contain a slit and a grating, which limits the amount of light reaching the detector. In contrast, HERA features a large clear aperture to provide very high light throughput, high sensitivity, and a variable spectral resolution. It can be used even in low-light illumination conditions, for instance, to record faint fluorescence spectra emitted by different regions of a sample.Its camera operates in static mode, without the need to add cumbersome translation stages to move either the camera or the sample, and its large depth of focus allows for HSI of 3D objects and landscapes. HERA is an easy-to-use, point-and-shoot camera, suitable for transmission, reflectance, and fluorescence measurements in various applications, ranging from material sorting and food inspection, cultural heritage, environmental monitoring, and remote sensing.
HSI is seldom used for microscopy applications, but this is an area Lambda-X (Nivelles, Belgium) is exploring. For a former space project, Lambda-X developed an original HSI camera to perform the chemical analysis of combustion events when space objects were re-entering the atmosphere. For this purpose, they developed a dedicated HSI architecture capable of measuring scenes in constrained environments, especially vibrations.
This innovative HSI system is based on a common-beam interferometer architecture and a FT interferometer operating in the visible and NIR range (VNIR; 400–950 nm). The camera operates with diffraction limit optics that make it compatible with microscopic applications. At the same time, it provides high enough spectral resolution, down to 8 cm-1 or 0.13 nm at 400 nm, to address advanced fluorescent imaging applications as well as Raman imaging applications. In this context, Lambda-X recently demonstrated the usability of such a high-resolution imager to distinguish melamine from milk powder grains in the food sector. This technology can be used for any other applications demanding both high spectral and high spatial resolution such as remote atmospheric gas analyses or remote vegetation fluorescence detection.
HySpex, a division of Norsk Elektro Optikk (Oslo, Norway), offers three lines of push-broom hyperspectral imagers in the VNIR and shortwave-infrared (SWIR) for airborne, laboratory, unmanned aerial vehicle (UAV), field, and industrial uses (see Fig. 5). Their Mjolnir line of cameras is specifically designed with a small form factor and low mass to collect scientific-grade data from UAVs. For example, a Mjolnir-equipped drone can fly over an open pit mine and rapidly map chemical, physical, and mineralogical characteristics. And their Baldur line of industrial cameras have applications in the food industry for quality control, foreign material detection, sorting, and classification. The Baldur line gives a spectral resolution of two spectral bands and captures 4X as much light as conventional systems, enabling the automatic detection of parasitic nematodes in cod fillets to cut costs, improve food safety, and reduce waste. The newest HySpex model, the VS-1200, is a high-resolution instrument designed for airborne applications. With a 40° field of vision, the camera is ideal for mapping large areas with high accuracy and resolution.
In 2022, HySpex designed a new stratospheric VNIR and SWIR system, which was bought by NASA for their first airborne campaign conducted on the ER-2 manned aircraft in September 2022.
Remaining challenges
Although HSI produces much greater resolution than MSI systems, it has disadvantages of cost and complexity. Hyperspectral cubes are large, multidimensional datasets, and analyzing hyperspectral data requires fast processing, sensitive detectors, and large data storage capacities. These factors greatly increase the cost and complexity of obtaining and processing hyperspectral data. In space applications, for example, a major problem is how to program hyperspectral satellites to sort through data on their own and transmit only the important images, since transmission and storage of the large amount of data can be difficult and costly.
Similarly, with industrial applications, the decision to use local or remote image storage and data output can impact the speed of the application, which must keep pace with production. Other problem areas include determining proper wavelengths, selecting the right camera with suitable spectral and spatial resolution, and choosing lighting with adequate intensity at key wavelengths. And because HSI generally requires a high level of physical understanding, the software needs to be as intuitive as possible and support easy establishment of material classifications.
To address these issues, developments in HSI technology to date have the common aim of reducing complexity and making HSI industrial applications easier, faster, and cheaper to use than previously possible. These include a CMOS process to eliminate complex optical camera design; a complete, easy-to-use spectral imaging platform comprising a camera, software tool, and processing unit; and a camera based on the FT approach that enables high light throughput and can be used in low-light illumination conditions.
During the next few years, as HSI technology becomes less complex and less expensive, it will inevitably move into mainstream imaging applications because it is the only technology with sufficient spectral and spatial resolution that can satisfy the increasing demand for accurate and reliable analysis of materials in a wide variety of sectors. The expansion of HSI applications into new industries such as agriculture, environmental monitoring, and biomedicine will be next. With increasing demand for more precise and accurate material analysis, HSI technology is poised to play a significant role in these markets. Furthermore, the growth of remote sensing and UAVs is expected to drive the demand for HSI technology in environmental monitoring and mapping applications. As HSI technology continues to evolve, it has the potential to revolutionize the way we understand and analyze materials, offering new insights and opportunities for a wide range of industries.
HSI is a tool with the potential to make a significant positive impact on our lives and our planet. Whether it’s being used to promote sustainability, improve public health, or increase safety and security, this cutting-edge technology is helping us see the world in new and exciting ways and inspiring us to tackle some of the biggest challenges facing humanity.
REFERENCES
1. A. F. H. Goetz, G. Vane, J. E. Solomon, and B. N. Rock, Science, 228, 4704, 1147–1153 (Jun. 7, 1985); doi:10.1126/science.228.4704.1147.
2. See www.researchandmarkets.com/reports/4535750/global-hyperspectral-imaging-market-growth.
Jérémy Picot-Clémente
Jérémy Picot-Clémente is photonics technologies manager for the European Photonics Industry Consortium (EPIC), and is based in Auvergne-Rhône-Alpes, France.