Hyperspectral imaging captures spatial and spectral data of the human landscape

Red, green, blue (RGB)—these three colors underpin our ability to perceive and navigate our day-to-day lives on Earth. Advances in imaging and satellite technology over decades have enabled deeper, richer, and more complex visualizations of our planet’s landscape beyond the RGB, in bands and layers. Some of the same technologies help researchers, surgeons, and clinicians see more deeply into the landscapes of our bodies.

Nearly 50 years ago, a NASA/USGS collaborative launched Landsat 1, a four-band, multispectral scanner system designed by Virginia Norwood. Its mission was to gather visible/near-infrared (VIS-NIR) digital data records of Earth from space. Since then, satellites have been continuously collecting spectral images. Thanks to multispectral imaging (MSI) and mature remote sensing capabilities, including hyperspectral imaging (HSI), five decades of spatial and spectral data are available in a vast, public domain archive.¹

Between 250 and 2500 nm, HSI also captures spatial and spectral data of our bodies. With sensors and software that capture reflected, absorbed, and emitted radiance, HSI turns each pixel into not just four, five, or six layers of data, but four, five, or six hundred layers.

Unfortunately, there is no NASA for the human body, no single lab with a mission and budget to gather reference-quality 3D data—for quantitative analysis or modeling—of our “beyond-the-visible” anatomy and physiology. The look and behavior of our healthy and diseased organs, tissues, fluids, and cells in the UV, IR, NIR, and shortwave-infrared (SWIR) can be patient-specific and difficult to catalog.

So, those who rely on imaging for health-related problem-solving have additional problems to solve; not only are they seeking answers to other high-stakes diseases, but they may also have to create their own imaging tools and high-quality spectral libraries to do so.

Additionally, teams must determine how to manage the data they gather. A single, unprocessed 3D hypercube file, which contains one spectral and two spatial dimensions, can exceed hundreds of gigabytes. For data manipulation and storage, 4-terabyte (TB) computers are the norm. And, teams must learn how to denoise, classify, detect, and cluster patterns, and extract essential information from the mountain of information that hypercubes generate, which necessitates multiple algorithms, and—more recently—deep learning and neural networks.

The sophistication of “hyperspectral medicine” might deter some, but software companies and university-private sector collaborations are taking on the challenge.

Managing the data

One software company, HSpeQ (St. Louis, MO), that spun off from Washington University in St. Louis in 2019, is making it easier for those with limited coding experience to acquire, classify, and display their 3D data sets.

CEO and founder Mikhail Berezin, an associate professor of radiology at the university, licensed the software—initially funded by the National Science Foundation (NSF)—from the university. Then, in 2020, he and his team began offering IDCube Lite, a free, downloadable introductory software for processing datacubes (see Fig. 1). It has been downloaded 400 times.

The software has, according to Berezin, an easy-to-use, intuitive interface that can be used by anyone familiar with basic imaging principles. It enables users to develop cancer’s optical signature and then use it for automatic detection of precancerous tissue, tumor staging, and monitoring the efficacy of treatment. Additionally, it features many HSI algorithms, including those based on machine learning and AI, which rapidly help detect and identify lesions, tissue abnormality, or aberrant cells (see Fig. 2).

FIGURE 2. Hyperspectral datacube of a fatty liver biopsy under a 50x microscope: a 3D view of biopsy demonstrating the various slices in a datacube (left) and hyperspectral datacube analysis using endmembers algorithm (right) are shown. This datacube was obtained by members of the Berezin Lab at Washington University in St. Louis.

The software can be used across wavelengths, from the UV to SWIR and beyond, and can store and support libraries of images of different formats. It can also analyze up to 10 image sets simultaneously.

Middleton Spectral Vision (Middleton, WI), Specim (Oulu, Finland), Arion Optics (Toronto, ON, Canada), and CytoViva (Auburn, AL) are a few of the companies that manufacture HSI instrumentation supported by HSPeQ’s software.

Studying the surface: skin

Research on skin’s reflectance, transmittance, and absorptive properties dates back at least to the last century.² More recently, researchers at the University of Las Palmas de Gran Canaria in Spain have facilitated the detection and classification of pigmented skin lesions using snapshot HS cameras within the VIS-NIR (400–1000 nm), “demonstrating its potential as a noninvasive imaging modality for in situ clinical support during routine practice.”³

However, there is still work to be done before HSI can be successfully integrated into effective, noninvasive, and contactless bedside instrumentation.

In one use case, Rajagopal Srinivasan, a clinical assistant professor at the University of Maryland School of Medicine (Baltimore, MD), attempted to study rashes due to COVID-19, or “COVID toes,” using a portable proprietary device. He found it to be compact and lightweight, but because it required (gentle) contact with the surface to be imaged, it raised issues of disinfection before and after each use as well as increased risks to patient and operator from physical contact.

The device scanned from 400 to 1000 nm in 2 nm intervals, and the wavelength resolution (full-width at half-maximum) was approximately 4 nm. Each scan took about half a minute. He needed to 3D-print a stabilizing brace because the slightest movements in the z-axis during the scan—inevitable with living, breathing human beings and soft tissue surfaces such as skin—resulted in blurring, loss of focus, and misregistration of image features between slices in the hypercube.

While Srinivasan is enthusiastic about HSI in medical application, interpreting and generalizing data is affected by lack of calibration, lack of reproducibility, and lack of reference-quality databases with curated examples of spectral data in health and disease.

Beneath the skin, noninvasively

Angela Belcher is determined to use imaging for diagnosis and to help resolve the problems of detecting late-stage cancers, particularly ovarian. She imagines systems analogous to (but less invasive than) mammograms, that can discover tumors at Stage 1 or 2, or possibly even sooner.

As the James Mason Crafts Professor of Biological Engineering and Materials Science and Engineering at the Massachusetts Institute of Technology (MIT; Cambridge, MA) and head of the biological engineering department, Belcher informs her imaging work with a background in evolving organisms, electronics, batteries, solar cells, and advanced materials. As a deep collaborator, she leads a team at MIT (The Belcher Lab) and is also an active member of the Koch Institute for Integrative Cancer Research.

Several years ago, Belcher realized she would have to build her own imaging technologies to see what she wanted to see. So, she has cut a two-pronged path into systems design.

One novel system is called DOLPHIN (Detection of Optically Luminescent Probes using Hyperspectral and diffuse Imaging in Near-infrared). It combines HSI with hyperdiffuse NIR-II (NIR-II: 950–1650 nm) to see millimeter-sized tumors and can detect signal at an unprecedented 7–9 cm beneath the skin.⁴ Most tumors discovered with CT, ultrasound, or other imaging technology are about 1 cm (see Fig. 3).

FIGURE 3. DOLPHIN system hyperspectral imaging of the “MIT” motif (left) and scattering radius obtained from hyperdiffuse imaging of the same motif (right).

The second, called SWIFTI (short-wave infrared fluorescence tri-band imager), developed in collaboration with surgeons and oncologists at Massachusetts General Hospital and engineers from MIT Lincoln Lab, is an image guided, real-time surgical support system that helps surgeons locate and remove very small tumors (0.2 mm), a process called “debulking.” It relies on fluorescence-emitting nanoparticle probes (see Fig. 4) consisting of single-walled carbon nanotubes coated with a peptide that binds to a protein overexpressed by ovarian cancer cells.⁵

FIGURE 4. SWIFTI system schematic of real-time near-infrared (NIR) fluorescence image-guided surgery for tumor debulking in ovarian cancer patients.

Over the next few years, Belcher will be studying lesions and precursors. Because 80% of ovarian cancer arises from fallopian tubes, they plan to adapt DOLPHIN technology to work on early lesions there.

Within the brain

For decades, biopsy has been a primary method for brain cancer detection. Whether needle, stereotactic (using MRI), or open, each technique is time-consuming, invasive, and expensive. Needle biopsies require drilling, and open biopsies are risky and require long recovery time. Moreover, once the brain tissue is exposed during surgery, slight shifting may occur, which is problematic because brain cancer tissue closely resembles surrounding healthy tissue.

When the only means of differentiating between the two is the human eye, a surgeon may unknowingly inflict damage. Hyperspectral systems, which typically operate in VIS-NIR (400–1000 nm) or NIR (900–1700 nm), offer the potential for more precise, less invasive cancer detection.

Noninvasive disease detection is a strategic initiative for Headwall Photonics (Bolton, MA), which offers researchers and surgeons aberration-corrected sensor systems (for data acquisition) and software (for specific detection algorithms). Given the range and diverse nature of disease states, the spectral assessment usually requires collaboration with researchers who can assist Headwall with the spectral data exploitation for different diseases states.

For example, Headwall collaborated with a seven-team European imaging consortium, the HELICoiD (HypErspectraL Imaging Cancer Detection) Project, which ran from 2014 to 2016. Headwall supplied a two-sensor system to the group, which included four universities, three industrial partners, and two hospitals. Together, they successfully accomplished both goals, which according to David Bannon, Headwall’s president and CEO, were first to develop and implement a neurosurgical spectral instrument that would be used in the neurosurgical suite during brain cancer surgeries; and second to establish the capability to discriminate between healthy and cancerous tissue.

The knowledge gained from this collaborative expands the awareness of spectral imaging’s capabilities in noninvasive treatment and precision medicine (see Fig. 5). Beyond brain cancer, HSI may influence cellular spectroscopy, digital pathology, rapid-turn diagnostics, and endoscopic techniques.

FIGURE 5. Tumor tissue identification results obtained from the validation database employing the HELICoiD demonstrator: synthetic RGB color image and spectral classification map of the primary glioblastoma hyperspectral image (a, b); synthetic RGB image and spectral classification map of the primary-grade I meningioma hyperspectral image (c, d); synthetic RGB image and spectral classification map of the primary grade II oligodendroglioma hyperspectral image (e, f); and synthetic RGB image and spectral classification map of the primary glioblastoma hyperspectral image (g, h).

HSI is still considered an emerging technology. According to Berezin of HSPeQ, “There is an emerging trend in global healthcare to utilize geospatial imaging to predict the spread of certain diseases.”

If so, then progress indicates that the landscapes of our planet and our bodies will be linked by HSI.

ACKNOWLEDGMENT

The author wishes to thank Helena Hurbon, Mikhail Berezin, Rajagopal Srinivasan, Angela Belcher, Peter Jenson, Ross Nakatsuji, and David Bannon.

REFERENCES

1. See https://go.nasa.gov/3ANxsiQ.

2. C. Sheard and L. A. Brunsting, J. Clin. Investig., 7, 559–574 (1929).

3. R. Leon et al., J. Clin. Med., 9, 6, 1662 (2020).

4. See https://bit.ly/Petrie-Ref4.

5. See https://bit.ly/Petrie-Ref5.