Electro-optical solution for virtual material capture and rendering

HP’s Z Captis is an electro-optical solution designed to enable creative professionals—within fields like fashion, architecture, game development, interior decorating, 3D animation, and shoe design—to create lifelike digital models of materials under various virtual lighting and orientation conditions.

Captis is fully integrated with Adobe’s Substance 3D Sampler to convert PBR data into 3D textures and material assets. The device captures over 2,200 shots with its 108-MP color imager, generating more than 0.5 TB of data. This data is processed internally by its embedded Nvidia’s Jetson Xavier AGX, a powerful graphics processing unit (GPU)-based compute module.

HDR imaging, time-delay and integrate (TDI) signal-to-noise enhancement, as well as 8x lamp orientations and 8x elliptical polarization captures, are key features built into the system capture stream. Only ~3% of the data needs to be transferred to an HP Z Workstation via USB3 or WiFi6 for final rendering in the Substance 3D Sampler. Captis processes 8k × 8k images in less than 10 minutes, and software updates are continuously improving performance.

The AGX module, interestingly also used in autonomous vehicle systems, supports a containerized software architecture to allow Adobe, independent software vendors, and researchers to capture and pre-process PBR data using HP’s custom software development kit (SDK). The Captis Capture Management SDK includes a Python-based framework with OpenCV, OpenEXR, CUDA, CuPy, and PyTorch, which enables efficient artificial intelligence (AI) tensor handling and accelerated GPU processing for image processing and neural network training.

The Captis solution is designed to accommodate the varied environments in which creative professionals may need to capture surface appearance topographies. It offers two operation modes: Studio and Explorer (see Fig. 1).

Studio Mode is ideal for capturing material surface properties within controlled settings like a studio or office. Samples are placed on Captis' rotating tray, where they are digitally imaged under carefully managed lighting conditions.

Explorer Mode allows greater flexibility. By removing the backlighting and material-handling base, the light projection cone and imager can be mounted onto a tripod or placed directly onto a surface. This setup enables the capture of surface details from any orientation, which makes Captis portable for use in remote environments.

How does it work?

An overview of the critical electro-optic functional elements of Captis is shown in Figure 2. Let’s now explain how photometric stereo and polarimetric methods are used to obtain the true-to-life material appearance data using Captis.

Photometric stereo is a technique used to recover the surface shape and texture of an object by capturing multiple images under different lighting conditions. In this method, the object is illuminated from various angles, and each image is taken with the same viewpoint (monocular) but with different lighting directions. By analyzing how the light interacts with the surface—how the brightness of each point on the surface changes depending on the direction of the light—photometric stereo estimates the surface normals (the perpendicular vectors at each point on the object’s surface). These surface normals are then used to reconstruct the object's 3D shape and reflective physics.

Polarimetric methods are used to analyze the polarization state of light reflected from an object's surface, which provides information about the surface material and structure. Polarization refers to the orientation of light waves as they travel. When light reflects off a surface, its polarization can change depending on the surface properties like texture, smoothness, or material type. Polarimetric imaging involves capturing images via polarizing filters, which are rotated to measure how the light's polarization changes after reflecting off the surface. These measurements provide insights into surface features like roughness, reflectivity, and the material’s refractive index.

For PBR implementations, polarimetric systems use linear polarizers placed in front of both the camera lens and light sources. The camera captures multiple images with different polarizer orientations, and the resulting data is processed to extract information about the surface's albedo or diffuse vs. specular reflective characteristics. Together, these methods offer a comprehensive view of both surface geometry and material omni-orientational reflective properties.

Captis PBR electro-optic element requirements and challenges

Let’s dive into an examination of the more critical electro-optic elements of the design and how challenges with their functional requirements were addressed. Several PBR-directed electro-optical elements and their particular utilities are currently the subject of HP pending and issued patents.

108-MP mobile imager w manual mode optical image stabilization (OIS). Captis aimed to break from traditional solutions that rely on expensive, high-resolution DSLR (digital single-lens reflex) cameras or costly 30-MP industrial machine vision cameras. Instead, the goal was to use an ultrahigh-resolution camera module designed for high-end mobile phones at a lower cost. The solution was LG Innotek's (LGIT) camera module that features the Samsung Isocell HM2 Imager.

The camera module features a 7-P autofocus (AF) lens, and Captis uses custom macro-focus algorithms to achieve sharp resolution across the entire material sample. It enables high fidelity captures of more than 8k × 8k. The MIPI Type-C interface allows for fast image transfers at ~9 fps, a significant improvement compared to ~0.5 fps for similarly high resolution DSLRs, a 20x boost.

A significant imaging alignment challenge arose for Studio Mode, when large material samples (30 × 30 cm) required a wide field-of-view (~84° diagonal), which makes the system highly sensitive to small planar orientation changes, as well other camera-to-material orientation offsets. For instance, a tilt of just 0.1° between the camera and the sample can cause a 1-mm material shift. To address it, Captis includes factory camera alignment stages for precise tuning of tilt, rotation, and x/y offsets, ensuring exact and repeatable capture fields. Figure 3 shows these alignment stages (a) and the software (b) developed by HP for real-time alignment with laser-ablated fiducials.

One of the key challenges for a photometric stereo system is minimizing pixel shift between images to avoid post-capture digital realignment, which greatly enhances PBR results. A unique feature of the Captis mobile phone imager is its nitinol alloy-based actuator, which provides optical image stabilization (OIS) with a mechanical bandwidth of up to 30 Hz.

While typically used for dynamic stabilization in mobile phone cameras, which is crucial for Captis’ Explorer mode, it operates more disruptively by improving Studio Mode image capture alignments image-to-image. It allows precise control to offset image shifts caused by events like polarizer rotation or thermal drift in the camera lens’ focal properties.

In Studio Mode, this actuator enables manual adjustment of the image position relative to the material with ±160-µm motion in both x and y directions. With 12-bit resolution (4096 steps), the system can achieve control increments of ~0.08 µm, or about 1/9th of a pixel, offering included and future novel image capture capabilities. These will include: real-time pixel shift correction between images, deterministic super-resolution image captures, and true 108-MP RGB-Nona 3 × 3 color pixel cluster RGB conversion.

Polarization viewing mechanism with high-accuracy discrimination. Captis uses linear polarized (LP) lighting from eight orientations at a low-grazing angle (30°) to measure how a surface reflects light—either specularly (mirror-like) or diffusely (matte-like). By analyzing the behavior of the incident polarized light, the device can distinguish between surfaces that maintain the polarization (specular) and those that scramble it (diffuse).

The polarization-state viewer, shown in Figure 4, includes a glass linear polarizer with an extinction ratio of over 2500:1, a hybrid stepper motor with 3,200 micro-steps per revolution (0.1125° accuracy), and a 14-bit absolute rotary encoder. This system allows precise calibration of each lamp’s polarizer orientation to within ±0.25°. During a material scan, the device captures data at eight polarization states, each 22.5° apart, with each transition taking just 0.3 seconds, which enhances overall system capture speed.

Conditioned near-light octagonal-oriented material sample illumination lamps. The Captis mechanism to determine the directionality and reflective properties (diffuse vs. specular) of a material’s pixel-sized area (~34 × 34 µm) works by illuminating the sample at a low grazing angle (~30°) from multiple directions. It offers eight lighting orientations, spaced 45° apart, around the material sample. These lamps meet strict conditioning requirements, including:

• Wide-angle divergence in the horizontal plane to support their close proximity to the sample (near-light operation);
• Vertically linear polarized lamp outputs; 
• High brightness with 256+ levels of brightness control (50 W 100% to 5%);
• High color temperature (6500K) and color-rendering-index or CRI (90%) that is maintained at all LED brightness levels and with high color uniformity across sample;
• Illuminance distribution over the entire sample area and technical areas ~3 Klux average at 50 W;
• Homogeneous specular reflection of the source (no LED hot spots) and with blended brightness edges;
• High brightness stability with no fixed-pattern noise (FPN).

The design requirements are fulfilled using a light box with three stacked layers, as shown in Figure 5. The Back LED/Optics Stack consists of a 65-white LED array on an aluminum printed circuit board with custom refractive optics to evenly illuminate the camera’s field of view. The Middle Specular Profile Conditioning Stack uses a spatial opaque white dot-pattern filter to smooth out edge brightness and begins diffusion through a Luminit holographic light-shaping diffuser (LSD). The Front Stack further diffuses light with additional LSD, which incorporates linear polarization and uses Mosmite antireflective film to reduce internal reflections and minimize reflective noise when other lamps are in use.

Dark color hue and low-reflection backlight with magnetic attach. To capture the reflective properties of nonopaque materials like silk, Captis includes an 85% uniform backlight behind the rotating sample tray. This backlight provides up to 700 NITs of polarized light, is adjustable to 256 levels, and shares key characteristics with the top lightboxes. Uniquely, it appears dark gray (black) when off, and its Gorilla glass cover with a 30% haze etch creates a diffuse surface that prevents bright reflections from interfering with top illumination for smaller material samples (<30 × 30 cm).

Its sample tray features a magnetic return path-free design, with a ferro-magnetic plate beneath the entire area. This allows the use of custom magnets with an elastomeric Cat’s-Tongue-like gripping surface to hold samples flat and in place.

In situ image calibration with polarization-state reflective mapping tags. The Captis material sample capture area is 30 x 30 cm, and the camera’s 4:3 aspect ratio enables two image calibration areas along the wider field of view (FoV). Figure 6 shows these in situ calibration areas, which use 2400-dpi printing for precise geometric alignment and color-accurate calibration.

What’s next?

Captis is poised to revolutionize how creatives source and integrate materials. Instead of relying on physical samples for projects like auto interiors, shoe designs, or buildings, material suppliers can use it to digitally capture the true appearance of material surfaces like cloth, leather, or structural elements. This data is provided in metadata layers such as base color, normal, height, opacity, roughness, specular level, and Metallic Maps, among others, which are the visual rendering products of Captis’ PBR material captures.

The material supply chain for the design phase of products and environments will be disrupted by moving more universally to this type of workflow as enabled by Captis. The positive impact on the material supply chain—by both improving supply chain efficiency and reducing the carbon footprint—will be significant.

The HP Advanced Compute and Solutions Business Unit (BU) in Fort Collins, Colorado, is an evolution of the long-standing HP Z Workstations BU. The mobile versions of these workstation solutions were selected by NASA for use on the International Space Station. We are now expanding our portfolio to meet the edge computing needs of both existing and new customers.

A key focus is on machine vision (MV) solutions, combining high-end embedded GPU computing with advanced smartphone camera modules. The convergence of AI and machine vision at the edge, supported by HP's SDK and camera drivers, will drive the development of new HP PBR products and collaborations with academia and industry to bring innovative HP hardware-based edge MV solutions to market.

ACKNOWLEDGEMENTS

The author would like to acknowledge the wide, but relatively small and dedicated team within HP and Adobe R&D. The principal collaborator at Adobe was PBR Software Architect Jerome Derel, who in collaboration with Josh St. John, the HP Captis product manager, were the genesis for this product collaboration. The HP R&D hardware team included EE Lisa Tesdahl and MEs JC Castillo and Keith Harral. The HP Industrial Design team included Matt Tivnon and Owen Columbus. HP R&D software developers in lead roles were Santi Reyero (architect) and Miles Claver, with contributions from Aaron Huskerson, Stephen Sutherland, and Alex Shellum. HP R&D Management for Captis included Gina Thompson, Chris Damir, and Michael Brinkman. David Wright-Huynh and James Egan were solutions program manager and R&D program manager.