PHOTONICS APPLIED: UAV VISION: Rigors of unmanned aerial vehicles challenge vision technology
Unmanned aerial vehicles (UAVs) are used for both commercial and industrial surveillance and reporting, as well as covert military operations in an oftentimes hostile environment. These UAVs need to be small, lightweight, and highly maneuverable, placing rigorous demands on the image sensors, optics, and communications equipment and software algorithms that provide their intelligence.
Unmanned flight
The history of UAVs reaches all the way back to the U.S. Civil War, when the opposing North and South armies equipped balloons with explosives and attempted to launch them over enemy territory where they would hopefully fall and explode.2 Of course, these earliest of UAVs were entirely blind, lacking any sort of vision capability or intelligence. According to Pete Lewis, U-systems program manager at SRA International (Arlington, VA) and director of the UAVforum (www.uavforum.com), an online community for all things UAV related, it wasn’t until 1924 that the British RAE made the first successful radio-controlled flight without a pilot.
Today, no less than 45 countries fly hundreds of different UAV models; in the U.S. alone, Lewis estimates that there are approximately 80 companies, academic institutions, and government groups developing more than 200 different UAV designs ranging in price from $1000 to $26 million dollars (see also www.thirtythousandfeet.com/uav.htm). Japan, says Lewis, leads in commercial UAV use with approximately 2000 unmanned helicopters spraying fields for pest abatement or seeding crops. Worldwide, these UAVs range in size from the palm-size Black Widow “micro” UAV spy plane (6 in. in diameter, 1.5 oz) to Northrop Grumman’s RQ-4/Global Hawk (weighing in at 25,600 lb).
One of the most well-known and successful UAVs is the Predator, which has been used by the U.S. Air Force (USAF) since 1995 in combat operations over Bosnia, Serbia, Yemen, Afghanistan, and Iraq (see Fig. 1). Manufactured by General Atomics Aeronautical Systems (San Diego, CA), it serves in a reconnaissance role and can also fire two Hellfire missiles. The USAF calls the Predator a “system” consisting of four air vehicles with sensors, a ground-control station, and a staff of 55 people. Sensor communications—from its multispectral targeting system, color nose camera (used primarily for flight control), variable-aperture day and infrared cameras, and its laser designator—are relayed by either a line-of-sight or satellite data link. In addition to dangerous combat operations, the Predator has also been used for civilian search and rescue and scientific research.
“When we first went public in 2004 the only vision offerings available were exceedingly heavy and cumbersome for our small helicopters,” said Jei Wheeler, director of unmanned aircraft systems for Tactical Aerospace Group (TAG; Corozal, Belize), which offers four different UAV helicopter models with payload capacity from 60 up to 600 lb. “Realizing that the majority of UAVs would be in the smallest payload categories, vision suppliers quickly reduced form factors and new smaller systems began appearing on the market almost quarterly throughout 2005 and beyond. Now in late 2008 we have some remarkable systems coming to market, in particular a new 4 in. system from Controp (Israel) with full visible and infrared capability and optional laser targeting/tracking. Cost is also being reduced from the $250,000 range to near $100,000, keeping the same or better resolution and capabilities in these much smaller packages.”
Another company at the forefront of this balancing act of sensor size/weight versus capability is Sensors Unlimited (Princeton, NJ; a Goodrich company). “A very interesting part of the whole UAV imaging story has to do with the famous U-2 spy plane,” said Ed Hart, general manager of Sensors Unlimited. “About five years ago, the secret was let out that Goodrich made the cameras for the U-2—cameras using sensors that can see in the short-wave-infrared (SWIR) part of the spectrum, enabling the U-2 to see as far as 130 miles through clouds, fog, high humidity, and even dust when visible cameras cannot.” Hart explains that the U-2 camera does this with cryogenically cooled indium antimonide (InSb) technology to span both the mid-wave and short-wave parts of the spectrum.
Because cryogenic cooling and its large lens system makes the U-2 camera too large for all but the largest UAVs, Goodrich—through its acquisition of Sensors Unlimited (Princeton, NJ) in 2005—developed indium gallium arsenide (InGaAs) sensors that allow imaging in the SWIR band without cooling, enabling exceptionally small cameras. The Sensors Unlimited SU640KTSXInGaAs SWIR camera, for example, weighs less than 90 g, operates at room temperature, and uses less than 2.5 W at 20°C. The 640 × 512-pixel camera gathers full-size images at 30 frames/s and features a 25-µm-pixel-pitch focal-plane array responsive from 0.7 to 1.7 µm. And because InGaAs technology detects reflected light rather than heat signatures, this SWIR camera captures images both day and night and during the dawn and dusk thermal crossover periods when mid- and long-wave cameras are unable to distinguish targets against backgrounds at similar temperatures.
New challenges
The war on terror has flooded defense companies with funding to develop more-sophisticated UAVs. Using the proceeds of an Office of Naval Research (Arlington, VA) contract award, Imperx (Boca Raton, FL) developed a prototype UAV that integrates vision, global positioning, and infrared (IR) ranging technology. Designed to operate within the payload compartment of an Aerosonde (Notting Hill, Australia) UAV, the system captures images using Imperx’s own IPX-4M15C sensor—a 2048 × 2048-pixel color camera with 8- or 10-bit digital output and programmable frame rate and integration time.
According to Petko Dinev, CEO of Imperx, PC-104 frame grabbers were not powerful enough to handle the MJPEG compression required by the system for the massive amounts of data collected. As a result, images are sent to Imperx’s FrameLink-JFLEX frame grabber that is integrated with a JRex-PM CPU board from Kontron (San Diego, CA) through a proprietary PC-104-like interface; the frame grabber/CPU combination is also being offered separately by Imperx and Kontron as a product for quality assurance, production automation, materials flow, and machine-control applications. Image and position data are captured by a ground station using a radio-frequency data transceiver; ground-station operators configure the camera’s operating parameters in real time. The UAV is targeted for deployment in Iraq to search for roadside munitions.
Vision capabilities such as target tracking for UAVs are improved, not only through better sensors and data transmission architectures, but also through sophisticated image-processing algorithms. Researcher Pascual Campoy and colleagues in the Computer Vision Group at Universidad Politécnica Madrid (Madrid, Spain) are applying visual servoing techniques—based on the displacement in the image plane of a visual patch or a characteristic extracted from an image—to control the position of a UAV (see Fig. 3).3 Global-positioning-system technology guides the UAV to the general region of a target and then selected features in the image plane are tracked using such algorithms as the Harris Corner detector, Hough transform, the Lukas-Kanade algorithm, or the scale-invariant feature transform to provide references for flight control.
Vision-system development for UAVs will only grow in complexity as the value of unmanned flight is further recognized for both military and civilian operations. “Rotor F/X (Van Nuys, CA) just partnered with a Las Vegas sensor technology company to combine a UAV with vision sensors and a data-analysis package developed by JPL for use on the Mars Rover,” says Jei Wheeler. “This new system will also be used in open-pit mining for high-resolution measurement of pit slopes and tailings, and for the oil and gas industries for exploration and pipeline leak detection,” adds Wheeler.
“UAVs are taking on more and more sophisticated tasks—watching day and night over several days,” says Ed Hart. “If you’re used to seeing a stable market, UAV vision is not that. Rapid, continuous improvement is the order of the day.”
REFERENCES
- Http://en.wikipedia.org/wiki/List_of_unmanned_aerial_vehicles.
- www.defendamerica.mil/articles/apr2002/a041702a.html.
- P. Campoy et al., J. Intelligent and Robotic Systems, DOI 10.1007/s10846-008-9256-z (June 30, 2008).
Gail Overton | Senior Editor (2004-2020)
Gail has more than 30 years of engineering, marketing, product management, and editorial experience in the photonics and optical communications industry. Before joining the staff at Laser Focus World in 2004, she held many product management and product marketing roles in the fiber-optics industry, most notably at Hughes (El Segundo, CA), GTE Labs (Waltham, MA), Corning (Corning, NY), Photon Kinetics (Beaverton, OR), and Newport Corporation (Irvine, CA). During her marketing career, Gail published articles in WDM Solutions and Sensors magazine and traveled internationally to conduct product and sales training. Gail received her BS degree in physics, with an emphasis in optics, from San Diego State University in San Diego, CA in May 1986.