Editor’s Note: Read Part 1 of this article, “Eyes of a ship: Laser detection at sea”.
Ships calling at American ports in 2018 carried almost 1601 million tons of international traded goods for a total value of $ 1.762 billion, according to the U.S. International Trade Administration. Because of the quantities involved, ship cargo never has a low value, even if the vessel is small and the unit price of the traded commodity modest.
A 2020 report by the European Maritime Safety Agency (EMSA) about the causes of 1801 ship accidents in Europe within the 2014-2019 timeframe indicates humans were responsible for 54% of accidents. The primary factor was noncompliance with collision prevention rules issued by the International Maritime Organization (IMO).
The U.S. National Transportation Security Board reached similar conclusions. In “Safer Seas Digest 2020,” the board examined the causes of 42 major accidents in recent years within American waters or American vessels and concluded “those who do not attend (or attend to) accident lessons most risk paying a steep price—not necessarily only a financial one.”1
Paradoxically, the increased flow of information on the bridge provided by improved monitoring technology may increase cognitive stress and lead to delayed decisions by the officers on watch. Technology itself may trigger accidents it is supposed to prevent.
In the future, by ever-increasing vessel dimensions and sea traffic to a global scale, an intelligent, autonomous ship will be able to process information flows quickly enough to react accordingly and guarantee safe delivery of valued cargoes (see Fig. 1).
Efforts to produce a fully autonomous ship of IMO degree four, in which “the operating system of the ship is able to make decisions and determine actions by itself,” are underway in Scandinavia and other neuralgic maritime areas.
The four IMO degrees of autonomy are:
Degree 1: Ship with automated processes and decision support. Seafarers are on board to operate and control shipboard systems and functions. Some operations may be automated and at times be unsupervised, but with seafarers on board ready to take control.
Degree 2: Remotely controlled ship with seafarers on board. The ship is controlled and operated from another location. Seafarers are available on board to take control and operate the shipboard systems and functions.
Degree 3: Remotely controlled ship without seafarers on board. The ship is controlled and operated from another location. There are no seafarers on board.
Degree 4: Fully autonomous ship. The operating system of the ship is able to make decisions and determine actions by itself.
Beyond radar: electro-optical ship sensors
In 2015, Rødesth and Burmeister created a modularization scheme of components for the design of autonomous vessels based on the concept of an unmanned dry bulker of approximately 50,000 tons dwt.2 IMO recommended their Formal Safety Analysis (see Fig. 2).
Several factors limit the identification of potential collision risks by radar and automatic identification systems (AIS). Radars have a blind spot and do not register smaller wood or fiberglass craft unless they carry a reflector. They are further subject to signal clutter in traffic-intensive ports or in access waters to ports. AIS is a passive type of sensor that provides information only on targets also carrying an AIS that must be switched on (not always the case).
Rødesth and Burmeister proposed integrating information from electronics with the input of electro-optical sensors: a visual camera and a FLIR camera in the far-infrared.
The specific advantages to seafaring of each IR waveband are:
MWIR and LWIR. A bit ahead of the times, the German Navy in WWII feared the Allies could track submarines on surface at night by the IR emissions of their diesel engines. Thermal emissions of a ship cause a temperature difference with its surrounding background of sky and sea that can be readily identified in darkness by thermal sensors—no external target illumination is needed. Some masters use thermal cameras to improve night navigation. Castaways, if alive, can be identified in darkness by the temperature difference between the sea and their bodies. Small craft, sail ships, buoys, or lost containers with little or no thermal emission are hardly perceived in the midwave-infrared (MWIR), but by residual temperature difference with the waters, it can occasionally be done in the longwave-infrared (LWIR).
SWIR and NIR. Signals in shortwave-infrared (SWIR) and near-infrared (NIR) are less influenced by scatter and allow more efficient long-distance sensing in adverse weather and visibility conditions (snow, rain, fog, haze, and smoke). Smaller craft and objects with temperatures similar to the sea background are more easily detected. SWIR cameras, like those in the visible, rely on reflected light via the sun, moon, or artificial sources. Data fusion is favored by analogous imaging of the two, and some NIR sensors perform down to 500 nm with a favorable overlap that enriches images in the visible.
Almost half of recent research on algorithms for ship detection focus on IR imaging, according to a 2021 survey by Liquian Wang and colleagues of Shandong University, with 20% dealing with the combination of visible plus IR.3 Because of the reduced atmospheric absorption, the majority of the researchers (68%) use LWIR. SWIR ship detection is primarily done via satellites in space.
Multispectral sensing, with information extraction from visible, SWIR, MWIR, and LWIR images, provides comprehensive target detection and discrimination the shipping community needs for autonomous vessels. An intelligent system capturing and evaluating data in different spectral bands is more error-proof. The increase of contrast leads to more sensitivity—for instance, the advantage of blending visible and IR images (see Fig. 3).
The three bands IR sensor
Four distinct cameras onboard a ship are located in strategic positions, which can be a rather cumbersome solution. More input sources complicate the data fusion algorithm and the detection process, but favorable developments of IR sensor technology may help in the near future.
Quantum well photoconductors (QWIPs) and sensors based on mercury cadmium telluride (HgCdTe; MCT) are available and feature dual-band IR detection. But QWIPs have a low response, and the use and production of mercury cadmium photodetectors face regulatory limits and cooling requirements that currently aren’t attractive solutions on merchant vessels.
Competing with MCT, a more ecofriendly antimonide technology, T2SL, may enable more flexible solutions. Research activity based on Type-II InAs/GaSb superlattices is targeting a new generation of IR imaging instruments with multispectral response. There is a further drive to reduce the cost of systems and increase their operation temperature to be more conducive to implementation on ships.
At Northwestern University’s Center of Quantum Devices, Professor Manijeh Razeghi is pursuing development of T2SL materials with “a remarkable tuning capability from SWIR to long infrared.”4 In 2020, Razeghi presented bias-selectable, multiband photodetectors based on indium arsenide (InAs)/gallium antimonide (GaSb)/aluminum antimonide (AlSb) and InAs/InAs 1-x Sb x type‐II superlattice that features a wide spectral response in the IR at temperatures of 77 and 150 K (see Fig. 4).
The system consists of three layers grown on each other for the LWIR, MWIR, and SWIR, respectively. Activation in sequence of each absorber depends on the applied bias voltage.
IR eyes on the seas: an example
Situational awareness systems for ships are designed to support and eventually surrogate humans in the key functions of organizing object detection, identification, classification, and tracking.
Finland is a pioneer in situational awareness systems for autonomous ships. A system designed by Groke Technologies is an example of IR cameras using the sensor pack for ship detection and identification in darkness. Mitsubishi Corp. is one of the main investors in the company, and Groke has a strong presence in Japan. Groke is participating in an R&D initiative led by Kawasaki Kisen Kaisha Ltd., JRC Ltd., and YDK Technologies Co. Ltd. Fujitsu is expected to develop the AI segment.
The goal of this R&D project is to design a system to prevent serious maritime accidents, such as ship collisions and groundings. The project aims to meet degree one of the IMO standard of autonomous ships development.
Groke has tested two awareness systems: the AS Lite is a pilot installation working on a fare in Japan, and the AS Pro, which has been in trials since March 2021 on the Ship Polaris VG on the route from Rauma in Finland to Rostock in Germany.
Data from AIS receivers, global navigation satellite systems (GNSS), and onboard inertial measurement units (IMU) are fused with those of a visible and thermal camera for measuring the position of the vessel and monitoring the sea surface.
“When we started at Groke, we evaluated the different technologies and we decided to go with the combination of daylight and thermal cameras,” says Iiro Lindborg, co-founder and VP of Groke; he is the former general manager of remote and autonomous operations at Rolls-Royce. “The unique part of this solution is the so-called ‘thermal blending,’ where we combine both visible and thermal image. We get the color from the camera in the visible and the thermal signature from thermal camera, and we join them together in our user interface.”
For data fusion, “our system uses data fusion by comparing the data from multiple sources, like AIS and camera detections,” Lindborg adds. “And then, if they are defined to be information on the same target, it is fused and only one object detection is shown. For that object, we can then provide details based on all the available sources.” The sea situation can be checked in real time on a bridge and from any location on the ship via handheld devices.
Groke had to develop its own IR sensor as well. “In the AS Pro, the 180 thermal camera is possibly the first in the world with such a field of view and performance,” says Teemu Helenius, hardware development lead at Groke. The situational awareness system developed by Groke will be installed on all company vessels owned by Tsurumi Sunmarine Co., Ltd, a chemical/clean tankers operator.
In 2018, the car ferry “Falco” of Finferry was the first vessel ever to navigate completely autonomously on a voyage from Parainen to Nauno within the Finnish archipelago. Rolls-Royce designed its situational awareness system, and Konghsbergs Maritime acquired Rolls-Royce Commercial Marine in 2019.
In 2020, the world’s merchant fleet consisted of 119,999 ships.5 Sooner or later, this expanding fleet will need some form of autonomy or at least more sophisticated sea-awareness systems. Supplying this promising market with broadband IR sensors based on antimonide technologies is the future task of the photonics industry.
REFERENCES
1. See www.ntsb.gov/news/press-releases/Pages/NR20210824.aspx.
2. Ø. J. Rødesth and H. C. Burmeister, TransNav, 9, 3, 357–364 (2015); https://doi.org/10.12716/1001.09.03.08.
3. L. Wang et al., J. Mar. Sci. Eng., 9, 1408 (2021); https://doi.org/10.3390/jmse912140.
4. M. Razeghi, A. Dehzangi, and J. Li, Res. Opt. (2021); https://doi.org/10.1016/j.rio.2021.100054.
5. See https://bit.ly/3CzYdvr.
Vittorio Lippay
Vittorio Lippay has been a member of the Institute of Physics since 2018 and a member of the Institute of Chartered Shipbrokers (London branch) since 2012.