Image-guided Surgery/Optical Tools: Augmented reality project aims for broad acceptance by surgeons

It's been a year or more since Pokémon Go enabled consumers to overlay photo imagery with digital data, and since pioneering surgeons began testing Google Glass to "float medical images in their field of view," as the New York Times reported in June 2014. But while augmented reality (AR) for surgery has been a popular concept since the 1990s, a tool for combining the surgeon's natural view with patient data has not yet been widely implemented.

Knowing that the quality of the AR experience is a major determinant for acceptance among physicians, European scientists are working to effectively integrate virtual content with real-world imagery—spatially, photometrically, and temporally. The three-year Video Optical See-Through Augmented Reality Surgical system (VOSTARS) project was launched in December 2016 with a grant of €3,816,440 from Horizon 2020 via Europe's Photonics Public Private Partnership. The project partners, representing Italy, Germany, the United Kingdom, and France, are basing their work on the notion that, for many surgeries, head-mounted displays (HMDs) offer the most ergonomic and easily translatable solution.

The first hybrid system

VOSTARS aims to exploit the best aspects of existing systems, along with knowledge acquired within the consortium. Current AR HMDs fall into one of two categories: optical see-through (OST) and video see-through (VST). In OST systems (such as the Microsoft HoloLens), a semitransparent mirror enables superimposing small amounts of computer-generated imagery into the user's line of sight, providing a more-natural experience. In VST systems (as seen in Oculus Rift headsets), virtual content is merged with images captured by two external cameras mounted on the visor. The user is submerged in the virtual world, experiencing life through screens.

Each approach has strengths and shortcomings, and Fabrizio Cutolo of the University of Pisa (Italy) states that neither is sufficient: "For something as critical as an operation, we had to have the naturalistic 'feel' of OST, while having the fluid interaction of the VST," Cutolo says. "When operating on a real person, it was clear to us that the benefits of OST and VST systems could be combined to make a hybrid device."

Key to development of the VOSTARS visor are photonics components such as a small, high-performance microdisplay and an LED optical waveguide. VOSTARS uses a head-mounted camera to capture the surgeon's view, and then merges this footage with the patient's medical images, from CT, MRI, or 3D ultrasound scans, using advanced image-registration techniques to precisely superimpose the digital imagery with the patient's three-dimensional anatomy in real time on the surgeon's visor dashboard. The visor also conveniently displays vital data, including anesthesia information, heart rate, body temperature, blood pressure, and breathing rates.

"A clinician can move freely while still seeing the patient, the hybrid x-ray image, and all of critical data all at once in a surgical 'dashboard' inside the screen. The surgeon, the patient, and the procedure are all at one," says project coordinator Dr, Vincenzo Ferrari, biomedical engineering researcher at the University of Pisa's Department of Information Engineering. "With this state-of-the-art, highly ergonomic visor, we intend to provide all the information required to improve surgery. The primary goal is to reduce not just surgery times, but also the time spent under anesthetic and the cost involved in any operation."

The project forecasts a significant improvement of intervention accuracy and a reduction in procedure time of at least 11%. It will save "20 minutes of every three hours of surgery and the guarantee of an extremely accurate intervention," says Ferrari.

Timeline and trials

VOSTARS aims to have a working prototype of the first-ever hybrid OST/VST HMD surgical navigator in May 2018, and to achieve mass production by 2022. Meantime, the device will undergo three clinical trials, initially on a number of procedures to the head, including maxillofacial (jaws and face) surgery, neurosurgery, ear, nose, and throat (ENT) procedures, and orthopedic surgery. The results of these trials will be key for helping to achieve economic viability.

Although VOSTARS is being specifically designed for medical procedures, the developers intend for it to evolve into a multipurpose AR HMD platform.

The VOSTARS project is being coordinated in Italy at Department of Information Engineering and the EndoCAS Center for Computer Assisted Surgery (Pisa University). Other participants from Italy are Alma Mater Studiorum-Universita Di Bologna, and Scuola Superiore Di Studi Universitare E Di Perfezionamento Sant'Anna. The German organizations are SCOPIS GMBH, Charité – Universitätsmedizin Berlin, Sankt Gertrauden Krankenhaus GMBH, Technische Universitaet Muenchen, and Pilotfish GMBH. VREO Innovation is the UK representative, and from France is Optinvent and the Commissariat A L'Energie Atomique et aux Energies Alternatives.