Expression-filled remote meetings possible with robotics, head-mounted displays
Barcelona, Spain--Scientists from across Europe have now combined a set of robotic technologies, including full head-mounted displays, that could effectively transport someone into a meeting room on the other side of the world. While "telepresence" robots already exist that can stalk the workplace aisles, the European Union (EU)-funded project is a concerted effort to get people (or more accurately, their avatars) to boost their meeting attendance.
Teleconferences, video conferencing, and web exchanges have become the norm for most multinational businesses—however, many people (although not all) still prefer face-to-face.
Subtle cues are key
"When we meet people in the flesh we can pick up on subtle cues—facial expressions, quirks, who is looking at who," says Stephen Dunne from Starlab in Barcelona. "There is so much nonverbal communication that you miss, even with the highest quality video-conferencing technology. You can't shake hands or decide to look around the room, for example."
The project, called "Beaming through augmented media for natural networked gatherings," (Beaming, for short) uses immersive virtual-reality techniques and technologies so that people are embodied by their avatars -- which could, for example, be real-life robots that become the attendees' eyes, ears, and mouth(s).
Attendees are rigged up with multiple sensors and have their heads covered in full head-mounted displays with video-camera eyes and microphone ears. Head motions, speech, and, with the addition of arm sensors, arm motions—all are beamed to and from the meeting. And if someone touches a robot's hand, this pressure is relayed back to pressure pads on the attendee's arm apparatus. Newer robot versions allow expressions such as eye-rolling to be transmitted.
Strong sense of presence
One of the biggest tasks for the Beaming partners has been to develop a framework data architecture for the system that defines how all the visual, audio, motion, and pressure data are packaged and transmitted between the attendees and their remote environments. It also establishes how the 3D model of the remote location is generated so that the attendees have a strong sense of presence.
The technology was demonstrated in an interview between a project scientist in Spain and a BBC journalist in University College London. The journalist was able to interview the scientist and even give a "high five" at the end of the demonstration (follow link below to find videos). "It was important for us to make the interaction as natural as possible. Whether people are interacting with a robot or some kind of virtual avatar, we want the experience to feel natural and not get in the way of normal communication," says Dunne.
One simple improvement was to give the robot a more expressive head. The team replaced its robot's mechanical eyes and mouth with an LCD display. "We used graphics for the eyes and mouth; they look much more natural," says Dunne. "People seem to relax more if they can look a robot in the eye and the eyes look right. It is easier to forget that they are dealing with a machine."
Getting the emotional state right
The researchers have experimented to see how attendees' heart rates, facial expressions, and even brain waves could be clues to their emotional states. So far, technology can be trained to use these signals to recognise basic emotional states—for example, to differentiate between relaxation and stress. The challenge now is to work out how to communicate their signals to everyone else through the avatar.
The Beaming project received EUR 9.2 million (of total EUR 12.4 million project budget) in research funding under the EU's Seventh Framework Programme (FP7).
John Wallace | Senior Technical Editor (1998-2022)
John Wallace was with Laser Focus World for nearly 25 years, retiring in late June 2022. He obtained a bachelor's degree in mechanical engineering and physics at Rutgers University and a master's in optical engineering at the University of Rochester. Before becoming an editor, John worked as an engineer at RCA, Exxon, Eastman Kodak, and GCA Corporation.