US team demonstrates sonar-based gesture control technology for mobile devices.

Engineers at the University of Washington have developed a sonar-based gesture control system that they claim could enable users to interact with mobile devices by writing or gesturing on any nearby surface or even in mid-air.

The so-called FingerIO system works by using a device’s own microphones and speakers to emit an inaudible sound wave. This signal bounces off the users finger, and these “echoes” are recorded by the device’s microphones and used to calculate the finger’s location in space.

The team claims that the system can accurately track two-dimensional finger movements to within 8mm, which is sufficiently accurate to interact with today’s mobile devices. What’s more, because sound waves travel through fabric and do not require a line of sight, users can even interact with a phone inside a front pocket or a smartwatch hidden under a sweater sleeve.

“You can’t type very easily onto a smartwatch display, so we wanted to transform a desk or any area around a device into an input surface,” explained lead author of a paper on the project Rajalakshmi Nandakumar, a UW doctoral student in computer science and engineering. “I don’t need to instrument my fingers with any other sensors – I just use my finger to write something on a desk or any other surface and the device can track it with high resolution.”

Using sound waves to track finger motion offers several advantages over cameras — which don’t work without line-of-sight said UW assistant professor of computer science and engineering Shyam Gollakota. “Acoustic signals are great — because sound waves travel much slower than the radio waves used in radar, you don’t need as much processing bandwidth so everything is simpler.”

He added that the fact that most devices already feature a speaker and microphone would mean that the cost of the technology would be relatively low.

The technology has already been trialed on a modified Samsung smartphone. The next step is to demonstrate how it could be used to track multiple fingers moving at the same time, and extending its tracking abilities into three dimensions by adding additional microphones to a device.