The dawn of multi-touch

It was carved out of wood, had only one button, and was arguably one of the great inventions of the 20th century.

The world’s very first computer mouse, invented in the US in 1964 by Douglas Engelbart of the Stanford Research Institute, was instrumental in transforming computers from highly specialised scientific tools into the user-friendly devices that they are today. While early computer users had to learn the lingo like a visitor to a foreign country, Engelbart’s device paved the way for the point-and-click environment that lets us do everything from browsing the internet to playing games.

But the dear old mouse’s days may be numbered. A host of new computer interface systems are under development that use gentle finger movements and intuitive hand gestures to replace the RSI-inducing clicks and taps of traditional user interfaces. Many commentators believe these technologies will free us from the constraints of the traditional PC environment, usher in a new era of ubiquitous computing and take our interactions with the electronic world to new, unimagined levels.

Consumers got perhaps their first proper glimpse of this brave new world earlier this year, when Apple unveiled its iPhone. The company’s web-enabled, media playing, mobile phone sparked a bout of unprecedented global-salivation as gadget fiends pored over the product’s numerous features and pencilled its summer launch date in to their diaries.

At the heart of iPhone’s appeal lies a slick-looking touch-screen display. Trumpeted by the company as the most revolutionary interface development since the mouse, this multi-touch display allows users to flick through album covers, manipulate images, resize them, and zoom in and out using nothing more than a series of finger movements.

Though Apple is tight-lipped over the design of this screen, it seems likely that it is based on technology developed by a small display company acquired by Apple two years ago.

The organisation, Fingerworks, was something of a multi-touch pioneer and won an enthusiastic online army of admirers for TouchStream, a touch-sensitive keyboard that enabled particular finger movements to be interpreted as commands. It worked by measuring the disruptions caused by hand and finger movements to an electric field generated by the pad’s sensor array.

This technology made it possible to process information from multiple points on the screen. This meant, for example, that placing three or more fingers on the pad and twisting to the left would cause an open file command to appear, while twisting right would close the selected object. The system even introduced technology where spreading your fingers and closing them together would cause you to zoom in and out of an image — one of iPhone’s most superficially impressive features.

Meanwhile, Apple’s nemesis, Microsoft, is busy developing its own take on multi-touch technology. Andy Wilson of Microsoft Research has developed a novel, interactive display technology, where the outputs of two video cameras behind a transparent projection display are combined to produce an image of objects on the display surface.

Wilson said the idea behind the system, known as Touchlight, is to make everyday surfaces such as walls and tables interactive.

He believes the technology has implications for a future of ubiquitous computing in which potentially any surface is an input and computation device and the very displays we use and spaces we inhabit are aware of our presence.

The current system uses a pair of small, webcam-sized infrared video cameras mounted behind a transparent display surface. This surface is coated with a refractive, holographic film that diffuses light coming from a correctly positioned rear projector but allows all other light to pass straight through it.

So the cameras are able to see through the display but because they are infrared, they can’t see what’s being projected on the display at the same time. The touch image is produced by applying sophisticated image processing algorithms to the information gathered by the cameras. ‘The idea is to take those images from the camera and then figure out what the user is doing in front of the display. That’s it,’ said Wilson.

While the technology offers similar levels of interaction to that used on iPhone it is now treading a different path after Microsoft licensed the system to a company called Eon Reality, which is developing the system for use in high-end presentation systems.

But Wilson is investigating a range of other uses. ‘We’re trying to figure out the sort of things that make sense for two-handed input,’ he said.

One particularly impressive application is using the technology to manipulate an on-screen map. ‘You can do this very easily by moving your hands around rather analogously in the way you might interact with a real map,’ said Wilson.

A big advantage of the Touchlight system is that because it recognises gestures, the user doesn’t actually have to be touching the screen. Wilson’s group is working on methods of computing the 3D position of a user’s hands, so that not only can it sense when hands are placed on the surface but it can also sense where they are in 3D space. He said this capability could enable users to interact with 3D objects in a variety of interesting ways.

And, depending on how it is configured, a user could interact with the system from the other side of the room, opening up still more applications. ‘Potentially if you were to walk into your office and look in the direction of Touchlight it could recognise you at that moment and bring over your documents, or figure out where your eyes are and render an appropriate viewpoint based on that,’ said Wilson.

He added that the technology could also be used as the basis for a new, improved video conferencing system. Most existing set-ups use a camera mounted just above the display. This accounts for the disconcerting off-target eye contact associated with such systems. Wilson said that with Touchlight, because the camera is directly behind the display, it is possible to set up a video conferencing system that tracks where you are and always places the graphic of the person you are talking to directly between the camera and your eye.

Despite this early promise though, Wilson thinks the technology is a long way from fulfilling its potential. ‘We’re playing at the moment and aren’t exactly sure what the killer app is yet,’ he said. ‘I think there are some interesting advantages to it — but I don’t think it has really been thought out what they are. Nobody has a really good handle on this multi-touch stuff yet as far as I’m concerned — right now it’s about this demo of moving and spinning things and scaling things up and down, which is nice but I hope there will be more to it than that.’

One person that has perhaps exploited the eye-catching potential of multi-touch more than anyone else is Jeff Han, a researcher at New York University’s computer science department.

When he first presented his work at last year’s Technology Entertainment Design Conference in California, the audience — which included some of the biggest hitters in the technology world — gave him the kind of reception usually reserved for a rock star.

‘I think this is going to change the way we interact with computers,’ said Han, wowing the gathered throng as he used nothing more than his fingertips to manipulate images, create moving puppets and swoop through mountainous landscapes.

Using tracking cameras and rear projection, Han’s technology works by interrupting the passage of light supplied by an LED light source through the display screen. As well as being able to tell where the fingers are being placed on the screen, thanks to a phenomenon known as frustrated total internal refraction, it can also gauge the contact pressure and, potentially, even the approach of your hand to the screen.

Han has founded a company called Perceptive Pixel to develop his technology further, and has reportedly already shipped touch screens to sections of the military.