Live music production system is controlled by hand gestures

The team said the system will allow performers to wirelessly compose, arrange and perform music live on stage using intuitive hand movements.

The interface of the system is a pair of sensory gloves designed by Fifth Dimension Technologies (5DT) with additional sensors from x-IO providing pitch, roll and yaw input.

Dr Tom Mitchell, an acoustic researcher from the University of the West of England, told The Engineer: ‘The majority of the development has been in integrating these sensors then mapping them.

‘So we analyse the data to identify gestures and once those gestures have been identified it’s a case of mapping those onto audio processes.

‘We’re well aware that other people have used gloves for musical purposes and even used comparable sensors, but we’re using a mapping system that’s designed in collaboration with Imogen so that she can move away from her equipment and just perform without disengaging from the audience.’

As well as this basic recall, Mitchell said the system allows the performer to adapt and improvise. For example, by making a grasping motion the performer can ’catch’ vocals or any other instrument, and filter these sounds by clasping their hands together and then ’release’ the sound again by opening their hands.

However, Mitchell said one of the key challenges was to integrate everything seamlessly without any lag.

‘We worked on trying on to get the latency as low as possible, which is why all the software has been written from scratch, so when Imogen waves her arm the sound happens… it’s perceivable, instantaneous.’

Heap performed with the gesture system at the recent TED (Technology Entertainment and Design) concert in Edinburgh.

’I walked onto the rehearsal stage, gloved up, without any other equipment and began to sample, loop and apply effects to my voice, acoustic and virtual instruments. It feels so fluid to have everything on me without the need to keep flitting back to my gear on stage,’ she said in a statement.