The 2011 Consumer Technology Winner - Fascinate

FascinatE

Joanneum Research, Arri, Softeco Sismat, Interactive Institute, Salford University, Technicolor, BBC R&D, Fraunhofer HHI, Alcatel-Lucent, TNO

A pub on Tottenham Court Road in London is one of many in Britain that shows football matches from Sky Sports in 3D.

The sight of adults wearing stereoscopic 3D glasses and staring at a screen is quite disconcerting but lots of people do it. Importantly, they are prepared to pay the landlord of the Rising Sun for the privilege of watching their team play in 3D.

This viewing phenomenon is just one of a host of advances that have heightened the TV-watching experience. HD, On Demand services and the ability to pause TV shows demonstrate how consumers are increasingly being given control of how they use TV.

Consumers are set to become more involved in their television viewing thanks to the FascinatE (a tortuous acronym for Format-Agnostic SCript-based INterACTive Experience) project, a €9.35m (£9m) scheme funded by the EU’s Seventh Framework Programme.

With 11 partners across the EU, the FascinatE team is developing a system that allows consumers to view and navigate around an ultra-high-resolution video panorama showing a live event, with the accompanying audio automatically changing to match the selected view.

The output can be adapted to their particular kind of device, covering anything from a mobile handset to an immersive panoramic display.

At the production side, this has required the development of new audio- and video-capture systems, plus scripting systems to control the shot-framing options presented to the viewer.

Moreover, to make it work, intelligent networks with processing components are being developed to re-purpose the content to suit different types of device and framing selections, and user terminals supporting interaction methods will allow viewers to control and display the content.

The system is expected to deliver a unique experience that will transform how live events are enjoyed

Once complete, the system is expected to deliver a unique interactive experience for consumers that will transform how live events can be enjoyed on TV, on large-screen public displays and on mobile devices.

Project member Ben Shirley, a senior lecturer at Salford University, is involved in the audio aspect of the project.

He explained to The Engineer that FascinatE could already be implemented on current smartphones.

’The only problem is network speeds,’ he said. ’Some of the context of the full FascinatE system, namely fully immersive audio, large full-resolution panorama — being able to zoom in and out to the full resolution standard — needs network bandwidth that’s projected to be there in four to five years’ time.’

One interactive element of FascinatE lets the viewer watching a live event ’grab’ a participant using a remote control or gesture interface, then places that person at the top of the screen. This screen selection lets the viewer follow the person around, for example, a football pitch with the rest of the game dominating the screen.

’My job on the audio side is to make sure you experience appropriate sound for exactly where your camera is pointing,’ said Shirley. ’So you would hear everything the sportsperson is saying and doing by positioning all the audio objects taking place during the game.’

Shirley added that an essential element of the project requires the format-agnostic delivery of audio-visual content.

’When you’re broadcasting football or music you’re looking at what the end-user device is,’ explained Shirley. ’If you’re broadcasting for 3D it’ll be a completely different workflow with different engineers and production values to if you’re viewing in 2D.

’If you’re packaging it for IPTV or mobiles it’ll be broadcast differently too.

’The idea of the format-agnostic part of it is it’s a single end-to-end chain from the acquisition through to the packaging, transmission and distribution, which is not dependent on the end-user device.’

To test the system, the team took FascinatE to a Premier League football match between Chelsea and Wolverhampton Wanderers in October 2010.
The aim of the trial was to gather a complete set of audio-visual material to take the concept forward.

The team set up an omni-directional high-resolution camera system from HHI, the Alexa high-dynamic-range camera from ARRI, an Eigen-mike from Technicolor and two Soundfield mics.

Shirley said that the camera set-up was relatively straightforward but the audio presented its own set of challenges, adding that the trial led to a development that could see an aspect of the technology being spun off from the main project.

From the end-user perspective it is important to know where a selected player is relative to the ball. Fifa, football’s governing body, will not, however, let balls be fitted with tracking devices.

’In an ideal world we’d have liked to have had large arrays of microphones so that we could very accurately pinpoint where the sound was coming from,’ added Shirley. ’However, we were only allowed to deploy 12 shotgun microphones around the pitch and a couple of stereo pairs.

“You’ve got 47,000 people behind them shouting at the top of their voice, which isn’t ideal”

BEN SHIRLEY, SALFORD UNIVERSITY

’We put in a couple of additional ones: an Eigenmike microphone, which is a 32-capsule surround-sound microphone that is very accurate, and two Soundfield mics that are lower-order surround-sound mics.

’Restrictions meant we couldn’t put more in. In addition to that, when you’re trying to identify where on the pitch the sound is coming from, you have mics that can be triangulated and are reasonably accurate. However, despite being well spaced out around the pitch, you’ve got 47,000 people immediately behind them shouting at the top of their voice, which isn’t ideal.’

Despite this, the team developed methods for extracting audio objects off the pitch without the crowd noise, thus pinpointing where the players were during the game.

This in turn has led to aspects of the audio side finding new applications.

Following a conference, two broadcast companies expressed an interest in automating the mixing process for live sound.

’So we’re pursuing that as a separate spin-out from the project as it provides benefits to existing broadcast workflows,’ said Shirley.

The team has yet to decide where the next trial will take place.

Shirley said that between 20 and 25 events have been suggested and that the final decision will be based on TV rights issues and how challenging it will be to deliver video and audio capture.

’It has to be a fairly high-profile event so it’ll attract interest from industry and the general public in terms of dissemination,’ said Shirley.
The project concludes in 2013 and includes input from partners including the BBC, Technicolor, Alcatel Lucent, Fraunhofer HHI, Joanneum Research and TNO.