Sign system

A system designed by scientists at Oxford and Leeds universities can learn British Sign Language (BSL) signs from overnight television broadcasts by matching subtitled words to the hand movements of an on-screen interpreter.

The work is a crucial step towards a system that can automatically recognise BSL signs and translate them into words.

A major challenge in recognising signs is tracking the signer’s hands as they move on the broadcast. This is no mean feat, as they can get lost in the background, blur or cross, and the arms can assume a vast number of configurations.

The new system tackles this problem by overlaying a model of the upper body onto the video frames of the signer, looking for probable configurations, finding the large number of frames where these can be correctly identified and then ‘filling in the gaps’ to infer how the hands get from one position to another.

Another big challenge was to match a target word that appears in a subtitle to the corresponding sign – this is particularly difficult as words and signs often appear separated in time and words can be signed in many different ways so the corresponding sign may not appear at all.