AI system gives highly precise recognition of hand gestures

According to NTU Singapore, gesture recognition precision is currently hampered by the low quality of data arriving from wearable sensors, often due to their bulkiness and poor contact with the user, plus the effects of visually blocked objects and poor lighting. Further challenges arise from the integration of visual and sensory data as they represent mismatched datasets that must be processed separately and then merged at the end, which is inefficient and leads to slower response times.

FingerTrak wearable captures human hands in 3D

US team controls robot with brainwaves and hand gestures

The NTU team’s 'bioinspired' data fusion system uses skin-like stretchable strain sensors made from single-walled carbon nanotubes, and an AI approach that resembles the way that the skin senses and vision are handled together in the brain.

The NTU scientists developed their bio-inspired AI system by combining three neural network approaches in one system: a 'convolutional neural network', which is a machine learning method for early visual processing, a multilayer neural network for early somatosensory information processing, and a 'sparse neural network' to 'fuse' the visual and somatosensory information together.