Robot chef mimics simple recipes from food videos

Using an off-the-shelf neural network, the Cambridge team trained the robot by showing it eight simple salad recipes in action, filmed by themselves. The neural network had already been programmed to identify a range of different objects, including the fruits and vegetables in the recipes like broccoli, carrot, apple and banana. Using computer vision, the robot was able to analyse the different objects in each frame of video, converting the recipes and actions to vectors which could then inform mathematical operations.  

Of the 16 videos it was shown, the robot recognised the correct recipe 93 per cent of the time, even though it only detected 83 per cent of the human chef’s actions. It was also able to detect that slight variations in a recipe - such as making a double portion - were not a new recipe. The robot also correctly recognised the demonstration of a new, ninth salad, added it to its ‘cookbook’ and made it. The work is published in IEEE Access.