The Toyota Research Institute (TRI) is using generative artificial intelligence (AI) a to teach robots new, dexterous skills.

This advancement improves robot utility and is a step towards building large behaviour models (LBMs) for robots, analogous to the large language models (LLMs) that have recently revolutionised conversational AI.

“Our research in robotics is aimed at amplifying people rather than replacing them,” said Gill Pratt, CEO of TRI and chief scientist for Toyota Motor. “This new teaching technique is both very efficient and produces very high performing behaviours, enabling robots to much more effectively amplify people in many ways.”

Previous techniques to teach robots new behaviour were slow, inconsistent, inefficient and often limited to narrowly defined tasks performed in constrained environments. Roboticists needed to spend many hours writing sophisticated code and/or using numerous trial and error cycles to programme behaviour.

TRI has already taught robots more than 60 difficult, dexterous skills using the approach, including pouring liquids, using tools and manipulating deformable objects. These achievements were realised without writing a single line of code; the only change was supplying the robot with new data. Building on this success, TRI has set an ambitious target of teaching hundreds of new skills by the end of the year and 1000 by the end of 2024.

Today’s news also highlights that robots can be taught to function in new scenarios and perform a wide range of behaviour. These skills are not limited to just pick and place or simply picking up objects and putting them down in new locations. TRI’s robots can now interact with the world in varied and rich ways, which should one day allow robots to support people in everyday situations and unpredictable, ever-changing environments.

“The tasks that I’m watching these robots perform are simply amazing,” said Russ Tedrake, vice president of robotics research at TRI. “Even one year ago, I would not have predicted that we were close to this level of diverse dexterity.”

Tedrake is also the Toyota professor of electrical engineering and computer science at MIT.

“What is so exciting about this new approach is the rate and reliability with which we can add new skills,” he said. “Because these skills work directly from camera images and tactile sensing, using only learned representations, they are able to perform well even on tasks that involve deformable objects, cloth and liquids, all of which have traditionally been extremely difficult for robots.”

TRI and collaborators at Columbia University developed a generative-AI approach to behaviour learning. This approach, called diffusion policy, enables easy and rapid behaviour teaching from demonstrations.

The robot behaviour model learns from haptic demonstrations from a teacher, combined with a language description of the goal. It then uses an AI-based diffusion policy to learn the demonstrated skill. This process allows a new behaviour to be deployed autonomously from dozens of demonstrations. Not only does this approach produce consistent, repeatable, and performant results, but it does so with tremendous speed.

The robot platform is custom-built for dexterous dual-arm manipulation tasks with a focus on enabling haptic feedback and tactile sensing.

Part of the not-so-secret sauce is Drake, a model-based design for robotics that provides a toolbox and simulation platform. Drake’s high degree of realism allows development in simulation and in reality at an increased scale and velocity than would otherwise be possible. The internal robot stack is built using Drake’s optimisation and systems frameworks, Drake has been made open source to catalyse work across the entire robotics community.

Safety is core to robotics efforts at TRI. It has designed the system with strong safeguards, powered by Drake and a custom robot control stack to ensure the robots respect safety guarantees such as not colliding with themselves or their environment.

TRI (tri.global) conducts research to amplify human ability, focusing on making lives safer and more sustainable. The team of researchers develops technologies to advance energy and materials, human-centred AI, human interactive driving, machine learning, and robotics. Established in 2015, TRI has offices in Los Altos, California, and Cambridge, Massachusetts.