Original date published: March 10, 2024

Foundation Models for Robotics Series Part 1 - Introduction

The first part of a series covering the use of Foundation Models in Robotics, in which we discuss application areas and the RT-2 model as an example

I'm starting my first blog series, which covers the use of foundation models in various areas of robotics. This series is intended to serve as a personal guide for my journey, as well as an overview for readers interested in discovering how to build and apply foundation models in robotics.

Foundation models are models that have been trained extensively on large, internet-scale amounts of data to solve a specific type of machine learning problem. They are not trained to be problem-specific, but to generalize to a wide range of problems within a machine learning approach [1]. The approach of training foundation models is really powerful because it allows us to fine-tune these base models for use in a particular downstream task, domain or use case while retaining their generalizability from pre-training.

Traditional deep learning models in robotics are trained on small datasets tailored to a specific task, which limits their ability to be used in diverse tasks. For example, Deep Reinforcement Learning, which has up until this point been the most popular approach for training deep learning models for robotics applications, allows a model to learn how to operate in a single environment or solve a single problem. It is impossible to use these models outside the specific domain they were trained for. Examples of this approach are AlphaGo [2] and OpenAI Five [3].

Foundation models enable us to train models that generalize to different applications, tasks and embodiments using the same foundation model.

Foundation models can be applied to robotics broadly in the following areas:

Perception (high-level planning and low-level planning)
Planning
Manipulation
Navigation

There have been research projects that have explored the use of foundation models in each of these areas. In this series, we will move between these application areas in no particular order.

Map of Applications of Foundation Models in Robotics [4].

The transformer architecture [5] has made it possible to build and scale foundation models in various areas by increasing the amount of compute hardware and/or data. The initial training process is then followed by post-training processes such as instruction tuning and, in some cases, RLHF (reinforcement learning with human feedback). This approach has become commonplace with the proliferation of large language models such as ChatGPT, LLaMA, Mistral and Phi. These LLMs can then be further tuned for more domain-specific applications while retaining their pre-training reasoning and generalizability.

In robotics, we make use of the following kinds of foundation models:

Large Language Models (LLMs): LLMs are a type of foundation model trained on a large corpus of text and used to produce language output through next-token prediction. The most common approach for building LLMs uses the decoder-only transformer architecture.
Vision-Language Models (VLMs): VLMs are foundation models trained to use both text and images as input. This is achieved through the use of contrastive learning to build mapping representations of both images and language tokens. These models also produce natural language as output through next-token prediction.
Vision-Language Action Models (VLAs): VLAs are foundation models similar to VLMs that are trained to use both images and text as input. The modification in this architecture is that they are also able to generate action tokens which describe a specific control instruction to an embodied agent either through a physical robot or software agent.
Audio-Language Models (ALMs): ALMs are foundation models that use a combination of audio sequences and text tokens to generate an output text sequence using next-token prediction.
Visual Navigation Models (VNMs): VNMs are foundation models that take an image stream as input and produce a navigation action sequence to reach a target location.

For starters, we will discuss RT-2 [6] as an example to guide future discussions about foundation models in robotics. RT-2 is a multimodal foundation model from Google DeepMind that uses both language and images as input, and it generates not just English language but robot language tokens for taking actions. RT-2 was trained on both language-image pairs and embodied robotics action data. It's important to highlight that an approach like RT-2 combines perception, semantic understanding (planning) and action into a single model. This allows the model to understand objects it has not seen before, understand broad concepts such as making coffee, and understand the low-level action sequence required to perform the manipulations required to “make coffee.” Let's break this down a bit further. RT-2 is one of the models that offers an end-to-end approach for robotics.

Perception is handled by the vision component in the VLA architecture. This uses object generalization learned from training on a large amount of images similar to CLIP [7] and PaLM-E [8]. Some other approaches build the perception module separately using a Vision Transformer (ViT) architecture [9] and use natural language as the communication medium. We will cover this in more detail in future articles in this series.

Planning is handled through the semantic understanding, reasoning and generalizability common in LLMs such as GPT-4. The model can be given an abstract problem or instruction and, provided with enough state context, can decide on the constraints and steps required to accomplish that task. This approach is also used to solve Navigation problems.

RT-2 plans based on high-level instruction [6].

Finally, Manipulation involves generating sequences that control the joints or provide pre-defined high-level instructions on how to execute the generated plan on the robot hardware. This is done through the 'action' part of the VLA, where the model is trained with special tokens which define actions that can be performed by the robot, and during the training process, the model builds a semantic understanding of each action and its effects.

RT-2 is a unique example in that it is one of the few approaches that integrate all these problems into a single model. As we discuss other approaches in this series, we will see how some projects chose to focus on a few of these problems by combining various foundation models together and defining a medium of communication between these models to execute an language instruction.

While RT-2 does not focus too much on its post-training processes, a logical next step for this approach would be to enable fine-tuning the RT-2 model for specific robot hardware embodiments (quadruped, humanoid, differential drive, etc.) and build a reward model to validate the model's task execution by following RLHF procedures [10].

I'm really excited about this series and want to get it right. There is so much to share and there are many downstream applications. These ideas can be applied beyond robotics and autonomous agents. I would also appreciate it if you could provide feedback on the blog post. I would like to hear your thoughts on the series. Did you learn something new? Should I go more technical? Should I go less technical? Should I add more references or discuss everything in the posts? Should I make the posts longer or shorter? I'd love to hear what you think. Thank you for making it this far. I hope you're as excited to read the next part of this series as I am to write it.

References

Last updated at: March 10, 2024