Original date published: March 24, 2024

Foundation Models for Robotics Series Part 3 - Types of Foundation Models

The third post in this series covers the various types of foundation models that are commonly used in robotics

In the previous post in this series, we looked at the foundational concepts and building blocks used to architect and train foundation models. This blog post focuses on the types of foundation models used in robotics. We divide these foundation models into specific categories and define the properties and examples of each foundation model type.

Large Language Models (LLMs)

LLMs are the most common type of foundation model used today. They have billions of parameters and are trained on trillions of text tokens. This large scale allows these models to achieve state-of-the-art performance on the General Language Understanding Evaluation (GLUE) benchmark [2]. GPT-2 [3] and BERT [4] are early examples of this model type. GPT-2 was trained as a decoder-only model, while BERT is an encoder-only model. Decoder models are more common today because they perform next-token prediction given an existing sequence of previous tokens. Successors of this decoder model architecture include GPT-3 [5], LLaMA [6] and PaLM [7]. The number of parameters in these models has grown significantly, reaching up to trillions of parameters, as with GPT-4 [8]. GPT-3 was trained on Common Crawl, which contains petabytes of publicly available data from over 12 years of web crawling [9]. After training, these language models are fine-tuned for chat and instruction following. GPT-3 and GPT-4 use a process known as reinforcement learning with human feedback (RLHF), which involves a human instructor guiding the model towards a desired output pattern by providing prompts and rating the outputs generated by the model [10].

Vision Transformers (ViT)

A Vision Transformer is a transformer architecture used for computer vision tasks such as object detection, image classification and segmentation [11]. The ViT architecture splits an image into patches and treats each patch as a token. To maintain spatial relationships between patches in the image, positional information is added to each token; this process is known as positional embedding. The tokens and their positional embeddings are fed as a sequence to the transformer encoder, allowing the self-attention mechanism to capture dependencies and patterns in the image sequence. Similar to LLMs, ViT models have also been scaled to large numbers of parameters, such as ViT-22B, which is a vision transformer with 22 billion parameters [12]. DINO is a self-supervised learning method for training ViT models [13]. It uses knowledge distillation, a learning framework where a smaller model is trained to mimic the behavior of a larger teacher model. Self-supervised ViT features learned using DINO contain explicit information about the semantic segmentation of an image, including scene layout and object boundaries; such clarity is not achieved using supervised ViTs or Convolutional Neural Networks (CNNs).

Multimodal Vision-Language Models (VLMs)

As discussed in previous posts, multimodal refers to the ability of a model to accept different “modalities” of input, such as images, text, audio or video signals. Vision-language models are a type of multimodal model that takes both images and text. The most commonly used VLM is the Contrastive Language-Image Pre-training (CLIP) model [14]. As the name implies, CLIP was trained using contrastive learning, as discussed in the previous post. Other CLIP variants used to build VLMs include BLIP [15], $CLIP^2$ [16], FILIP [17] and FLIP [18]. They all introduce unique variations to the training or similarity-matching process between text and images.

Embodied Multimodal Language Models

An embodied agent is an AI system that interacts with a virtual or physical world; examples include virtual assistants, game characters and robots. Embodied language models are foundation models that incorporate real-world sensor and actuation modalities into pre-trained large language models. For example, PaLM-E [19] is a multimodal language model trained simultaneously on internet-scale general vision-language data and embodied robotics data. PaLM-E is built from an LLM and a ViT. The ViT transforms an image into a sequence of embedding vectors, which are projected into the language embedding space through an affine transformation. The whole model is trained end-to-end, starting from a pre-trained LLM and ViT model. The outputs of the model are then connected to a robot for control through a high-level control policy.

Visual Generative Models

Diffusion models trained on web-scale data, such as DALL-E and Stable Diffusion, provide zero-shot text-to-image generation. They are trained on hundreds of millions of image-caption pairs from the internet. The models learn language-conditioned distributions over images, from which an image can be generated given a prompt.

Conclusion

In this post, we gave a brief overview of the various foundation model types used in robotics. We did not explicitly cover how each has been applied in robotics applications, but gave a high-level overview of how these models are trained and used. In the next part of this series, we will cover core concepts and terminology in AI-driven robotics and discover how these foundation models can fit into the robotics stack.

References

Last updated at: March 24, 2024