Original date published: March 29, 2024

Foundation Models for Robotics Series Part 4 - Robot Decision-Making, Planning and Control with Foundation Models

In this post, we will cover decision-making, planning and control, and how foundation models have the potential to serve as valuable tools for enhancing robotic capabilities.

In this blog post, we will cover decision-making, planning and control, and how foundation models have the potential to serve as valuable tools for enhancing robot learning capabilities. Vision-language goal-conditioned policy learning, whether through imitation learning or reinforcement learning, holds promise for improvement using foundation models. Language models also play a role in offering feedback for policy learning techniques. This feedback loop can foster continual improvement in robot decision-making as robots refine their policies based on the feedback provided by the language model. An example of such work is the Eureka model, which uses a human-level reward-design algorithm powered by LLMs to train a robot policy using reinforcement learning [2].

Eureka Architecture — Eureka takes the unmodified environment code and the high-level task description in natural language and continually iterates to improve the reward code for training the RL algorithm [2].

In this post, we will discuss robot policy learning and language-image goal-conditioned value learning. We will cover current standards for training such models, as well as how foundation models have been applied to improve the performance of these model architectures.

Robot Policy Learning for Decision Making and Control

In this section, we discuss robot policy learning, including language-conditioned imitation learning and language-assisted reinforcement learning.

Language-Conditioned Imitation Learning: In language-conditioned imitation learning, a goal-conditioned policy is learned that outputs actions conditioned on the current state and language instructions. The model is trained using demonstrations, and the loss is calculated as the log-likelihood of selecting the right action at a particular time step according to the policy conditioned on the current state and language instruction. The demonstrations can be represented as trajectories or sequences of images and RGB-D voxel observations. Language instructions are then paired with demonstrations to create the training dataset. Each language-annotated demonstration consists of a state, a language instruction and an action. Challenges in this domain include obtaining a sufficient volume of demonstrations and conditioning labels to train a policy, as well as distribution shift under a closed-loop policy. Policy feedback can lead the robot into regions of the state space that are not well covered in the training data, negatively impacting performance. One approach to the demonstration-data annotation challenge in language-conditioned imitation learning is to use foundation models. This involves gathering a diverse set of trajectories and using a foundation model to annotate the expected language instruction for performing each trajectory. CLIPort presents a language-conditioned imitation learning policy for vision-based manipulation [4]. CLIPort is a language-conditioned imitation-learning agent that combines the broad semantic understanding (what) of CLIP with the spatial precision (where) of TransporterNets [5]. CLIPort is an end-to-end framework that solves language-specified manipulation tasks without any explicit representation of object poses or instance segmentation, but it is limited to 2D observations and action spaces. PerAct (Perceiver-Actor) aims to mitigate this limitation by representing observations and action spaces with 3D voxels [6]. PerAct takes language goals and RGB-D voxel observations as inputs to a Perceiver Transformer and outputs discretized actions by detecting the next-best voxel action. Voltron presents a framework for language-driven representation learning in robotics [7]. Voltron captures semantic, spatial and temporal representations learned from videos and actions. Deploying robot policy-learning techniques that leverage language-conditioned imitation learning with real robots presents ongoing challenges; they are susceptible to issues related to generalization and distribution shift. Techniques such as data augmentation and domain adaptation can improve robustness and adaptability by making policies more resilient to distribution shift. Beyond language, recent work has investigated other forms of task specification, such as MimicPlay, which uses human play data and low-level motor commands to condition the robot policy [8]. This allows the MimicPlay model to learn to perform new tasks based on one human video demonstration at test time.
Language-Assisted Reinforcement Learning: Reinforcement learning is a family of methods that enables a robot to optimize a policy through interaction with its environment by optimizing a reward function. Unlike imitation learning, reinforcement learning does not require a prior dataset of demonstrations and can theoretically reach superhuman performance since it does not rely on human samples to understand its environment. Foundation models have been applied to reinforcement learning algorithms in recent research, which we will cover in this section. Recent work by Google DeepMind proposes a unified RL architecture for foundation models that unifies reinforcement learning, VLMs and LLMs. Their work focuses on addressing core RL challenges related to exploration, experience reuse and transfer, skill scheduling and learning from observation. The researchers use an LLM to decompose complex tasks into simpler subtasks, which are then used as inputs for a transformer-based agent that interacts with the environment.

Unified Agent Architecture — High-level architecture of a unified agent that combines reinforcement learning with foundation models (VLM and LLM).

Language-Image Goal-Conditioned Value Learning

In value learning, the aim is to construct a value function that aligns goals in different modalities and preserves temporal coherence. Value-Implicit Pre-training (VIP) employs time-contrastive learning to capture temporal dependencies in videos, but does not require video-language alignment [10]. VIP focuses on robot manipulation tasks. Language-Image Value Learning (LIV) is a control-centric vision-language representation that generalizes the prior work, VIP, by learning multimodal vision-language value functions and representations using language-aligned videos [11]. The LIV model is pre-trained on arbitrary video activity datasets with text annotations, and the model can be fine-tuned on small datasets of in-domain robot data to ground language in a context-specific way. Both VIP and LIV learn a self-supervised goal-conditioned value function objective using contrastive learning. Value functions can be used to help ground semantic information obtained from an LLM in the physical environment in which a robot is operating. In SayCan [12], researchers investigate the integration of LLMs with the physical world through learning. They use the language model to provide task grounding (Say), enabling the determination of useful subgoals based on high-level instructions, and a learned affordance function to achieve world grounding (Can), enabling the identification of feasible actions to execute the plan.

Conclusion In this part of the series, we considered how foundation models have been applied to imitation, reinforcement and representation learning in robotics. We considered how foundation models such as LLMs and VLMs have been used to guide policy creation and how foundation models have been combined with existing policies to mitigate distribution shift and generalization issues. We also considered how policies could be used to guide the output generation of foundation models.

References

Last updated at: March 29, 2024