Vision-Language Models Provide Promptable Representations for Reinforcement Learning (2024)

4.1 Promptable Representations

Why do we choose to use VLMs in this way, instead of the many other ways of using them for control? In principle, one can directly query a VLM to produce actions for a task given a visual observation. While this may work when high-level goals or subtasks are sufficient, VLMs are empirically bad at yielding the kinds of low-level actions used commonly in RL (Huang etal., 2022a). As VLMs are mainly trained to follow instructions and answer questions about visual aspects of images, it is more appropriate to use these models to extract semantic features about observations that are conducive to being linked to actions. We thus elicit features that are useful for the downstream task by querying these VLMs with task-relevant prompts that provide contextual task information, thereby causing the VLM to attend to and interpret appropriate parts of observed images. Extracting these features naïvely by only using the VLM’s decoded text has its own challenges: such models often suffer from both hallucinations (Ji etal., 2023) and an inability to report what they “know” in language, even when their embeddings contain such information (Kadavath etal., 2022; Hu & Levy, 2023). However, even when the text is bad, the underlying representations still contain valuable granular world information that is potentially lost in the projection to language (Li etal., 2021; Wiedemann etal., 2019; Huang etal., 2023; Li etal., 2023b). Thus, we disregard the generated text in our approach and instead provide our policy the embeddings produced by the VLM in response to prompts asking about relevant semantic features in observations instead.

Which parts of the network can be used as promptable representations? The VLMs we consider are all based on the Transformer architecture (Vaswani etal., 2017), which treats the prompt, input image(s), and decoded text as token sequences. This architecture provides a source of learned representations by computing embeddings for each token at every layer based on the previous layer’s token embeddings. In terms of the generative VLM formalism introduced prior, a Transformer-based VLM’s representations $\phi_{t}(I,c,x_{1:t-1})$ consist of $N$ embeddings per token (the outputs of the $N$ self-attention layers) in the input image $I$, prompt $c$, and decoded text $x_{1:t-1}$. The decoder $p(x_{t}|\phi_{t})$ extracts the final layer’s embedding of the most recent token $x_{t-1}$, projecting it to a distribution over the token vocabulary and allowing for it to be sampled. When given a visual observation and task prompt, the tokens representing the prompt, image, and answer consequently encode task-relevant semantic information. Thus, for each observation, we use the VLM to sample a response to the task prompt $x_{1:K}\sim p(x_{1:K}|I,c)$. We then use some or all of these token embeddings $\phi_{K}(I,c,x_{1:t-1})$ as our promptable representations and feed them, along with any non-visual observation information, as a state representation into our neural policy trained with RL.

In summary, our approach involves creating a task-relevant prompt that provides context and auxiliary information. This prompt, alongside the current visual observation from the environment, is fed to into the VLM to generate tokens. While these tokens are used for decoding, they are ultimately discarded. Instead, we utilize the representations produced by the VLM (associated with the image, prompt, and decoded text) as input for our policy, which is trained via an off-the-shelf online RL algorithm to produce appropriate actions. A schematic of our approach is depicted in Figure 2.

4.2 Design Choices for Instantiating PR2L

To instantiate our method, several design choices must be made. First, the representations of the VLM’s decoded text depend on the chosen decoding scheme: greedy decoding is fast and deterministic, but may yield low-probability decoded tokens; beam search improves on this by considering multiple “branches” of decoded text, at the cost of requiring more compute time (for potentially small improvements); lastly, sampling-based decoding can quickly yield estimates of the maximum likelihood answer, but at the cost of introducing stochasticity, which may increase variance. Given the inherent high-variance of our tasks (due to sparse rewards and partial observability) and the expense of VLM decoding, we opt for greedy decoding.

Second, one must choose which VLM layers’ embeddings to utilize in the policy. While theoretically, all layers of the VLM could be used, pre-trained Transformer models tend to encode valuable high-level semantic information in their later layers (Tenney etal., 2019; Jawahar etal., 2019). Thus, we opt to only feed the final few layers’ representations into our policy. As these representation sequences are of variable length, we incorporate an encoder-decoder Transformer layer in the policy. At each time step in a trajectory, this layer receives variable-length VLM representations, which are attended to and converted into a fixed-length summarization by the embeddings of a learned “CLS” token(Devlin etal., 2019) in the decoder (green in Figure 2). We also note that this policy can receive the observed image directly (e.g., after being tokenized and embedded by the image encoder), so as to not lose any visual information from being processed by the VLM. However, we do not do this in our experiments in order to more clearly isolate and demonstrate the usefulness of the VLM’s representations in particular.

Finally, while it is possible to fine-tune the VLM for RL end-to-end with the policy, akin to what was proposed by Brohan etal. (2023), this incurs substantial compute, memory, and time overhead, particularly with larger VLMs. Nonetheless, we find that our approach performs better than not using the language and prompting components of the VLM. This holds true even when the VLM is frozen, and only the policy is trained via RL, or when the decoded text occasionally fails to answer the task-specific prompt correctly.

4.3 Task-Relevant Prompt

How do we design good prompts to elicit useful representations from VLMs? As we aim to extract good state representations from the VLM for a downstream policy, we do not use instructions or task descriptions, but task-relevant prompts: questions that make the VLM attend to and encode semantic features in the image that are useful for the RL policy learning to solve the task(Borja-Diaz etal., 2022). For example, if the task is to find a toilet within a house, appropriate prompts include “What room is this?” and “Am I likely to find a toilet here?” Intuitively, the answers to these questions help determine appropriate actions (e.g., look around the room or explore elsewhere), making the corresponding representations good for representing the state for a policy. Answering the questions will require the VLM to attend to task-relevant features in the scene, relying on the model’s internal conception of what things look like and common-sense semantic relations.Note that prompts based on instructions or task descriptions do not enjoy the above properties: while the goal of those prior methods is to be able to directly query the VLM for the optimal action, the goal of task-relevant prompts is to produce a useful state representation, such that running RL optimization on them can accelerate learning an optimal policy. While the former is not possible without task-specific training data for the VLM in the control task, the latter proves beneficial with off-the-shelf VLMs. Finally, these prompts also provide a place where auxiliary helpful information can be provided: for example, one can describe what certain entities of interest look like, aiding the VLM in detecting them even if they were not commonly found in the model’s pre-training data.

Evaluating and optimizing prompts for RL. Since the specific information and representations elicited from the VLM are determined by the prompt, we want to design prompts that produce promptable representations that maximize performance on the downstream task. The brute-force approach would involve running RL with each candidate prompt to measure its efficacy, but this would be computationally very expensive. In lieu of this, we evaluate candidate prompts on a small dataset of observations labeled with semantic features of interest for the considered task. Example features include whether task-relevant entities are in the image, the relative position of said entities, or even actions (if expert demonstrations are available). We test prompts by querying the VLM and checking how well the resulting decoded text for each image matches ground truth labels. As this is only practical for small, discrete label spaces that are easily expressed in words, we also draw from probing literature (Shi etal., 2016; Belinkov & Glass, 2019) and see how well a small model can fit the VLM’s embeddings to the labels, thus measuring how extractable said features are from the promptable representations (without memorization). While this approach does not directly optimize for task performance, it does act as a proxy that ensures a prompt’s resulting representations encode certain semantic features which are helpful for the task.

5.1 Domain 1: Minecraft

We first conduct experiments in Minecraft, which provides control tasks that require associating visual observations with rich semantic information to succeed. Moreover, since these observations are distinct from the images in the the pre-training dataset of the VLM, succeeding on these tasks relies crucially on the efficacy of the task-specific prompt in meaningfully affecting the learned representation, enabling us to stress-test our method. E.g., while spiders in Minecraft somewhat resemble real-life spiders, they exhibit stylistic exaggerations such as bright red eyes and a large black body. If the task-specific prompt is indeed effective in informing the VLM of these facts, it would produce a representation that is more conducive to policy learning and this would be reflected in task performance.

Minecraft tasks. We consider all programmatic Minecraft tasks evaluated by Fan etal. (2022): combat spider, milk cow, shear sheep, combat zombie, combat enderman, and combat pigman¹¹1Fan etal. (2022) also consider hunt cow/sheep. However, we omit them as we were unable to replicate their results on those tasks; all approaches failed to learn them..The remaining tasks considered by Fan etal. (2022) are creative tasks, which do not have programmatic reward functions or success detectors, so we cannot directly train RL agents on them. We follow the MineDojo definitions of observation/action spaces and reward function structures for these tasks: at each time step, the policy observes an egocentric RGB image, its pose, and its previously action; the policy can choose a discrete action to turn the agent by changing the agent’s pitch and/or yaw in discrete increments, move, attack, or use a held item. These tasks are long horizon, with a maximum episode length of 500 - 1000 and taking roughly $200$ steps for a learned policy to complete them. See Figure 3 for example observations and Appendix B.1 for more details.

Comparisons. To answer the first two questions from the start of this section, we compare our approach to: (a) methods utilizing non-promptable representations of visual observations and (b) methods that directly “asks” the VLM to output actions to execute on the agent. Comparison (b) attempts to adapt the approach of Brohan etal. (2023) to our setting and directly outputs the action from the VLM. While Brohan etal. (2023) also fine-tune the VLM backbone, we are unable to do so using our compute resources. To compensate, we do not just execute the action from the VLM, but train an RL policy to map this decoded output action into a better one. Note that, if the VLM already decodes good action texts for the specified task, then simply copying over this action via RL should be easy. Finally, comparison (a) does not utilize the task-specific prompt altogether, instead using task-agnostic embeddings from the VLM’s image encoder (specifically, the ViT-g/14 from InstructBLIP – blue in Figure 2). While this representation of the observation still benefits from pre-training, PR2L utilizes prompting to produce task-specific representations. For a fair comparison, we use the exact same Transformer-layer policy architecture and hyperparameters for this baseline as in PR2L, ensuring that performance differences come from prompting for better representations from the VLM. We use PPO (Schulman etal., 2017) as our base RL algorithm for all Minecraft comparisons. For more details, see Appendix B.2.

5.2 Domain 2: Habitat

PR2L augments state representations with additional useful features, elicited by prompting the VLM. It does not add any sort of exploration incentive. We thus expect our approach to provide the most benefit on tasks that are not bottlenecked by exploration. To highlight this, we run offline RL experiments in the Habitat household simulator. In contrast to Minecraft, tasks in this domain require connecting natural images with real-world common sense – namely, about the structure and contents of typical home environments. Moreover, it provides tools for automatically generating high-quality training data for offline learning (Ehsani etal., 2023). Habitat therefore lets us evaluate the feasibility of prompting VLMs for representations of knowledge or useful abstractions about the world that may aid in learning.

Habitat tasks. We consider the ObjectNav suite of tasks in 3D reconstructed household scenes from the HM3D dataset (Savva etal., 2019; Yadav etal., 2023; Ramakrishnan etal., 2021). These tasks involve a simulated robot traversing a home environment to find an instance of a specified object in the shortest path possible. We consider three such objects: toilets, beds, and sofas. We adopt the same reward structure as Yadav etal. (2023), but change the observation space to consist of just RGB vision, previous action, pose, and target object class, omitting depth images to better ensure that experimental performance differences come from the quality of promptable representations vs. unpromptable ones. Like with MineDojo, these tasks are long horizon, taking 80 steps for a privileged shortest path follower to succeed and $200+$ for humans. See Figure 3 for example observations and Appendix C for more details.

Comparisons. Our primary comparison is once again between our promptable representations and general-purpose non-promptable ones. We thus repeat the baseline described previously for Minecraft in Section 5.1, training a single agent for all three ObjectNav tasks using both PR2L and the VLM image encoder representations. We empirically note that longer visual embedding sequences tend to perform better in Habitat. To control for this, we opt to use InstructBLIP’s Q-Former unprompted embeddings instead of the ViT embeddings directly (which are much longer than PR2L’s embedding sequences). As InstructBLIP uses the former representations to extract visual features to be projected into language embedding space, this serves to close the gap in embedding sequence length between our two conditions while still providing us with general visual features that the VLM processes via prompting. We likewise fix all training hyperparameters and offline data to ensure that performance differences come from the quality of state representations, as yielded by prompting. We use CQL with QR-DQN as our offline RL algorithm (Kumar etal., 2020; Dabney etal., 2017). Following Ehsani etal. (2023), we use Habitat’s built-in shortest path follower to generate training data, though add noise to ensure that our dataset contains a mixture of good, mediocre, and failed trajectories. See Appendices C.1 and C.2 for additional details.

5.3 Designing Task-Specific Prompts for PR2L

We now discuss how to design prompts for PR2L. As noted in Section 4.3, these are not instructions or task descriptions, but prompts that force the VLM to encode semantic information about the task in its representation. The simplest relevant feature for our Minecraft tasks is the presence of the target entity in an observation. Thus, we choose “Is there a [target entity] in this image?” as the base of our chosen prompt. We also pick two alternate prompts per task that prepend different amounts of auxiliary information about the target entity. E.g., for combat spider, one candidate is “Spiders in Minecraft are black.” To choose between these candidate prompts, we measure how well the VLM is able to decode a correct answer to the prompt question of whether or not the target entity is present in the image on a small annotated dataset. Full details of this prompt evaluation scheme for the first three Minecraft tasks are presented in Appendix A and Table 3. We find that auxiliary text only helps with detecting spiders while systematically and significantly degrading the detection of sheep and cows. Our ablations show that this detection success rate metric correlates with performance of the RL policy. Additionally, the prompts used for comparison (b) follow the prompt structure prescribed by Brohan etal. (2023), which motivated this comparison. In these prompts, we also provide a list of actions that the VLM can choose from to the policy. All chosen prompts are presented in Table 1.

For Habitat, we note that ObjectNav would benefit from including the type of room in the state representation, as it informs actions at a high level (e.g., if the agent is looking for a toilet, it is helpful to know when it is in a bathroom). We thus choose “What room is this?” as the prompt for all ObjectNav goal objects.

5.4 Results and Ablations

Minecraft results. We report the interquartile mean (IQM) and standard error number of successes over 16 seeds for all Minecraft tasks in Figure 4. As shown in Figure4, PR2L tends to outperform both using the VLM image encoder (comparison (a)) and the method that directly “asks” the VLM for the action (comparison (b)) inspired by RT-2. We provide an analysis of why PR2L states are better than RT-2 ones in Appendix E.1. We observe that PR2L embeddings are bimodally distributed, with transitions leading to high reward clustered at one mode. This structure likely enables more efficient learning, thereby showing how control tasks can benefit from extracting prior knowledge encoded in VLMs by prompting them with task context, even in single-task cases where the generalization properties of instruction-following methods do not apply.

Habitat results. We report evaluation success rates and average returns for Habitat ObjectNav in Figure 5. PR2L achieves nearly double the average success rate of the baseline ($60.4\%$ vs. $35.2\%$), supporting the hypothesis that PR2L works especially well when exploration is not needed. Lastly, in Appendix E.2, we find that PR2L causes the VLM to produce highly structured representations that correlate with an expert policy’s value function: high-value states are typically labeled by the VLM as being from a room where one would expect to find the target object.

Ablations and additional baselines. We run three ablations on combat spider, milk cow, and shear sheep to isolate and understand the importance of various components of PR2L. First, we run PR2L with no prompt to see if prompting with task context actually tailors the VLM’s generated representations favorably towards the target task, improving over an unprompted VLM. Note that this is not the same as just using the image encoder (comparison (a)), as this ablation still decodes through the VLM, just with an empty prompt. Second, we run PR2L with our chosen prompt, but no generation of text – i.e., the policy only receives the embeddings associated with the image and prompt (the left and middle red groupings of tokens in Figure 2, but not the right-most group). This tests the hypothesis that representations of generated text might make certain task-relevant features more salient: e.g., the embeddings for “Is there a cow in this image?”, might not encode the presence of a cow as clearly as if the VLM generates “Yes” in response, impacting downstream performance. Finally, to check if our prompt evaluation strategy provides a good proxy for downstream task performance while tuning prompts for P2RL, we run PR2L with alternative prompts that were not predicted to be the best, as per our criterion in Appendix A. We thus remove the auxiliary text from the prompt for combat spider and add it for milk cow and shear sheep.

We also provide more baselines for these three tasks. First, as InstructBLIP’s image encoder may just be bad for control tasks, we run RL on the MineCLIP embeddings(Fan etal., 2022), which is obtained by fine-tuning CLIP(Radford etal., 2021) on Minecraft data. This serves as an “oracle” comparison, as this representation was explicitly fine-tuned on Minecraft YouTube videos, whereas our pre-trained VLM is both frozen and not trained on any Minecraft video data. We also train policies on embeddings from VC-1 and R3M – two image encoders that are specifically pretrained for embodied control (Majumdar etal., 2023; Nair etal., 2022). All three encoders produce a single embedding (not a sequence), so they share an architecture and hyperparameters. Lastly, to disambiguate whether PR2L is simply more performant because it can more reliably detect the task-relevant entity, we train a policy on top of both the VLM image encoder embeddings and an indicator of whether the entity is in view based on a privileged oracle semantic detector provided by MineDojo²²2As InstructBLIP uses its image encoder’s embeddings as its sole source of visual information, if the VLM is just doing better object detection, then any information about the entity’s presence must also be available to the image encoder baseline.. This baseline uses the same hyperparameters as the VLM image encoder baseline.

Results from these additional experiments are presented in Table 2. In general, all ablations perform worse than PR2L. For milk cow, we note the most performant ablation is no generation, perhaps because the generated text is often wrong; among the chosen prompts, it yields the lowest true positive and negative rates for classifying the presence of its corresponding target entity (see Table 3 in Appendix A), though adding auxiliary text makes it even worse, perhaps explaining why milk cow experienced the largest performance decrease from adding it back in. Based on these overall trends, we conclude that (i) the promptable and generative aspects of VLM representations are important for extracting good features for control tasks and (ii) our simple evaluation scheme is an effective proxy measure of how good a prompt is for PR2L.

While PR2L does not outperform the “oracle” MineCLIP policy on combat spider, it performs competitively or better than MineCLIP on the other two tasks and beats VC-1, R3M, and the entity detector baselines on all three. We hypothesize that MineCLIP outperforms PR2L on the spider task because, out of all the entities that we study, Minecraft spiders are the most different visually from real spiders, giving rise to comparatively poor representations in the VLM (which is trained exclusively on natural images). Still, our results in Table 2 show that PR2L provides an effective approach to transform a general-purpose VLM into a strong task-specific policy (without fine-tuning) that can often outperform policies trained on control-centric representations.

B.1 Environment Details

World specifications. MineDojo implements a fast reset functionality that we use. Instead of generating an entirely new world for each episode, fast reset simply respawns the player and all specified entities in the same world instance, but with fully restored items, health points, and other relevant task quantities. This lowers the time overhead of resets significantly, but also means that some changes to the world (like block destruction) are persistent. However, as breaking blocks generally takes multiple time steps of taking the same action (and does not directly lead to any reward), the agent empirically does not break many blocks aside from tall grass (which is destroyed with a single strike from any held item). We keep all reset parameters (like the agent respawn radius, how far away entities can spawn from the agent, etc) at their default values provided by MineDojo.

We stage all tasks in the same area of the same programmatically-generated world: namely, a sunflower plains biome in the world with seed 123. This is the default location for the implementation of the spider combat task in MineDojo. We choose this specific world/location as it represents a prototypical Minecraft scene with relatively easily-traversable terrain (thus making learning faster and easier).

Additional task details and reward functions. We provide additional notes about our Minecraft tasks.

Combat spider: Upon detecting the agent, the spider approaches and attacks; if the agent’s health is depleted, then the episode terminates in failure. The agent receives $+1$ reward for striking any entity and $+10$ for defeating the spider. We also include several distractor animals (a cow, pig, chicken, and sheep) that passively wander the task space; the agent can reward game by striking these animals, making credit assignment of success rewards and the overall task harder.

Milk cow: The agent also holds wheat in its off hand, which causes the cow to approach the agent when detected and sufficiently nearby. For each episode, we track the minimum visually-observed distance between the agent and the cow at each time step. The agent receives $+0.1|\Delta d_{\text{min}}|$ reward for decreasing this minimum distance (where $\Delta d_{\text{min}}\leq 0$ is the change in this minimum distance at a given time step) and $+10$ for successfully milking the cow.

Shear sheep: As with milk cow, the agent holds wheat in its off hand to cause the sheep to approach it. The reward function also has the same structure as that task, albeit with different coefficients: $+|\Delta d_{\text{min}}|$ for decreasing the minimum distance to the sheep and $+10$ for shearing it.

Combat zombie: Same as combat spider, but the enemy is a zombie. We increase the episode length to 1000, as the zombie has more health points than the spider.

Combat enderman: Same as combat spider, but the enemy is an Enderman. As with combat zombie, we increase the episode length to 1000. Note that Endermen are non-hostile (until directly looked at for sufficiently long or attacked) and have significantly more health points than other enemies. We thus enchant the agent’s sword to deal more damage and decrease the initial spawn distance of the enderman from the agent.

Combat pigman: Same as combat spider, but the enemy is a hostile zombie pigman. As with combat zombie, we increase the episode length to 1000.

B.2 Policy and Training Details

We also present the policy and VLM hyperparameters in Table 5. The hyperparameters and architecture of the MLP part of the policy are primarily defined by the default values and structure defined by the Stable-Baselines3 ActorCriticPolicy class. Note that the no generation ablation, VLM image encoder baseline, and MineCLIP trials do not generate text with the VLM, and so all do not use the associated process’s hyperparameters. The MineCLIP trials also do not use a Transformer layer in the policy, due to not receiving token sequence embeddings. It instead just uses a MLP, but with two hidden layers (to supplement the lowered policy expressivity due to the lack of a Transformer layer).

Additionally, InstructBLIP’s token embeddings are larger than ViT-g/14’s (used in the VLM image encoder baseline), and so may carry more information. However, the VLM does not receive any privileged information over the image encoder from the task environment – any additional information in the VLM’s representations is therefore purely from the model’s prompted internal knowledge. Still, to ensure consistent policy expressivity, we include a learned linear layer projecting all representations for this baseline and our approach to the same size (512 dimensions) so that the rest of the policy is the same for both.

C.1 Environment Details

The spaces and agent/task specifications are largely the same as the defaults provided by Habitat, as specified in the HM3D ObjectNav configuration file (Savva etal., 2019).

World specifications. In ObjectNav, an agent is spawned in a household environment and must find and navigate to an instance of a specified target object in as efficient a path as possible. Doing so effectively requires a common-sense understanding of where household objects are often found and the structure of standard homes.

We find that most Habitat learning algorithms require either significant system engineering (e.g., semantic SLAM systems, like in Luo etal. (2022)) or very large training datasets (Ramrakhya etal., 2023; Ehsani etal., 2023; Khanna etal., 2023). Such works also often use more complex policies that incorporate histories of observations and actions, either via recurrent or Transformer architectures. To better study the effects of promptability on representations, we choose to simplify the ObjectNav task in several ways.

We train and evaluate on ObjectNav episodes from the Habitat-Matterport 3D v2 dataset’s validation split (Ramakrishnan etal., 2021). We choose said dataset because its environments are the most realistic, being generated from real-world 3D scans. Likewise, they offer better visual fidelity than simulated counterparts (despite sometimes having reconstruction artifacts), allowing VLMs trained on naturalistic images to better parse observations. We pick 32 reconstructed 3D home environments with at least one instance of each of the three target objects (toilet, bed, and sofa) and an annotator quality score of at least 4 out of 5. We choose to remove plants and televisions from the goal object set due to finding numerous unlabeled instances of them. Additionally, we remove chairs, as they are significantly more common than other goal objects and thus usually can be found in much shorter episodes. This simplified problem formulation enables us to remove many of the “tricks” that aid ObjectNav, such as using omnidirectional views or policies with history; our agent makes action decisions purely based on its current visual observation and pose, allowing us to do “vanilla” RL to better isolate the effect of PR2L.

To generate data, we use Habitat’s built-in greedy shortest geodesic path follower. Imitating such demonstrations allows policies to learn unintuitively emergent and performant navigation behaviors (Ehsani etal., 2023) at scale. For each defined starting location in our considered households, we autonomously collect data by using the path follower to navigate to each reachable instance of the corresponding goal object. This yields high quality, near-optimal data. We then supplement our dataset by generating lower-quality data. Specifically, for each computed near-optimal path from a starting location to a goal object instance, we choose to inject action noise partway through the trajectory (uniformly at random from $0-90\%$ of the way through). At that point, all subsequent actions have a $0-50\%$ probability (again chosen uniformly at random) of being a random action other than the one specified by the path follower. To ensure that paths are sufficiently long, we choose to make the probability of choosing the stop action $10\%$ and the other two movement actions $45\%$. In total, we collect 107518 observations over 2364 trajectories.

Reward functions. The ObjectNav challenge evaluates agents based on the average ”success weighted by path length” (SPL) metric (Yadav etal., 2023): if an agent succeeds at taking the stop action while close to an instance of the goal object, it gets $SPL(p,l)=\frac{l}{\max(l,p)}$ points, where $l$ is the actual shortest path from the starting point to an instance of the goal object and $p$ is the length of the path that the agent actually took during that particular episode. If the agent stops while not close to the target object, the SPL is 0. Thus, taking the most efficient path to the nearest goal object and stopping yields a maximum SPL of 1.

We use this to design our reward function. Specifically, when the agent stops, it receives a reward of $+10SPL(p,l)$. Additionally, we add a shaping reward of the change in geodesic distance to the nearest goal object instance each time the agent moves (where lowering that distance yields a positive reward).

C.2 Policy and Training Details

For our offline RL experiments in Habitat, we use Conservative Q-Learning (CQL) on top of the Stable-Baslines3 Contrib codebase’s implementation of Quantile Regression DQN (QR-DQN). We choose to multiply the QR-DQN component of the CQL loss by $0.2$. Using the notation proposed by Kumar etal. (2020), this is equivalent to $\alpha=5$, which said work also uses. Other hyperparameters are $\tau=1$, $\gamma=0.99$, fixed learning rate of $1e-4$, 100 epochs, and 50 quantiles (no exploration hyperparameters are specified, since we do not generate any new online data).

The policy architecture used for Habitat experiments are the same as those used for PPO, though the final network outputs quantile Q-values for each action (rather than just a distribution over actions). The action with the highest mean quantile value is chosen at evaluation time.

During training, we shuffle the data and load full offline trajectories until the buffer has at least $32\times 1024=32768$ transitions or all trajectories have been loaded once that epoch. We then uniformly sample and train on batches of size 512 transitions from the buffer until each transition has been trained on once in expectation (e.g., $\sim\frac{\text{number of transitions in the buffer}}{512}$ batches). Each batch is used for 8 gradient steps before the next is sampled. We choose this data loading scheme to fit the training infrastructure provided by Stable-Baselines3 while not using up too much memory at once.

E.1 Minecraft Analysis

To investigate this, we use the embeddings of our offline data from the BC experiments (collected by training a MineCLIP encoder policy to high performance on all of our original three tasks, as discussed in Appendix D). We specifically look at the embeddings produced by a VLM when given our standard task-relevant prompts and when given the instructions used for our RT-2-style baseline. We then perform principal component analysis (PCA) on the tokenwise average of all embeddings for each observation, thereby projecting the embeddings to a 2D space with maximum variance.

We visualize these low-dimensional space in Figure 8 for the final 20 successful observations from each task, with the point colors of orange and blue respectively indicating whether the observation results in a functional action (attack or use item) or movement (translation or rotation) by the expert policy. Additionally, we enlarge points corresponding to when the agent received rewards in order to recognize which actions aided in or achieved the task objective.

We find that our considered prompts resulted in a bimodal distribution over representations, wherein the left-side cluster corresponds to the VLM outputting “yes (the entity is in view)” and the right-side one corresponds to “no.” Additionally, observations resulting in functional actions that received rewards (large orange points in Figure 8) tend to be on the left-side (“yes”) cluster for representations elicited by our prompt, but are more widely distributed in the instruction prompt case, in agreement with intuition. This is especially clear in the milk cow plot, wherein nearly all rewarding functional actions (using the bucket on the cow to successfully collect milk) are in the lower left corner.

This analysis supports that the representations yielded by InstructBLIP in response to our chosen style of prompts are more structured than representations from instructions. Such structure is useful in identifying and learning rewarding actions, even when said actions were taken from an expert policy trained on unrelated embeddings. This suggests that such representations may similarly be more conducive to being mapped to good actions via RL, which we observe empirically (as our prompt’s representations yield more performant policies than the instructions for the RT-2-style baseline).

E.2 Habitat Analysis

Likewise, we conduct a similar analysis on the Habitat data. Specifically, we wish to see if PR2L produces representations that are conducive to extracting the value function of a good policy. Since the chosen Habitat ObjectNav prompt is “What room is this?” we expect the state representations to be clustered based on room categories. Intuitively, states corresponding to the room one is likely to find the target object should have the highest values.

As shown in Figure 9, we thus used PCA to project expert trajectories’ PR2L and general image encoder state representations (generated with Habitat’s geodesic shortest path follower) to two dimensions, then colored each one based on their value under said near-optimal policy. We also plotted the mean and standard deviation of all points labeled as each room, visualizing them as axis-aligned bounding boxes. Note that each upper subplot in Figure 9 has a cluster of points far from all boxes. These correspond to the VLM generating nothing or garbage data with no room label.

This visualization qualitatively agrees with intuition. High value states tend to be grouped with the room the corresponding target object is often found in: find toilet corresponds to bathrooms, find bed to bedrooms, and find sofa to living rooms. Comparatively, the general image encoder features do not have such semantically meaningful groupings; all observations are clustered together and, within that single grouping, high-value observations are more spread out. This all supports the idea that prompting allows representations to take on structures that correlate well to value functions of good policies.