Frozen Language Model

The defining characteristic is that all weights of the pre-trained language model (e.g., BERT, RoBERTa, GPT) are locked and not updated via gradient descent during the downstream navigation task training. This contrasts with fine-tuning, where model parameters are adjusted. The FLM acts as a feature extractor, converting the natural language instruction into a fixed-dimensional semantic embedding that conditions the navigation policy.

The FLM's sole function is to encode the natural language instruction into a rich, contextual representation. This encoded instruction vector is then fused with visual features from the agent's current panorama (e.g., via a cross-modal transformer) to inform the action decision. Its output provides the semantic goal and spatial relations (e.g., 'turn left after the kitchen') that the visuomotor policy must execute.

Using a frozen model offers significant advantages:

• Reduced Memory Footprint: Only the navigation policy's weights require gradient storage, not the massive LM's.
• Faster Training: Backpropagation does not pass through the LM's deep architecture.
• Mitigates Overfitting: Prevents the large LM from overfitting to limited navigation trajectory data, preserving its general linguistic knowledge.
• Leverages Pre-training: Directly utilizes knowledge from web-scale text corpora without costly domain-specific LM retraining.

Because the LM is frozen, the instruction representation it produces becomes a static semantic bottleneck. The navigation policy must learn to interpret and ground this fixed representation. This can limit an agent's ability to perform instruction decomposition or iterative reasoning mid-trajectory, as the instruction encoding does not evolve based on visual context. Advanced architectures may use cross-modal attention after encoding to overcome this limitation.

FLMs are typically selected from models pre-trained on masked language modeling or causal language modeling objectives.

• BERT & Variants: Frequently used for their bidirectional context understanding.
• DistilBERT: A lighter, faster option for efficient deployment.
• Sentence Transformers: Models like Sentence-BERT, optimized for producing sentence embeddings.
• T5 Encoder: The encoder half of a T5 model can be used frozen for instruction encoding.

An FLM represents one point on the spectrum of LM adaptation:

• Frozen LM: All parameters fixed; only the navigation network is trained.
• Fine-Tuned LM: All LM parameters are updated end-to-end with the navigation loss. More expressive but prone to overfitting and computationally expensive.
• Prompt-Tuned / Adapter-Based LM: A middle ground where a small number of parameters (prompt tokens or adapter layers) are trained while the core LM remains frozen, offering a trade-off between adaptability and efficiency.

Vision-and-Language Navigation (VLN) is the core task where an embodied agent follows natural language instructions to navigate through a 3D environment using visual perception. The frozen language model acts as the instruction encoder in this pipeline, providing a semantic understanding of the goal that the agent's visual and policy networks must execute.

• Primary Inputs: A natural language instruction and a stream of egocentric visual observations.
• Core Challenge: Grounding linguistic concepts (e.g., 'turn left after the red sofa') to specific visual features and actionable paths.
• Benchmarks: Includes datasets like Room-to-Room (R2R), REVERIE, and Touchdown.

A Language-Conditioned Policy is a neural network controller that outputs actions (e.g., move forward, turn left) based on both the current visual observation and an embedded natural language instruction. The frozen language model is typically used to generate the fixed instruction embedding that conditions this policy.

• Architecture: Often a recurrent network (LSTM) or transformer that fuses visual features with language embeddings.
• Training: The policy is trained via imitation learning or reinforcement learning, while the language encoder's weights remain frozen.
• Output: A probability distribution over low-level navigation actions or waypoint coordinates.

Cross-Modal Alignment is the learning objective that forces the representations of visual scenes and linguistic instructions into a shared semantic space. Even with a frozen language model, the visual encoder and fusion modules are trained to align their outputs with the pre-computed language embeddings.

• Mechanism: Achieved through contrastive loss functions (e.g., InfoNCE) or attention-based fusion in a Cross-Modal Transformer.
• Purpose: Enables the agent to determine if a visual observation is relevant to the current instruction step.
• Result: The agent can 'ground' phrases like 'blue chair' to specific pixels in its panoramic view.

Instruction Grounding is the real-time process by which an agent maps semantic concepts and spatial relations from the language instruction to specific visual observations and actionable locations. The frozen language model provides the high-level semantic parsing that makes this grounding possible.

• Involves: Resolving references (anaphora), understanding spatial prepositions ('between', 'beyond'), and identifying object attributes.
• Output: A belief state about which observed landmark corresponds to the instruction's next directive.
• Evaluation: Measured by metrics like grounding success in addition to overall navigation success.

Behavior Cloning for Navigation is a supervised imitation learning approach where an agent's policy is trained to mimic expert action sequences from demonstration trajectories. A frozen language model is a standard component in this paradigm, providing consistent instruction features without the risk of catastrophic forgetting during policy training.

• Data: Relies on Trajectory-Instruction Pairs where human demonstrators have recorded optimal paths.
• Advantage: Simpler and more sample-efficient than reinforcement learning for initial policy learning.
• Limitation: Can suffer from covariate shift if the agent deviates from the demonstrated state distribution.

Success weighted by Path Length (SPL) is the primary quantitative metric for evaluating navigation agents. It rigorously measures the efficiency of an agent using a frozen language model or any other architecture.

• Formula: SPL = (1 / N) * Σ (S_i * (L_i / max(P_i, L_i)))
  • S_i: Success (1 or 0) for trial i.
  • L_i: Length of the optimal (shortest) path.
  • P_i: Length of the agent's predicted path.
• Interpretation: Penalizes agents that reach the goal but take unnecessarily long routes. A perfect score of 1.0 indicates optimal, successful navigation on all trials.
• Standard: The definitive metric for benchmarks like R2R and REVERIE.