Prompt Transferability

The most significant factor influencing transferability is the semantic and structural relationship between the source and target tasks. High transferability is observed when tasks share:

• Similar output formats (e.g., both are classification or both are text generation).
• Overlapping latent concepts (e.g., sentiment analysis and toxicity detection both involve understanding emotional valence).
• Related domains (e.g., financial news summarization and earnings report summarization). Transferability degrades sharply when moving between fundamentally different task types, such as from named entity recognition to code generation.

A soft prompt's effectiveness is tied to the architectural specifics of the base model on which it was trained. Key alignment factors include:

• Embedding Space Geometry: Prompts tuned on one model (e.g., GPT-3) may not align with the embedding manifold of another (e.g., LLaMA), leading to poor performance.
• Layer Normalization & Activation Functions: Differences in how models process prefix vectors can cause distribution shifts.
• Vocabulary & Tokenization: Prompts are continuous vectors in a model-specific embedding space. Transfer between models with different tokenizers often requires prompt re-embedding or projection techniques to map the prompt into the target model's space.

A transferable prompt must exhibit strong generalization within its source task to have any hope of transferring. This is characterized by:

• Low prompt overfitting: The prompt should capture the task's abstract procedure, not memorize training examples.
• Invariance to input perturbations: The prompt should guide the model consistently despite variations in how the user query is phrased.
• Calibration across datasets: Performance should remain stable when evaluated on different datasets for the same task (e.g., moving from the SQuAD to the Natural Questions dataset for QA). Prompts that are brittle or over-specialized on their source data demonstrate poor transferability.

The length and parameter count of the soft prompt directly impact transferability.

• Shorter prompts (e.g., 20 virtual tokens) are highly parameter-efficient but may lack the expressive capacity to encode a complex task, limiting transfer to only very similar targets.
• Longer prompts (e.g., 100+ virtual tokens) can encode richer task semantics, potentially enabling transfer to a broader range of related tasks, but at the cost of increased prompt latency and a higher risk of overfitting. This trade-off must be optimized based on the intended transfer scope.

Transferability extends beyond text-to-text tasks. Advanced scenarios include:

• Cross-modal transfer: A prompt tuned for image captioning on a vision-language model (e.g., BLIP-2) may transfer weakly to video description if the underlying visual concepts are similar.
• Cross-lingual transfer: A soft prompt learned for English sentiment analysis might transfer to French sentiment analysis if the model is multilingual (e.g., mBERT, XLM-R), leveraging shared semantic representations across languages. These transfers are generally more challenging and require models with aligned multimodal or multilingual embedding spaces.

Transferability is quantified using specific evaluation protocols:

• Zero-shot transfer accuracy: Performance on the target task without any additional tuning.
• Few-shot adaptation speed: The number of gradient steps or examples needed to fine-tune the transferred prompt to match the performance of a prompt trained from scratch on the target task.
• Relative performance drop: The percentage decrease in performance (e.g., F1 score) when moving from the source to the target task, compared to a task-specific baseline.
• Transferability matrices: Research tools that systematically measure performance when transferring prompts between many task pairs to map the transferability landscape.

A soft prompt is a set of continuous, high-dimensional vector embeddings that are learned during training and prepended to the input of a frozen language model to condition its output for a specific task. Unlike a hard prompt, it is not composed of human-readable tokens.

• Core Mechanism: Acts as a tunable, task-specific context signal.
• Relation to Transferability: The learned embeddings are the object being transferred between tasks or models. Their quality and representation directly determine transferability.

Prompt generalization is the desired property of a tuned soft prompt to perform accurately on new, unseen examples from the task distribution it was trained on, without memorizing the training set. It is a prerequisite for transferability.

• Key Difference: Generalization applies to unseen data from the same task; transferability applies to different but related tasks, models, or datasets.
• Failure Mode: Prompt overfitting occurs when a soft prompt captures noise from the training data, harming both generalization and downstream transferability.

Prefix tuning is a parameter-efficient fine-tuning technique that prepends a sequence of trainable continuous vectors (the 'prefix') to the hidden states at every layer of a frozen pre-trained model. This is a more expressive variant of prompt tuning.

• Architecture: The trainable prefix influences the model's internal representations throughout its depth.
• Transferability Impact: Prefixes, by modifying deeper activations, may capture more abstract, transferable task representations than input-layer soft prompts, but can also become more tightly coupled to the specific base model's architecture.

A prompt pool is a set of multiple soft prompts that can be dynamically selected or combined during inference. This architecture is often used to manage knowledge across multiple tasks without interference.

• Use Case: Central to continual learning scenarios where a model must learn tasks sequentially.
• Transferability Link: The pool acts as a library of reusable skill modules. Transferability is studied by measuring how well a prompt trained for Task A can be retrieved and applied from the pool when a related Task B is presented.

Delta tuning is the general paradigm of adapting a pre-trained model by learning and applying a small parameter change (the 'delta') while keeping the original weights frozen. Prompt and prefix tuning are specific instances of delta tuning.

• Unifying Concept: Encompasses methods like Adapters, LoRA, and prompt-based techniques.
• Framework for Transferability: Studies the transferability of the delta itself—whether the small, learned adjustment for one task is beneficial when applied to a different, related task. This examines the modularity and composability of learned adaptations.

Multi-task prompt tuning involves learning a single soft prompt (or a shared set of prompts) concurrently on data from multiple related tasks. The goal is to create a unified prompt that performs well across all tasks.

• Training Strategy: Contrasts with sequential transfer, as tasks are learned jointly.
• Relation to Transferability: This method directly tests and optimizes for cross-task utility during training. A successfully multi-task-tuned prompt demonstrates high intrinsic transferability by design, as it must find a representation that generalizes across the training task distribution.