Prompt Initialization

The most straightforward strategy, where soft prompt embeddings are initialized with small random values, typically sampled from a Gaussian or uniform distribution. This treats the prompts as entirely new, learnable parameters with no prior semantic meaning.

• Pros: Simple to implement; makes no assumptions about the task.
• Cons: Can lead to slower convergence and higher variance in final performance; the model must discover useful prompt representations from scratch.
• Common Practice: Often used with a small standard deviation (e.g., 0.02) to keep initial embeddings near zero, stabilizing early training.

Initializes the continuous prompt embeddings with the pre-trained token embeddings of semantically relevant words from the model's vocabulary. This provides a meaningful starting point grounded in the model's existing knowledge.

• Method: Select a short sequence of task-descriptive words (e.g., for sentiment analysis: ['sentiment', 'positive', 'negative']). Use their corresponding token embeddings to initialize the soft prompt vectors.
• Pros: Faster convergence and often better final accuracy than random init; leverages the model's pre-existing linguistic understanding.
• Consideration: Performance can be sensitive to the choice of initial words. Automated selection via word embeddings (e.g., using GloVe or the model's own embedding space) can help.

A specialized form of vocabulary-based initialization that uses embeddings from tokens directly related to the task's label space or instruction. It directly encodes the task objective into the prompt's starting state.

• Example for Classification: For a topic classification task with labels 'sports', 'politics', 'technology', one might initialize the prompt with the embeddings of those label words.
• Example for Generation: For a summarization task, initializing with tokens like ['summarize', ':', 'article'] can be effective.
• Advantage: Can significantly reduce the number of training steps required, as the prompt begins close to a useful configuration.

Initializes soft prompts by sampling embeddings from the distribution of all token embeddings in the model's vocabulary. This provides a diverse, semantically plausible starting point without manually selecting specific words.

• Implementation: Randomly sample N tokens (where N is the prompt length) and use their pre-trained embeddings.
• Rationale: Avoids the bias of a single human-chosen phrase while still anchoring the prompt in the model's learned semantic space. It often performs more robustly than pure random initialization.
• Use Case: A good default when domain knowledge for selecting perfect initial words is lacking.

Initializes the soft prompts for a new task using prompts previously trained on a related source task. This is a form of transfer learning for prompts, based on the idea that useful prompting strategies may generalize across tasks.

• Process: 1. Train a soft prompt on a large, general task (e.g., natural language inference). 2. Use those learned embeddings as the starting point for training on the target task (e.g., sentiment analysis).
• Benefit: Can accelerate training on the target task and improve performance, especially when target data is limited. It leverages meta-knowledge about how to steer the model.
• Research Area: Explored in works on prompt tuning and cross-task generalization.

Specific to prefix tuning, this strategy initializes the trainable prefix matrices using the output of a small neural network (the prompt encoder) fed with a set of trainable embedding parameters. The embeddings are randomly initialized, but the encoder structure imposes a prior.

• Architecture: A multi-layer perceptron (MLP) takes a small matrix of trainable parameters as input and outputs the full prefix sequence for all transformer layers.
• Advantage: The encoder acts as a regularizer, improving generalization and stability compared to directly optimizing a large, unstructured prefix matrix. The trainable input to the encoder is the true parameter being optimized.
• Initialization: The weights of the prompt encoder itself are typically initialized using standard neural network methods (e.g., Xavier initialization).

A soft prompt is a set of continuous, high-dimensional vector embeddings that are learned during training and prepended to the input of a frozen language model to condition its output for a specific task. Unlike discrete text, these vectors exist in the model's embedding space.

• They are the primary tunable parameters in prompt tuning.
• Initialization strategies (e.g., from task-relevant word embeddings) are critical for their performance.

A virtual token is a learned embedding in prompt or prefix tuning that does not correspond to any discrete token in the model's vocabulary. It acts as a tunable parameter to guide model behavior.

• These are the constituent elements of a soft prompt or trainable prefix.
• Their initialization values are what define the starting point for prompt optimization.

The prompt gradient is the derivative of the loss function with respect to the parameters of a soft prompt or prefix. This gradient is computed during backpropagation and is used to update only the prompt parameters while the base model's weights remain frozen.

• It is the mechanism that enables gradient-based prompt initialization refinement.
• Efficient calculation of this gradient is key to the parameter efficiency of these methods.

Prompt tuning is the specific parameter-efficient fine-tuning (PEFT) method where prompt initialization occurs. It optimizes a small set of continuous, trainable embeddings (the soft prompt) prepended to the model input while keeping the underlying pre-trained model frozen.

• The performance is highly sensitive to the initialization strategy of the prompt embeddings.
• It is distinct from prefix tuning, which adds tunable vectors to the model's hidden states at every layer.

Prompt overfitting is a failure mode in prompt tuning where the learned soft prompt becomes too specialized to the training data, capturing noise and reducing performance on validation or test sets. This can be influenced by poor initialization.

• Mitigation strategies include using better initialization (e.g., from class labels) and regularization.
• It contrasts with the desired outcome of prompt generalization.

A prompt pool is a set of multiple soft prompts that can be dynamically selected or combined during inference. This architecture often requires initializing multiple prompts, which introduces complexity.

• Used in continual learning to manage catastrophic forgetting by associating different prompts with different tasks.
• The initialization of the entire pool is a key design consideration for stable multi-task performance.