RoPE Scaling

Linearly downscales all position indices to fit within the original pre-trained range. If a model was trained on 2048 tokens and needs to process 4096, PI divides every position index by 2. This effectively 'squeezes' the positions into the familiar range, preventing the model from encountering unseen rotation angles. Key trade-off: Requires fine-tuning on the extended length to recover performance, as the interpolation compresses the relative distance between nearby tokens, slightly degrading short-range attention resolution.

Applies a non-linear frequency scaling inspired by Neural Tangent Kernel theory. Instead of uniformly scaling all RoPE frequencies, NTK-aware methods scale the high-frequency components less and the low-frequency components more. Mechanism: High frequencies encode local, short-range positional information and are preserved for precision; low frequencies encode long-range positions and are stretched to cover the extended context. This approach often requires no fine-tuning and preserves perplexity better than linear interpolation.

Combines NTK-aware frequency scaling with a temperature adjustment on the attention softmax. YaRN observes that as sequences grow longer, the average attention entropy increases, causing the model to attend less sharply. Solution: Applies a learned or computed temperature factor that scales the attention logits, counteracting the entropy shift. This dual mechanism—frequency scaling plus attention tempering—achieves state-of-the-art long-context performance with minimal fine-tuning tokens.

Adjusts the NTK scaling factor on-the-fly based on the actual sequence length at inference time. A model loaded with dynamic scaling can process a 2k-token prompt followed by a 32k-token prompt in the same session without reloading weights. Implementation: The RoPE base frequency is recalculated for each forward pass as a function of the current sequence length, ensuring optimal frequency distribution regardless of input size. This is the default mode in many llama.cpp deployments.

Introduces two key innovations: progressive extension and a non-uniform frequency allocation search. LongRoPE first extends context to 256k tokens via NTK-aware scaling, then uses evolutionary search to identify optimal RoPE frequency allocations for different positional ranges. It identifies that certain frequency bands are critical for short-range performance while others dominate long-range extrapolation. The final method achieves effective context windows exceeding 2 million tokens while maintaining over 90% of the original short-context performance.

A training-free method that maps unseen large relative positions to known smaller ones during attention computation. For positions beyond the pre-trained window, Self-Extend applies a floor function to group distant positions into buckets that map to the original range. Advantage: Requires absolutely no fine-tuning and can be applied to any pre-trained RoPE model immediately. The grouping ratio is a hyperparameter that trades off between long-range precision and perplexity stability.

The foundational technique that linearly rescales position indices to fit within the pre-trained range. For a model trained on 2048 tokens extended to 4096, PI compresses positions by a factor of 2, turning position 4096 into 2048. While simple, this blurs the distinction between nearby positions, degrading short-range performance. PI was the catalyst that sparked the RoPE scaling research field.

A non-linear interpolation method inspired by Neural Tangent Kernel theory. Instead of uniformly rescaling all RoPE frequencies, NTK-aware scaling applies a higher interpolation rate to high frequencies (short distances) and a lower rate to low frequencies (long distances). This preserves local positional resolution while extending long-range capacity, often achieving better perplexity than linear PI without fine-tuning.

A state-of-the-art method combining NTK-aware scaling with a temperature-based attention entropy adjustment. YaRN addresses the attention distribution shift that occurs when sequences exceed training length. Key components:

• NTK-by-parts: Applies different scaling to different frequency bands
• Temperature tuning: A learnable scalar that sharpens or flattens the attention softmax
• Achieves strong performance at 2-4x context extension with minimal fine-tuning tokens

A deployment-time technique that adjusts the scaling factor on-the-fly based on the actual input sequence length. If a user sends a prompt of 10k tokens, the scaling factor adapts automatically without reloading the model. This enables a single model checkpoint to handle variable-length inputs efficiently, a critical feature for production inference servers like llama.cpp and vLLM.

The maximum token capacity a model can process in a single forward pass. RoPE scaling techniques aim to extend this window beyond the pre-trained limit. Key considerations:

• Training length: The original context size (e.g., 4096 for LLaMA 1)
• Extended length: The target after scaling (e.g., 32768)
• Effective length: The usable context before quality degrades, measured via Needle-in-a-Haystack benchmarks
• Scaling quality is highly sensitive to the fine-tuning data distribution used during the extension process

A Microsoft technique that extends context to over 2 million tokens by identifying and exploiting non-uniformities in RoPE's frequency utilization. LongRoPE introduces two innovations:

• Progressive extension: Gradually increases context length during fine-tuning
• Frequency search: Identifies critical frequency bands that require higher precision retention
• Achieves strong retrieval performance at 2048k tokens while maintaining original short-context quality