Sliding Window Attention

Each token attends only to W tokens on either side, where W is the window radius. This creates a banded attention matrix where the receptive field is strictly local. For a token at position i, the attention span is [i-W, i+W], making the computational cost O(n × W) instead of O(n²). This local bias aligns with the linguistic intuition that nearby words are often most relevant for disambiguation.

Although each layer has a fixed local window, stacking L layers expands the effective receptive field to L × W tokens. Information propagates through the sequence layer by layer, analogous to the increasing receptive field in Convolutional Neural Networks. A token in layer k can indirectly attend to tokens that were in the window of its neighbors in layer k-1, building global context hierarchically without ever computing a full N×N attention matrix.

Standard self-attention requires storing the full N × N attention score matrix, consuming O(n²) memory. Sliding Window Attention stores only the N × (2W+1) non-zero entries of the banded matrix. For a sequence of 100K tokens with a window of 512, this reduces memory from ~40 GB to ~200 MB for the attention scores alone, making long-document processing feasible on a single GPU.

Efficient implementations like FlashAttention and block-sparse kernels are critical for realizing the theoretical speedups. Naive sparse masking still materializes the full attention matrix in memory. Optimized kernels compute only the non-zero blocks, using tiling strategies that keep data in GPU SRAM. Frameworks like Longformer and Mistral use custom CUDA kernels to achieve near-linear wall-clock time for windowed attention patterns.

Pure sliding window attention can be augmented with dilated windows that skip tokens at regular intervals, exponentially increasing the receptive field without additional compute. Some architectures also designate a small set of global tokens (e.g., CLS tokens or task-specific tokens) that attend to the entire sequence, providing a pathway for global information to bypass the local bottleneck while keeping overall complexity linear.

The primary limitation is that direct long-range interactions are sacrificed. Two tokens separated by more than W positions cannot directly attend to each other in a single layer. This can degrade performance on tasks requiring distant token-to-token alignment, such as coreference resolution across long passages. The hierarchical propagation through layers mitigates but does not fully eliminate this constraint, making window size a critical hyperparameter.

The broader class of techniques that Sliding Window Attention belongs to. Sparse attention reduces the O(n²) complexity of standard self-attention by computing only a subset of the full attention score matrix. Instead of every token attending to every other token, connections are limited using predefined patterns.

• Local patterns: Tokens attend only to neighbors within a fixed window
• Dilated patterns: Gaps are introduced to expand the receptive field
• Global tokens: Special tokens attend to the entire sequence
• Random patterns: A subset of connections are sampled stochastically

This enables processing sequences of 100k+ tokens without running out of GPU memory.

The fundamental limitation that Sliding Window Attention addresses. In standard self-attention, an N × N attention matrix must be computed and stored, where N is the sequence length. This means:

• Memory scales quadratically: A 32k token sequence requires ~4GB just for the attention matrix at fp16
• Time scales quadratically: Doubling sequence length quadruples computation
• Practical ceiling: Full attention becomes infeasible beyond ~8k tokens on most hardware

Sparse patterns like sliding windows break this bottleneck by enforcing a fixed computational budget per token, regardless of total sequence length.

An extension of the basic sliding window pattern that introduces gaps between attended tokens, analogous to dilated convolutions in CNNs. This exponentially expands the model's receptive field without increasing the number of attention computations.

• Dilation rate d: Attend to every d-th token within the window
• A window of size w with dilation d covers a span of w × d tokens
• Multiple heads can use different dilation rates for multi-scale processing
• Often combined with standard windows in alternating layers

This technique allows a model to capture both fine-grained local context and broader structural patterns simultaneously.