Autoregressive Generation

The fundamental mechanism where a model factorizes the joint probability of a sequence into a product of conditional probabilities. At each time step t, the model computes P(x_t | x_1, ..., x_{t-1}) using the hidden state from the previous step. This left-to-right causality ensures that the generation of the current token cannot peek at future tokens, a constraint enforced by the causal attention mask in Transformer architectures. The output at step t is fed back as input at step t+1, creating a closed-loop generation process.

During training, Teacher Forcing feeds the ground-truth previous token to the model regardless of its own prediction, maximizing parallelization but creating an exposure bias between training and inference. At inference, the model must rely on its own potentially erroneous predictions. Scheduled Sampling mitigates this by randomly replacing ground-truth tokens with model-generated tokens during training, forcing the model to learn to recover from its own mistakes and stabilizing autoregressive rollouts.

A critical inference optimization that stores the Key and Value matrices from previous attention computations. Without a KV-cache, the model would recompute attention for the entire sequence at each new token, leading to quadratic scaling. The cache enables linear-time incremental decoding by only computing attention for the newest token. Memory pressure from the cache is a primary bottleneck for long-form generation, addressed by techniques like PagedAttention and Multi-Query Attention (MQA).

Pure greedy decoding (selecting the highest probability token) often leads to degenerate, repetitive text. Autoregressive models rely on stochastic sampling methods to introduce controlled randomness:

• Temperature: Scales logits to sharpen (T<1) or flatten (T>1) the probability distribution.
• Top-k Sampling: Restricts the sampling pool to the k most likely tokens, truncating the long tail of improbable vocabulary.
• Top-p (Nucleus) Sampling: Dynamically selects the smallest set of tokens whose cumulative probability exceeds p, adapting the candidate pool size to the confidence of the model at each step.

Unlike parallelizable encoding, autoregressive decoding is fundamentally memory-bandwidth bound. Each forward pass processes a single token but requires loading the entire model weight matrix from memory. The latency per token is determined by the time-to-first-token (TTFT) and the subsequent inter-token latency. Techniques like speculative decoding use a smaller draft model to propose multiple tokens autoregressively, which are then verified in parallel by the large target model, reducing wall-clock latency.

An autoregressive loop must have a defined termination condition to prevent infinite generation. The primary mechanism is the End-of-Sequence (EOS) token, a special vocabulary entry the model is trained to emit upon completion. Production systems implement secondary guardrails:

• Max Token Limit: A hard cutoff on sequence length.
• Repetition Penalty: A hyperparameter that penalizes tokens already present in the context to break degenerate loops.
• Structural Stop Strings: Regex-based detection of complete syntactic units (e.g., a closing JSON bracket).