Transformer Architecture

The core computational innovation that replaces recurrence. Self-attention computes a weighted representation of an entire input sequence for every position simultaneously. Each token generates a Query, Key, and Value vector. The attention score between token i and token j is the dot product of the Query of i and the Key of j, scaled and normalized via softmax.

• Parallelism: Unlike RNNs, all tokens are processed concurrently, drastically reducing training time.
• Long-Range Dependencies: A token can directly attend to any other token, regardless of distance in the sequence. This is critical for understanding context in long maintenance logs or sensor time-series.
• Scaled Dot-Product: The formula is Attention(Q, K, V) = softmax(QK^T / √d_k)V, where d_k is the dimension of the Key vector, preventing gradients from becoming too small.

Instead of performing a single attention function, the model projects the Queries, Keys, and Values h times with different, learned linear projections. This allows the model to jointly attend to information from different representation subspaces at different positions.

• Diverse Representations: One head might focus on syntactic structure, another on semantic meaning, and another on positional relationships.
• Parallel Execution: All heads compute attention in parallel, with their outputs concatenated and projected back to the model's dimension.
• Industrial Relevance: In a multimodal manufacturing model, separate heads can simultaneously attend to a visual defect region and the corresponding textual description in a work order.

Since the Transformer contains no recurrence or convolution, it has no inherent sense of token order. Positional information must be injected explicitly. The original architecture uses sinusoidal functions of different frequencies added to the input embeddings.

• Sinusoidal Encoding: PE(pos, 2i) = sin(pos / 10000^(2i/d_model)) and PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model)). This allows the model to extrapolate to sequence lengths unseen during training.
• Learned Positional Embeddings: Modern variants like GPT often use learned, absolute position embeddings.
• Rotary Position Embedding (RoPE): A widely adopted method that encodes position by rotating the Query and Key vectors, providing better relative position awareness, crucial for time-series sensor data.

Each sub-layer (attention and feed-forward) in the encoder and decoder is wrapped with a residual connection, followed by layer normalization. The output is LayerNorm(x + Sublayer(x)).

• Residual Connections: Allow gradients to flow directly through the network during backpropagation, mitigating the vanishing gradient problem and enabling the training of very deep models.
• Layer Normalization: Stabilizes the learning process by normalizing the inputs across the features for each training example, reducing training time.
• Pre-LN vs. Post-LN: Modern architectures often apply layer normalization before the sub-layer (Pre-LN), which provides more stable training dynamics than the original Post-LN design.

Each attention sub-layer is followed by a fully connected feed-forward network, applied identically and independently to each position. It consists of two linear transformations with a non-linear activation function in between: FFN(x) = W_2 * ReLU(W_1 * x + b_1) + b_2.

• Position-Wise: The same weights are applied to every token, but each token is processed independently, allowing the model to learn position-specific transformations.
• Dimensionality Expansion: The inner layer typically expands the dimension by a factor of 4 (e.g., from 512 to 2048), providing high capacity for non-linear feature transformation.
• Knowledge Storage: Empirical research suggests the FFN layers act as key-value memories, storing a significant portion of the model's factual and conceptual knowledge.

The original Transformer is a sequence-to-sequence model with an encoder and a decoder. The encoder processes the entire input sequence to generate a rich, contextualized representation. The decoder generates the output sequence auto-regressively, attending to both its own previous outputs (masked self-attention) and the encoder's output (cross-attention).

• Encoder: Composed of N identical layers, each with multi-head self-attention and an FFN. Used in models like BERT for understanding tasks.
• Decoder: Similar layers, but with an additional cross-attention sub-layer that queries the encoder's output. Used in GPT-style models for generation.
• Industrial Application: An encoder can process a full sensor log, while a decoder generates a natural language summary or a structured anomaly report.

The fundamental operation that allows a transformer to weigh the importance of every token in a sequence relative to every other token. Unlike recurrent networks that process data sequentially, self-attention computes pairwise relationships in parallel, capturing long-range dependencies in sensor time-series or maintenance logs. The mechanism projects input into Query (Q), Key (K), and Value (V) matrices, computing attention scores via scaled dot-product: Attention(Q,K,V) = softmax(QK^T/√d_k)V. Multi-head attention runs multiple attention operations in parallel, allowing the model to attend to information from different representation subspaces simultaneously.

Since self-attention is permutation-invariant and has no inherent sense of sequence order, positional encodings inject information about token position into the input embeddings. The original transformer uses sinusoidal positional encodings—fixed sine and cosine functions of different frequencies—added directly to input embeddings. Modern variants often use learned positional embeddings or rotary position embeddings (RoPE) that encode relative position directly into the attention computation. This is critical for industrial time-series data where temporal ordering carries essential meaning.

A model compression technique that reduces the numerical precision of a transformer's weights and activations—typically from 32-bit floating point (FP32) to 8-bit integers (INT8) or even 4-bit (INT4). Post-training quantization (PTQ) applies this after training without retraining, while quantization-aware training (QAT) simulates quantization during training for higher accuracy. This enables large industrial foundation models to run on resource-constrained factory-floor hardware with minimal accuracy degradation.

A parameter-efficient fine-tuning method that freezes the original transformer weights and injects trainable low-rank decomposition matrices into the attention layers. Instead of updating the full weight matrix W, LoRA learns a low-rank update ΔW = BA, where B and A are small matrices. This reduces trainable parameters by over 10,000x for large models while preserving performance. Critical for adapting a single foundation model to multiple manufacturing tasks without storing full model copies.

An exact-attention algorithm that dramatically reduces the memory footprint of self-attention by tiling the computation to minimize reads and writes between GPU high-bandwidth memory (HBM) and on-chip SRAM. Standard attention materializes the full N×N attention matrix, causing O(N²) memory usage. FlashAttention computes attention in blocks, recomputing softmax statistics on-the-fly rather than storing intermediate matrices. This enables transformers to process longer sequences—essential for high-frequency industrial sensor data.

During autoregressive text generation, transformers recompute Key and Value projections for all previous tokens at each step. KV-caching stores these computed projections in memory to avoid redundant computation. However, the cache grows linearly with sequence length, becoming a memory bottleneck. Techniques like multi-query attention (MQA) and grouped-query attention (GQA) share Key/Value heads across multiple Query heads, dramatically reducing cache size while maintaining model quality.