Self-Attention

Unlike recurrent neural networks (RNNs) that process tokens one by one, self-attention computes interactions between all positions in a sequence simultaneously. This enables massive parallelization during training on GPUs.

• Mechanism: Every token attends to every other token in a single matrix multiplication step
• Benefit: Training time scales with sequence length squared in memory but is constant in sequential operations
• Industrial relevance: Processes entire sensor time-series windows or maintenance log entries in one forward pass, enabling real-time anomaly detection on streaming factory data

Self-attention computes a unique attention score for every pair of tokens in a sequence, determining how much 'focus' each token should place on every other token when building its contextual representation.

• Query, Key, Value (QKV): Each input token is projected into three vectors. The Query asks 'what am I looking for?', the Key answers 'what do I represent?', and the Value holds the actual information to aggregate
• Scaled Dot-Product: Attention scores are computed as the dot product of Query and Key vectors, scaled down to prevent vanishing gradients
• Softmax normalization: Scores are converted to a probability distribution, creating a weighted sum of Value vectors
• Example: In the phrase 'The bearing on shaft 3 is overheating,' the word 'overheating' will assign a high attention weight to 'bearing' and 'shaft 3,' linking the symptom to the specific component

Self-attention creates direct pathways between any two positions in a sequence, regardless of their distance. A token at position 1 can influence the representation of a token at position 10,000 with the same number of computational steps as adjacent tokens.

• Constant path length: O(1) operations between any two positions, compared to O(n) for RNNs
• No vanishing gradient bottleneck: Gradients flow directly through the attention matrix during backpropagation, preserving signal across long sequences
• Manufacturing application: A foundation model analyzing a shift-long vibration log can directly correlate an early subtle frequency shift with a catastrophic failure event hours later, without losing the signal through recurrent state compression

Rather than computing a single attention function, the transformer runs multiple attention operations in parallel with different, learned linear projections of the Q, K, and V vectors. Each 'head' can specialize in a different type of relationship.

• Diverse subspace learning: One head might attend to syntactic structure, another to semantic similarity, another to positional proximity
• Concatenation and projection: The outputs of all heads are concatenated and projected back to the model dimension
• Industrial example: In a multimodal model analyzing a production line, one head might focus on temporal correlations in sensor data, another on spatial relationships in camera feeds, and a third on cross-modal alignment between a work order text and the visual inspection result

Because self-attention is inherently permutation-invariant—it has no built-in notion of sequence order—positional information must be explicitly injected. This is done by adding a unique positional encoding vector to each input embedding before the first attention layer.

• Sinusoidal encoding: The original transformer uses sine and cosine functions of different frequencies, allowing the model to extrapolate to sequence lengths unseen during training
• Learned positional embeddings: Many modern implementations use trainable position vectors, which can capture task-specific positional patterns
• Rotary Position Embedding (RoPE): Encodes relative position information directly into the attention computation, improving length generalization
• Manufacturing context: Positional encodings allow a model to distinguish 'pressure rose, then temperature spiked' from 'temperature spiked, then pressure rose'—two very different failure signatures in a time-series log

The standard self-attention mechanism has a quadratic memory and time complexity of O(n²) with respect to sequence length n, because an n×n attention matrix must be computed and stored.

• Memory bottleneck: A 4,096-token sequence requires a 4,096×4,096 attention matrix, consuming significant GPU memory
• Mitigation strategies: Sparse attention (Longformer), linearized attention (Performer), flash attention (hardware-aware exact attention), and sliding window attention reduce this to O(n) or O(n log n) in practice
• Edge deployment consideration: For factory-floor inference on long sensor sequences, optimized attention variants like FlashAttention-2 are critical to achieving sub-100ms latency on edge GPUs
• Trade-off: The quadratic complexity is the price paid for the mechanism's ability to model arbitrary pairwise interactions, making it ideal for tasks where global context matters

The dominant neural network design for foundation models that relies entirely on a self-attention mechanism to process sequential data in parallel. Unlike recurrent networks, transformers ingest entire sequences simultaneously, using positional encodings to preserve order. This parallelization enables training on massive industrial datasets and forms the backbone of modern language, vision, and time-series models used in manufacturing.

An extension of self-attention that runs the scaled dot-product attention mechanism multiple times in parallel with different learned linear projections. Each 'head' can specialize in different relational patterns:

• One head may attend to adjacent sensor readings for local trends
• Another may capture long-range dependencies across a production cycle
• A third may focus on causal relationships between equipment states The outputs are concatenated and projected, giving the model a richer representational capacity than a single attention function.

The mathematical core of self-attention, computed as: Attention(Q,K,V) = softmax(QK^T / √d_k)V. The scaling factor 1/√d_k prevents the dot products from growing too large in magnitude, which would push the softmax function into regions with extremely small gradients. This stabilization is critical for training deep transformers on high-dimensional industrial sensor embeddings without gradient vanishing.

Since self-attention is permutation-invariant—it has no inherent sense of sequence order—positional encodings inject information about token position. Original transformers use sinusoidal functions of varying frequencies. Modern variants often use learned positional embeddings or rotary position embeddings (RoPE). In manufacturing time-series, this is essential for distinguishing a temperature spike before a failure from one after normal operation.

A variant of the attention mechanism where queries come from one sequence while keys and values come from another. In encoder-decoder transformers, cross-attention allows the decoder to attend over the encoder's output. In manufacturing, this enables a model to align a work order's textual instructions (query) with relevant regions of a sensor time-series (keys/values) for contextual anomaly interpretation.

Techniques for inspecting the attention weight matrices to understand which input tokens the model considers relevant. Tools like BertViz or integrated gradient methods reveal attention patterns. In industrial applications, visualizing that a defect classification model attends to the exact pixel region of a scratch rather than background noise provides the explainability (XAI) required for quality assurance validation and operator trust.