Sequence Encoding

A deterministic function that injects information about the absolute or relative position of tokens in a sequence into their vector representations. This is critical for transformer architectures, which otherwise lack an inherent sense of order.

• Absolute Positional Encoding: Adds a fixed sinusoidal or learned vector to each token's embedding based on its index (e.g., position 1, 2, 3).
• Relative Positional Encoding: Models the distance between tokens (e.g., token A is 5 positions before token B), which can generalize better to longer sequences.
• Rotary Position Embedding (RoPE): A prevalent method that encodes relative position by rotating the token's embedding matrix, improving extrapolation to longer contexts.

A class of neural networks with internal loops, designed to process sequences by maintaining a hidden state that serves as a compressed memory of all previous inputs.

• Core Mechanism: At each timestep t, the network takes the current input x_t and the previous hidden state h_{t-1} to produce a new hidden state h_t and an output y_t.
• Limitations: Suffers from the vanishing/exploding gradient problem, making it difficult to learn long-range dependencies.
• Variants: Long Short-Term Memory (LSTM) and Gated Recurrent Units (GRU) were developed with specialized gating mechanisms to selectively remember or forget information, mitigating the long-term dependency issue.

Use one-dimensional convolutional layers applied across the time dimension to capture sequential patterns. They offer advantages in parallelization and stable gradients over long sequences.

• Causal Convolutions: Ensure the output at time t is only convolved with inputs from t and earlier, preventing information leakage from the future—essential for autoregressive tasks like forecasting.
• Dilated Convolutions: Introduce gaps (dilation) between kernel elements, exponentially increasing the receptive field without a proportional increase in parameters or depth. This allows the network to capture very long-range dependencies efficiently.
• Use Case: Often used in time-series forecasting and audio synthesis where long context and parallel training are beneficial.

The mechanism at the heart of modern sequence models, which computes a weighted sum of values from all positions in the sequence, with weights determined by the compatibility between queries and keys.

• Permutation-Invariant by Default: Without positional encoding, self-attention treats a sequence as an unordered set. The attention weights themselves create a dynamic, data-dependent graph of relationships.
• Scaled Dot-Product Attention: The standard formulation: Attention(Q, K, V) = softmax(QK^T / sqrt(d_k)) V. The scaling factor stabilizes gradients.
• Multi-Head Attention: Runs multiple attention operations in parallel, allowing the model to jointly attend to information from different representation subspaces at different positions.

A class of sequence models inspired by continuous-time systems that map a 1D input sequence to an output via a latent state vector. Modern structured state space sequence models (S4, Mamba) have shown remarkable efficiency on long sequences.

• Core Idea: Model the sequence as the evolution of a hidden state h(t) governed by a linear differential equation: h'(t) = A h(t) + B x(t), with output y(t) = C h(t).
• Discretization: For digital data, the continuous parameters (A, B) are discretized using a step size Δ, turning the system into a linear recurrence.
• Computational Efficiency: The linear recurrence form allows for fast training (parallel scan) and inference (constant-time step). Models like Mamba introduce input-dependent parameters (A, B, C, Δ), making them selective and context-aware.

Techniques to create a single, fixed-size vector representation for an entire variable-length sequence, often for classification or retrieval tasks.

• Temporal Embedding: Learns a dense vector that represents the sequence's temporal characteristics, often used as an input feature for downstream models.
• Pooling Operations:
  • Mean/Max Pooling: Aggregates across the time dimension by taking the average or maximum of all token embeddings. Simple but loses fine-grained order.
  • Attention Pooling: Uses a learned attention mechanism to compute a weighted sum of token embeddings, allowing the model to focus on the most salient parts of the sequence.
  • Last Hidden State: In RNNs, the final hidden state is often used as the sequence representation, encapsulating the history.
• Use Case: Creating a "memory summary" of an event stream for storage in a vector database or for quick similarity comparison.

A vector representation that encodes an item's position or characteristics within a time series. Unlike a standard embedding, it incorporates temporal context, enabling similarity search and reasoning over time-aware information.

• Key Mechanism: Often generated by models that process sequential data, such as Recurrent Neural Networks (RNNs) or transformers with positional encoding.
• Use Case: Finding similar patterns in sensor data that occurred at analogous phases in a process, regardless of absolute timestamp.

A mechanism within neural networks that dynamically weights the importance of past events or states based on their relevance to the current context, not just their temporal proximity.

• Core Function: Allows a model to focus on critical past moments, such as a decision point, while ignoring irrelevant intervening steps.
• Architecture: Fundamental to transformer models, where the self-attention mechanism computes relationships across all positions in a sequence.

A fixed-size, in-memory data structure that stores the most recent events or states in chronological order, acting as a short-term, rolling window of agent experience.

• Primary Role: Serves as working memory or short-term context for an autonomous agent.
• Eviction Policy: Typically uses a First-In-First-Out (FIFO) strategy; when full, the oldest event is discarded to make room for the newest.
• Example: A chatbot's immediate conversation history for maintaining dialogue coherence.

A continuous, time-ordered sequence of discrete events or state changes that serves as the foundational data source for temporal memory in autonomous agents.

• Characteristics: Immutable and append-only. Each event is a record with a payload and a timestamp.
• Infrastructure: Often processed using stream processing frameworks like Apache Kafka or Apache Flink.
• Applications: User interaction logs, sensor telemetry from IoT devices, financial transaction feeds.

An operation in Convolutional Neural Networks (CNNs) where filters are applied across the time dimension to extract local temporal patterns and features from sequential data.

• Mechanism: A sliding window performs element-wise multiplication and summation across adjacent time steps.
• Advantage: Highly efficient at capturing local dependencies and multi-scale patterns within a sequence.
• Use Case: Feature extraction from raw audio waveforms or time-series sensor data for anomaly detection.

The computational process of mapping and comparing two or more temporal sequences to identify correspondences, similarities, or differences in their event order.

• Key Algorithm: Dynamic Time Warping (DTW), which finds an optimal alignment between sequences that may vary in speed.
• Applications: Bioinformatics for aligning DNA/protein sequences, speech recognition to match audio samples, and human activity recognition from sensor data.