Temporal Convolution

A temporal convolution operates with a fixed-size kernel that slides across the input sequence. This creates a local receptive field, meaning the output at any timestep is computed from a small, contiguous window of previous inputs. This is fundamental for capturing short-term dependencies and local motifs, such as a specific sound in an audio clip or a short phrase in text, without the long-range modeling complexity of architectures like transformers.

The same filter weights are applied at every position in the sequence, a principle known as parameter sharing. This makes the operation translation equivariant with respect to time: if a pattern shifts in the input, the corresponding feature in the output shifts by the same amount. This efficiency and inductive bias are ideal for tasks where local patterns (e.g., phonemes, sensor spikes) are informative regardless of their absolute position in time.

By stacking multiple temporal convolutional layers, networks can build hierarchical representations. Lower layers detect simple, short-term features (e.g., edges in a signal). Subsequent layers, with their effectively enlarged receptive fields due to stacking, combine these into more complex, longer-term temporal structures. This multi-scale analysis is crucial for understanding sequences at different levels of abstraction.

In strict sequence prediction tasks (e.g., real-time audio synthesis, next-word prediction), causal (or left) padding is used. This ensures the convolution for timestep t uses only inputs from t and earlier, preventing information "leakage" from the future. This enforces the autoregressive property, making the model suitable for online, step-by-step generation and forecasting.

To capture longer-range dependencies without a proportional increase in parameters or layers, dilated convolutions are used. The kernel is applied over an input window with gaps, defined by a dilation rate. For example, a kernel of size 3 with a dilation rate of 2 skips one input between each tapped element. This allows the network to efficiently integrate information from a much wider temporal context.

• vs. RNNs/LSTMs: Temporal CNNs process sequences in parallel over the time dimension, leading to faster training. They lack an explicit hidden state that carries information indefinitely, instead relying on the depth of the network and kernel size for context.
• vs. Transformers: Transformers use self-attention for global, dynamic context weighting. Temporal convolution offers a fixed, local context window, which is often more computationally efficient and sufficient for tasks dominated by local patterns, though hybrid models (e.g., Conformers) combine both.

A vector representation that encodes an item's position or characteristics within a time series. Unlike static embeddings, temporal embeddings change to reflect an element's place in a sequence, enabling similarity search and reasoning over time-aware information.

• Key Use: Enables models to understand "when" something happened, not just "what" happened.
• Example: In a user behavior model, the embedding for "login" would differ if it's the first action of a session versus the last.

A mechanism within neural networks, such as transformers, that dynamically weights the importance of past events based on their relevance to the current context, rather than just their temporal proximity.

• Mechanism: Computes attention scores between the current query and all past keys in a sequence.
• Contrast with Convolution: While temporal convolution applies fixed, local filters, temporal attention learns global, content-dependent relationships across the entire sequence history.

A fixed-size, in-memory data structure that stores the most recent events or states in chronological order. It acts as a short-term, rolling window of agent experience, often implemented as a First-In-First-Out (FIFO) queue.

• Primary Function: Provides immediate context for real-time decision-making.
• Engineering Role: Serves as the direct input stream for temporal convolution operations, feeding the model the raw sequence of recent observations.

A knowledge graph structure where nodes represent events and directed edges represent inferred causal or temporal relationships (e.g., 'causes', 'precedes'). This enables complex reasoning about chains of influence over time.

• Abstraction Level: Operates at a higher, symbolic level compared to the low-level pattern detection of temporal convolution.
• Integration: Raw event sequences, processed by convolutional layers, can be abstracted into causal graphs for explainable reasoning.

A dimensionality reduction operation that aggregates features across the time dimension. Common methods include max pooling, average pooling, or attention-weighted pooling over a defined temporal window.

• Downstream Step: Often applied after temporal convolution layers to reduce the sequence length and extract dominant features.
• Example: After convolutions detect local patterns in sensor data, max pooling might extract the most salient spike from each time segment.