Memory-Augmented Networks

The foundational architecture that introduced the concept of a differentiable external memory. An NTM consists of a controller network (typically an LSTM or feedforward network) and a memory matrix. The controller uses content-based and location-based addressing heads to perform soft read and write operations, allowing it to learn algorithms for storing and retrieving information. This demonstrated that neural networks could learn to use memory in a programmable way.

An advanced successor to the NTM that improved memory access and allocation. Key innovations include:

• Temporal Link Matrix: Tracks the order in which memory locations were written, enabling the traversal of sequences.
• Dynamic Memory Allocation: A free-list mechanism that allocates and deallocates memory dynamically, preventing interference between unrelated data.
• Sharpened Attention: More precise read and write operations. The DNC could solve complex graph traversal and reasoning tasks that required maintaining state over long sequences.

An architecture designed explicitly for question-answering over large knowledge bases. It separates memory from reasoning:

• Memory: A set of continuous vector representations (embeddings) of stored facts or sentences.
• Inference Components: A network that computes the relevance of each memory to a query (attention) and then reasons over the retrieved memories in multiple hops. This multi-hop process allows the model to chain facts together to arrive at an answer, mimicking iterative reasoning.

A structured variant of Memory Networks where each memory slot is a (key, value) pair. The key is used for retrieval via similarity search against a query embedding. The corresponding value is returned and used for generating the output. This design is highly influential, forming the conceptual basis for modern retrieval-augmented generation (RAG), where a vector database acts as the key-value memory and the LLM is the reasoning controller.

Memory-augmented networks address the core limitation of a transformer's fixed context window. While a transformer's context is a temporary, contiguous working memory, a MANN's external store provides:

• Long-term Persistence: Information can be stored indefinitely and accessed later, beyond a single session.
• Scalable Capacity: Memory size can be scaled independently of model parameters.
• Dynamic Updates: Memory can be edited or appended to without retraining the core model. This architectural pattern is now realized in production systems through vector databases and knowledge graphs integrated with LLMs.

A fully differentiable implementation of multi-hop Memory Networks where all components—the input and output feature mappings, the memory representations, and the attention mechanisms—are trained jointly via backpropagation. This allows the model to learn optimal memory representations for a task directly from data. The end-to-end approach simplified training and demonstrated strong performance on language modeling and QA benchmarks requiring long-range dependency resolution.

The foundational Memory-Augmented Neural Network that introduced a controller network (often an LSTM) coupled with an external, addressable memory matrix. It uses differentiable, content-based and location-based attention mechanisms to read from and write to memory, allowing it to learn simple algorithms like copying and sorting. The NTM demonstrated that neural networks could learn to use memory as a programmable resource.

An advanced successor to the NTM that solved critical memory management issues. Its key innovations include:

• Dynamic Memory Allocation: A free-list system to track and allocate unused memory locations.
• Temporal Linkage: A mechanism to record the order in which memory locations were written, enabling the traversal of sequences in time.
• Content-Based Lookup with Location-Based Iteration: This allowed the DNC to perform more complex reasoning tasks, such as finding the shortest path on a graph or solving logic puzzles from textual descriptions.

A model architecture explicitly designed for Question Answering over large knowledge bases or story contexts. It separates memory (a set of stored sentences or facts) from inference components. The model operates in multiple hops:

1. Input: Embeds the question and each memory item.
2. Attention: Computes a relevance score between the question and each memory.
3. Reasoning: Uses the retrieved memory to update the question representation.
4. Output: Repeats the attention and reasoning steps over multiple hops before generating a final answer. This multi-hop process mimics iterative reasoning.

A refined version of Memory Networks where the external memory is structured as a set of key-value pairs. The key is used for addressing (retrieval via similarity to the query), and the value contains the information to be returned, which can be different from the key. This separation is powerful for tasks like:

• Factual QA: The key might be a subject-relation (e.g., "Paris-capital_of") and the value is the object ("France").
• Recommendation Systems: The key is a user/item ID embedding, and the value is a rating or feature vector.

A specialized architecture that uses attention over its input sequence as a form of content-addressable memory. Instead of generating output tokens from a fixed vocabulary, it produces a probability distribution over positions in the input sequence, effectively "pointing" to an input element to copy as the output. This makes it ideal for tasks where the output is a rearrangement or selection of the input, such as:

• Sorting variable-length sequences
• Solving combinatorial optimization problems like the Traveling Salesman Problem
• Text summarization via extractive sentence selection.

While not pure Memory-Augmented Networks, contemporary large language models and Retrieval-Augmented Generation (RAG) systems embody the core principles. The transformer's self-attention acts as a form of internal, working memory over the context window. RAG explicitly adds an external, non-differentiable memory (a vector database) that is queried via retrieval and injected into the context. This hybrid approach provides the scalable, long-term memory that differentiable architectures like the NTM aimed for, but with more practical engineering for massive scale.

A foundational memory-augmented neural network architecture that introduced a differentiable, external memory matrix accessible via content-based and location-based addressing. It consists of a controller network (typically an LSTM or feedforward network) that learns to read from and write to the memory bank using soft attention mechanisms, enabling algorithmic tasks like copying and sorting. The NTM demonstrated that neural networks could learn to use an external memory for complex, multi-step reasoning.

An advanced successor to the Neural Turing Machine that added mechanisms to prevent memory interference and improve long-term retention. Key innovations include:

• Temporal Link Matrix: Tracks the order in which memory locations were written, enabling sequential recall.
• Usage Vector: Manages memory allocation with dynamic free gates and allocation weighting.
• Content-Based Lookup with Temporal Links: Allows retrieval based on both content similarity and temporal sequence. The DNC can solve complex graph traversal and puzzle tasks requiring dynamic memory management.

A simpler architecture designed for question-answering over large knowledge bases. It structures memory as a set of fact embeddings (e.g., sentences) stored in slots. Inference involves:

1. Input Representation: Embedding the query.
2. Memory Addressing (Hop): Computing relevance scores between the query and each memory slot.
3. Memory Reading: Retrieving the most relevant memory vectors.
4. Output Generation: Combining retrieved memories to produce a final answer. Models like MemN2N (End-to-End Memory Network) made this process fully differentiable.

A variant of Memory Networks where each memory slot is a key-value pair. The key is used for addressing (similarity search against the query), and the value contains the information to be returned. This separation is powerful for tasks like knowledge base QA, where the key might be a subject-relation embedding and the value is the target object. It provides more flexible and structured retrieval compared to simple slot-based memories.

An architecture tailored for language understanding tasks like sentiment analysis and QA. It features an episodic memory module that performs multiple computational passes ("hops") over input facts. In each hop, an attention mechanism focuses on different parts of the input relative to the question, iteratively refining an internal memory state. This allows the model to perform multi-step inference and reason over narratives.

The broader paradigm of equipping neural models with access to non-parametric, external data stores. This includes:

• Retrieval-Augmented Generation (RAG): A dominant modern approach where a retriever fetches documents from a corpus (e.g., a vector database) to condition a generator's output.
• Sparse vs. Dense Retrieval: Trade-offs between keyword-based (BM25) and embedding-based (DPR) memory lookup. Memory-augmented networks are a specialized instance of this paradigm, focusing on a small, differentiable, and dynamically writable memory, whereas RAG typically uses large, static, read-only corpora.