Memory-Native Sequence Intelligence: Test-Time Learning, Fast Weights, and Recurrent State

Full report

On this page

The foundational architecture of sequence intelligence has historically been constrained by the dichotomy between compressed recurrent states and full-context attention mechanisms1. Traditional Recurrent Neural Networks (RNNs) and their modern state-space descendants continuously compress historical context into a fixed-size vector. While this achieves linear computational complexity, it inherently suffers from rapid information decay and catastrophic forgetting over long sequences3. Conversely, self-attention architectures maintain a lossless historical ledger by calculating the direct dependencies of all tokens across a sequence window1. This mechanism acts as an optimal short-term memory but incurs a quadratic computational and spatial cost ([Figure omitted from source export]), rendering standard attention architectures fundamentally incompatible with the million-step sequence horizons demanded by genomics, continuous agentic reasoning, and unbounded contextual analysis1. The emergence of memory-native sequence intelligence represents a paradigm shift away from the premise of "frozen inference." In traditional paradigms, neural networks act as static functions during inference, ceasing to learn once deployed. However, a convergence of research into Test-Time Training (TTT) layers, fast weight programmers, and neural long-term memory frameworks has demonstrated that the hidden state of a sequence model should not be a passive algebraic accumulator. Instead, the recurrent state operates optimally when conceptualized as an active machine learning model itself, continuously updating its internal parameters during the forward pass2. Under this framework, inference is redefined as an act of continuous, localized optimization, bridging the gap between meta-learning and real-time associative recall4.

Linear Attention and the Re-emergence of Fast Weights

To trace the evolution of active memory architectures, it is necessary to examine the trajectory of linear attention. The foundational premise of linear attention is the decomposition of the standard softmax attention matrix into a kernelized formulation, which allows associative computations to be expressed as a running sum10. This recursive form effectively treats the sequence layer as a "fast weight programmer" that dynamically binds query, key, and value vectors over time into a fixed-size memory matrix10. However, pure linear attention suffers from uniform associative interference. As new tokens are ingested into the sequence, their representations uniformly overwrite previous associations without regard for the structural geometry of the knowledge space. This limitation was systematically addressed by the reintroduction of the Delta Rule—originally formalized in 1960 by Widrow and Hoff—into sequence modeling via architectures like DeltaNet13.

Test-Time Regression and the Delta Rule

DeltaNet reconceptualizes the recurrent state as a dynamic weight matrix [Figure omitted from source export] that explicitly attempts to minimize a least-squares retrieval error at every sequential time step15. The model calculates the difference (the "delta") between the target value [Figure omitted from source export] and the memory's current prediction for the key [Figure omitted from source export], updating the state purely based on this residual error14. The standard DeltaNet recursive update is formulated as: [Figure omitted from source export] where [Figure omitted from source export] represents a learning rate or gain parameter15. This equation reveals that the delta rule is effectively a step of stochastic gradient descent (SGD) executed on a linear regression objective natively during the forward pass15. By isolating the error term [Figure omitted from source export], the network surgically overwrites specific associative bindings rather than uniformly decaying the entire memory matrix16.

Preconditioning and the DGPS Taxonomy

While the standard delta rule provides targeted memory updates, it constitutes a first-order optimization that ignores the underlying curvature of the data space15. In deeply correlated sequences, first-order approximations lead to suboptimal memory topologies. Advanced implementations introduce second-order "curvature-aware" information through exact preconditioning, effectively multiplying the update by the inverse key Gram matrix15. Because computing an exact inverse Gram matrix sequentially is computationally prohibitive, diagonal approximations are utilized to maintain hardware efficiency required for large language models15. This dynamic gives rise to the DGPS architectural taxonomy for classifying linear recurrences, which maps every linear state-space model against four variables15.

Gated Recurrences and Bilinear Interactions

The parallel evolution of State-Space Models (SSMs), particularly Mamba and Mamba2, introduced data-dependent scalar gating, allowing networks to adaptively filter out irrelevant tokens before they contaminate the memory state17. However, Mamba2 relies on a uniform decay mechanism; it applies the same decay coefficient to all dimensions of the hidden state, which proves highly inefficient for precise memory clearance or targeted forgetting20. The structural synthesis of Mamba2's scalar gating and DeltaNet's precision error-correction birthed the Gated DeltaNet architecture, recently adopted as the linear-attention layer for models such as Qwen3-Next18. Gated DeltaNet represents a class of Bilinear RNNs where the hidden state interacts directly with the input key vectors via multiplicative dynamics17.

The Surgical Eraser and the Power Washer

Gated DeltaNet enforces a "gated delta rule," which unifies broad state decay with precise associative erasure. The central recurrence is defined as: [Figure omitted from source export] where [Figure omitted from source export] serves as the global forget gate (often conceptualized as a "power washer"), while the Householder transformation [Figure omitted from source export] acts as a "surgical eraser"16. Assuming [Figure omitted from source export] is normalized, this transformation systematically projects the existing memory orthogonal to the current key. This orthogonal projection effectively deletes any prior association with [Figure omitted from source export] before the new value is written, guaranteeing that contradictory facts do not superimpose over one another16. The implications of this orthogonal projection are profound for language modeling and million-step sequence evaluation. In Synthetic Needle-in-a-Haystack (S-NIAH) benchmarks, models are tested on their ability to retrieve specific tokens buried within massive distractor contexts. Pure Mamba2 struggles with complex noise filtration due to uniform decay, while standard DeltaNet excels in long-term retention but fails to clear obsolete memory context when confronted with complex contradictions20.

By combining selective global decay with precise orthogonal erasure, Gated DeltaNet scales up to 99.8% accuracy in complex retrieval tasks, matching the retrieval fidelity of quadratic Transformers while operating strictly in linear time16. To further push performance, researchers have developed hybrid architectures that interleave these memory mechanisms. Gated DeltaNet-H1 intertwines Gated DeltaNet blocks with Sliding Window Attention (SWA), achieving a Wiki perplexity of 16.0720. Gated DeltaNet-H2 adopts a sequence of Mamba2, Gated DeltaNet, and SWA layers, lowering perplexity further to 15.91 and validating the hypothesis that merging global state-space modeling, precise delta-rule erasure, and local exact attention yields superior sequence comprehension20. Furthermore, chunkwise parallelization algorithms—which compute exact intra-chunk recurrences and pass states between inter-chunk boundaries using Unscented Transform (UT) matrix inversions—allow these models to utilize highly optimized matrix-matrix multiplications16.

Test-Time Training (TTT): The Hidden State as a Machine Learning Model

The conceptual boundary between inference and training is entirely dissolved by Test-Time Training (TTT) layers22. Motivated by the observation that Large Language Models compress massive datasets into their weights via self-supervised learning, TTT architectures hypothesis that the hidden state of a sequence model should perform the exact same compression heuristic on the sequence context during inference9. In TTT, the traditional static hidden state vector [Figure omitted from source export] is discarded and replaced by the weights [Figure omitted from source export] of an embedded neural network [Figure omitted from source export] (which can be a linear model or a Multi-Layer Perceptron)8. At each token step, this inner model is optimized via gradient descent against a self-supervised multi-view reconstruction task2. Given an input [Figure omitted from source export], the model generates a training view [Figure omitted from source export] and a label view [Figure omitted from source export]2. The weights are updated by minimizing the loss: [Figure omitted from source export] Following the gradient update: [Figure omitted from source export] the output token is generated by passing a query projection through the newly updated model: [Figure omitted from source export]2. Crucially, the projection matrices [Figure omitted from source export], [Figure omitted from source export], and [Figure omitted from source export] are learned end-to-end during the outer pre-training loop, allowing the self-supervised task to adapt optimally for the primary objective2.

Instantiations: TTT-Linear and TTT-MLP

The instantiation of the hidden state model determines the expressivity and computational overhead of the layer. TTT-Linear utilizes a simple linear model where [Figure omitted from source export], enhanced with layer normalization and residual connections for stability2. Because it requires only matrix-vector operations, it is highly computationally efficient. TTT-MLP introduces a two-layer multilayer perceptron with GELU activations and a hidden dimension four times the input size, allowing for non-linear transformations and greater expressivity at the cost of heavier memory I/O challenges2. Evaluations at scales ranging from 125M to 1.3B parameters demonstrate that similar to standard Transformers, TTT-Linear and TTT-MLP continue to reduce perplexity indefinitely by conditioning on more tokens, whereas models like Mamba hit an asymptote and cannot leverage context effectively past 16K tokens6.

Mini-Batch TTT and Dual Form Optimization

Naively executing gradient descent at every single sequential token step destroys the parallelization efficiency required for modern GPU/TPU execution, making real-world deployment impossible2. To circumvent this, TTT architectures group tokens into mini-batches (commonly sized at [Figure omitted from source export]). Inner-loop gradients are computed for the entire batch simultaneously, balancing online learning with hardware utilization2. More critically, TTT implements a "Dual Form" optimization. The Dual Form is a mathematical refactoring that bypasses the need to materialize the sequential hidden state at every single step9. By leveraging matrix-matrix multiplications, the Dual Form computes all outputs and gradient updates within a sequence chunk in bulk22. This guarantees an identical mathematical output to the sequential formulation but achieves hardware accelerations of over 5x on TPUs, ensuring that models like TTT-Linear scale efficiently past 1.3 billion parameters and achieve faster forward pass latencies than Transformers at context lengths above 8K2.

Neural Long-Term Memory: The Titan Architecture

Drawing direct inspiration from human neuropsychological dual-process memory, the Titan framework structures sequence intelligence into three interacting branches: Short-term memory (attention), Persistent memory (fixed knowledge parameters), and a Deep Neural Long-Term Memory (LMM)4. Developed by researchers at Google, Titans recognize that the most effective way to store unbounded context is not through passive key-value appending, but by training an associative multilayer perceptron during inference1. The core engine of the Titan architecture is governed by "surprise-based memorization," grounded in the psychological phenomenon that unexpected events trigger higher neuroplasticity and stronger memory encoding7.

Surprise, Momentum, and Consolidation

In the Titan framework, the memorability of a token [Figure omitted from source export] is dynamically scored based on its "surprise," mathematically defined as the gradient of the associative loss function with respect to the model parameters: [Figure omitted from source export]

To prevent the neural memory from hyper-focusing on isolated anomalies and causing associative drift, Titans introduce a momentum-based mechanism that tracks the temporal accumulation of surprise across the sequence, smoothing the data-dependent learning rate [Figure omitted from source export]5. Furthermore, an adaptive gating mechanism [Figure omitted from source export] acts as a data-dependent weight-decay scalar, selectively fading older parameters to prevent capacity saturation and manage memory drift over millions of tokens29.

1. If a new token is easily predicted by the current memory state, the gradient is near zero, indicating routine information, and no significant weight update occurs28. If the token wildly deviates from the learned state, the massive gradient triggers a proportional memory consolidation27.

Architectural Topologies: MAC, MAG, and MAL

Titans integrate this LMM module into deep learning networks through three primary architectural variants to balance short-term precision with long-term recall26.

• Memory as Context (MAC): The LMM outputs are treated as historical context and concatenated with the current input sequence and persistent memory. The short-term attention mechanism then natively attends over this enriched context, allowing the attention head to arbitrate between immediate recency and deep historical recall26. MAC excels in sequences exceeding 10 million tokens7.
• Memory as a Gate (MAG): Bypasses concatenation in favor of direct non-linear gating. MAG merges sliding window attention outputs with the neural memory outputs using a learned weighting operation, providing precise deterministic control over how short and long-term memories are integrated26.
• Memory as a Layer (MAL): Embeds the continuous learning mechanism as a distinct sequential layer, functioning similarly to TTT layers but uniquely utilizing the momentum-surprise update rule and data-dependent decay26.

In million-token language modeling and multi-million step needle-in-a-haystack tasks (such as the BABILong benchmark, which tests reasoning across extremely long documents), Titans operating in the MAC topology maintain upwards of 70% accuracy at [Figure omitted from source export] tokens, a horizon where purely recurrent models and standard quadratic transformers catastrophically fail, dropping below 40% accuracy7. In specialized time series forecasting benchmarks (ETTm1, Traffic, Weather), the Titan neural memory actively outperforms both Transformer-based and linear recurrent baselines27.

Mechanics of Memory: Erasure, Contradiction, and Replay

A second-order insight derived from these memory-native systems is that true sequence intelligence requires the ability to handle logical contradictions and consolidate facts asynchronously32. In a standard Transformer KV-cache, a contradiction (e.g., observing a fact "The sky is blue", followed 10,000 tokens later by "The sky is red") results in both facts existing simultaneously in the cache. The attention head must use positional embeddings to heuristically weight the later fact higher32. In contrast, active memory-native systems can perform explicit knowledge editing during the forward pass using exact geometric transformations32.

Orthogonal Projection Erasure for Contradiction Handling

When a deterministic memory engine encounters a contradictory update for a previously normalized key [Figure omitted from source export], it does not simply add the new gradient on top of the old weights. Instead, it executes an orthogonal projection erasure utilizing the identity matrix [Figure omitted from source export]: [Figure omitted from source export]

1. This projection matrix [Figure omitted from source export] zeroes out any weight parameters in the state [Figure omitted from source export] that align with the spatial vector of the key [Figure omitted from source export]. By doing so, the prior fact is surgically annihilated without causing collateral degradation to orthogonal concepts stored in the exact same memory matrix21. This mathematical property transforms the sequence model into a self-correcting knowledge base, heavily resistant to prompt-injection hallucinations and context degradation over time32.

Episodic Buffer and Memory Consolidation

Because single-pass stochastic gradient descent is prone to under-fitting, architectures modeled after mammalian memory systems utilize an episodic buffer to retain highly surprising tokens for "replay"32. This directly mirrors hippocampal replay observed during sleep cycles in biological brains. When the streaming sequence enters a low-information state, the engine accesses its buffer, sorting historical keys by their recorded surprise metrics32. It then executes supplementary forward passes on the top\-[Figure omitted from source export] most surprising facts, tightening the associative bindings (consolidation) and recalculating gradients to ensure the most volatile information is deeply embedded in the persistent state32.

Recurrent Checkpoints and State Serialization

For agents operating over unbounded temporal spans, the state matrix and momentum matrices must be checkpointed to disk or network securely. Because these systems are stateful, identical retrieval behavior across distributed systems requires exact binary parity32. Recurrent checkpoints are achieved by serializing the [Figure omitted from source export] and [Figure omitted from source export] matrices alongside the episodic buffer into a deterministic byte-stream, utilizing big-endian IEEE 754 64-bit floating-point layouts to bypass hardware endianness discrepancies, and verifying integrity through algorithms like the FNV-1a 64-bit hash32.

Cross-Platform Streaming Memory Engine Implementation

To empirically validate the universal compatibility, deterministic predictability, and architectural behavior of these memory mechanics, a complete Streaming Memory Engine is formalized below32. The implementation integrates the surprise-metric calculation, momentum tracking ([Figure omitted from source export]), state decay ([Figure omitted from source export]), orthogonal contradiction handling, episodic buffering, and consolidation replay32. Crucially, this system enforces mathematically identical behavior across Python, C\#, C, Java, and Rust. To guarantee cross-language deterministic verification, the engine implements a 64-bit FNV-1a checksum over the internal state matrices and episodic buffer, strictly utilizing big-endian byte representations32.

1. Python Implementation

The Python variant utilizes the numpy library for highly optimized, vectorized linear algebra operations and the built-in struct library for strict big-endian serialization of the floating-point memory parameters32.

Python import struct import numpy as np

def fnv1a\_64(data: bytes) \-\> int: FNV\_PRIME \= 0x100000001b3 FNV\_OFFSET\_BASIS \= 0xcbf29ce484222325 hash\_val \= FNV\_OFFSET\_BASIS for b in data: hash\_val ^= b hash\_val \= (hash\_val \* FNV\_PRIME) & 0xFFFFFFFFFFFFFFFF return hash\_val

class StreamingMemoryEngine: def \_\_init\_\_(self, d\_key=4, d\_val=4, capacity=32): self.d\_key \= d\_key self.d\_val \= d\_val self.capacity \= capacity self.S \= np.zeros((d\_val, d\_key), dtype=np.float64) self.M \= np.zeros((d\_val, d\_key), dtype=np.float64) self.step\_count \= 0 self.buffer\_keys \= \[\] self.buffer\_vals \= \[\] self.buffer\_surp \= \[\]

def \_norm\_k(self, k): norm \= np.linalg.norm(k) if norm \< 1e-12: return np.zeros\_like(k) return k / norm

def ingest(self, key, val, alpha=0.95, beta=0.8, eta=0.5): k\_hat \= self.\_norm\_k(np.array(key, dtype=np.float64)) v \= np.array(val, dtype=np.float64)

pred \= self.S @ k\_hat err \= pred \- v surprise \= float(np.linalg.norm(err))

grad \= np.outer(err, k\_hat) self.M \= eta \ self.M \+ (1.0 \- eta) \ grad self.S \= alpha \ self.S \- beta \ self.M self.step\_count \+= 1

if len(self.buffer\_keys) \>= self.capacity: self.buffer\_keys.pop(0) self.buffer\_vals.pop(0) self.buffer\_surp.pop(0)

self.buffer\_keys.append(k\_hat) self.buffer\_vals.append(v) self.buffer\_surp.append(surprise) return surprise

def retrieve(self, query, temp=0.5, alpha\_blend=0.7): q\_hat \= self.\_norm\_k(np.array(query, dtype=np.float64)) v\_fw \= self.S @ q\_hat

if len(self.buffer\_keys) \== 0: v\_ep \= np.zeros(self.d\_val, dtype=np.float64) else: sims \= np.array(\[np.dot(q\_hat, bk) / temp for bk in self.buffer\_keys\]) max\_sim \= np.max(sims) exp\_sims \= np.exp(sims \- max\_sim) weights \= exp\_sims / np.sum(exp\_sims) v\_ep \= np.zeros(self.d\_val, dtype=np.float64) for w, bv in zip(weights, self.buffer\_vals): v\_ep \+= w \* bv

v\_out \= alpha\_blend \ v\_fw \+ (1.0 \- alpha\_blend) \ v\_ep return v\_out, v\_fw, v\_ep

def handle\_contradiction(self, key, new\_val): k\_hat \= self.\_norm\_k(np.array(key, dtype=np.float64)) v\_new \= np.array(new\_val, dtype=np.float64) pred\_old \= self.S @ k\_hat severity \= float(np.linalg.norm(pred\_old \- v\_new))

I \= np.eye(self.d\_key, dtype=np.float64) P \= I \- np.outer(k\_hat, k\_hat) self.S \= self.S @ P \+ np.outer(v\_new, k\_hat) self.M \= self.M @ P

for i in range(len(self.buffer\_keys)): if np.dot(self.buffer\_keys\[i\], k\_hat) \> 0.9999: self.buffer\_vals\[i\] \= v\_new self.buffer\_surp\[i\] \= severity

return severity

def consolidate\_replay(self, top\_k=5, replay\_alpha=0.98, replay\_beta=0.5): if len(self.buffer\_keys) \== 0: return 0 indices \= np.argsort(self.buffer\_surp)\[::-1\]\[:top\_k\] for idx in indices: bk \= self.buffer\_keys\[idx\] bv \= self.buffer\_vals\[idx\] pred \= self.S @ bk grad \= np.outer(pred \- bv, bk) self.S \= replay\_alpha \ self.S \- replay\_beta \ grad return len(indices)

def checksum(self) \-\> int: byte\_stream \= bytearray() byte\_stream.extend(struct.pack("\>qqq", self.d\_key, self.d\_val, self.step\_count)) for r in range(self.d\_val): for c in range(self.d\_key): byte\_stream.extend(struct.pack("\>d", self.S\[r, c\])) for r in range(self.d\_val): for c in range(self.d\_key): byte\_stream.extend(struct.pack("\>d", self.M\[r, c\])) byte\_stream.extend(struct.pack("\>q", len(self.buffer\_keys))) for bk, bv, bs in zip(self.buffer\_keys, self.buffer\_vals, self.buffer\_surp): for val in bk: byte\_stream.extend(struct.pack("\>d", val)) for val in bv: byte\_stream.extend(struct.pack("\>d", val)) byte\_stream.extend(struct.pack("\>d", bs)) return fnv1a\_64(bytes(byte\_stream))

2. C# Implementation

To mirror the linear algebraic projections in C\# without relying on massive external dependencies like Math.NET, the implementation utilizes flattened 1D arrays to represent the [Figure omitted from source export] and [Figure omitted from source export] matrices, calculating row/column offsets using manual striding32. The system leverages BitConverter, applying explicit endianness reversal logic where required to strictly match the deterministic Python big-endian byte stream32.

C\# using System; using System.Collections.Generic; using System.Linq;

public class StreamingMemoryEngine { private readonly int d\_key; private readonly int d\_val; private readonly int capacity; private double\[\] S; private double\[\] M; private long step\_count;

private List\<double\[\]\> buffer\_keys \= new List\<double\[\]\>(); private List\<double\[\]\> buffer\_vals \= new List\<double\[\]\>(); private List\<double\> buffer\_surp \= new List\<double\>();

public StreamingMemoryEngine(int d\_key \= 4, int d\_val \= 4, int capacity \= 32) { this.d\_key \= d\_key; this.d\_val \= d\_val; this.capacity \= capacity; this.S \= new double\[d\_val \ d\_key\]; this.M \= new double\[d\_val \ d\_key\]; this.step\_count \= 0; }

private double\[\] NormK(double\[\] k) { double sum \= 0; foreach (var v in k) sum \+= v \* v; double norm \= Math.Sqrt(sum); double\[\] res \= new double\[k.Length\]; if (norm \< 1e-12) return res; for (int i \= 0; i \< k.Length; i++) res\[i\] \= k\[i\] / norm; return res; }

private double\[\] MatVec(double\[\] mat, double\[\] vec, int rows, int cols) { double\[\] res \= new double\[rows\]; for (int r \= 0; r \< rows; r++) for (int c \= 0; c \< cols; c++) res\[r\] \+= mat\[r \ cols \+ c\] \ vec\[c\]; return res; }

public double Ingest(double\[\] key, double\[\] val, double alpha \= 0.95, double beta \= 0.8, double eta \= 0.5) { double\[\] k\_hat \= NormK(key); double\[\] pred \= MatVec(S, k\_hat, d\_val, d\_key); double\[\] err \= new double\[d\_val\]; double sumErrSq \= 0; for (int i \= 0; i \< d\_val; i++) { err\[i\] \= pred\[i\] \- val\[i\]; sumErrSq \+= err\[i\] \* err\[i\]; } double surprise \= Math.Sqrt(sumErrSq);

for (int r \= 0; r \< d\_val; r++) { for (int c \= 0; c \< d\_key; c++) { int idx \= r \ d\_key \+ c; double grad \= err\[r\] \ k\_hat\[c\]; M\[idx\] \= eta \ M\[idx\] \+ (1.0 \- eta) \ grad; S\[idx\] \= alpha \ S\[idx\] \- beta \ M\[idx\]; } } step\_count++;

if (buffer\_keys.Count \>= capacity) { buffer\_keys.RemoveAt(0); buffer\_vals.RemoveAt(0); buffer\_surp.RemoveAt(0); } buffer\_keys.Add(k\_hat); buffer\_vals.Add((double\[\])val.Clone()); buffer\_surp.Add(surprise); return surprise; }

public Tuple\<double\[\], double\[\], double\[\]\> Retrieve(double\[\] query, double temp \= 0.5, double alpha\_blend \= 0.7) { double\[\] q\_hat \= NormK(query); double\[\] v\_fw \= MatVec(S, q\_hat, d\_val, d\_key); double\[\] v\_ep \= new double\[d\_val\];

if (buffer\_keys.Count \> 0) { double\[\] sims \= new double\[buffer\_keys.Count\]; double max\_sim \= double.MinValue; for (int i \= 0; i \< buffer\_keys.Count; i++) { double dot \= 0; for (int j \= 0; j \< d\_key; j++) dot \+= q\_hat\[j\] \ buffer\_keys\[i\]\[j\]; sims\[i\] \= dot / temp; if (sims\[i\] \> max\_sim) max\_sim \= sims\[i\]; } double sumExp \= 0; double\[\] weights \= new double\[buffer\_keys.Count\]; for (int i \= 0; i \< buffer\_keys.Count; i++) { weights\[i\] \= Math.Exp(sims\[i\] \- max\_sim); sumExp \+= weights\[i\]; } for (int i \= 0; i \< buffer\_keys.Count; i++) { double w \= weights\[i\] / sumExp; for (int j \= 0; j \< d\_val; j++) v\_ep\[j\] \+= w \ buffer\_vals\[i\]\[j\]; } }

double\[\] v\_out \= new double\[d\_val\]; for (int i \= 0; i \< d\_val; i++) v\_out\[i\] \= alpha\_blend \ v\_fw\[i\] \+ (1.0 \- alpha\_blend) \ v\_ep\[i\];

return new Tuple\<double\[\], double\[\], double\[\]\>(v\_out, v\_fw, v\_ep); }

public double HandleContradiction(double\[\] key, double\[\] new\_val) { double\[\] k\_hat \= NormK(key); double\[\] pred\_old \= MatVec(S, k\_hat, d\_val, d\_key); double sumSq \= 0; for (int i \= 0; i \< d\_val; i++) sumSq \+= Math.Pow(pred\_old\[i\] \- new\_val\[i\], 2); double severity \= Math.Sqrt(sumSq);

double\[\] P \= new double\[d\_key \ d\_key\]; for (int r \= 0; r \< d\_key; r++) for (int c \= 0; c \< d\_key; c++) P\[r \ d\_key \+ c\] \= (r \== c ? 1.0 : 0.0) \- (k\_hat\[r\] \* k\_hat\[c\]);

double\[\] S\_new \= new double\[d\_val \ d\_key\]; double\[\] M\_new \= new double\[d\_val \ d\_key\];

for (int r \= 0; r \< d\_val; r++) { for (int c \= 0; c \< d\_key; c++) { double s\_sum \= 0, m\_sum \= 0; for (int k \= 0; k \< d\_key; k++) { s\_sum \+= S\[r \ d\_key \+ k\] \ P\[k \ d\_key \+ c\]; m\_sum \+= M\[r \ d\_key \+ k\] \ P\[k \ d\_key \+ c\]; } S\_new\[r \ d\_key \+ c\] \= s\_sum \+ new\_val\[r\] \ k\_hat\[c\]; M\_new\[r \* d\_key \+ c\] \= m\_sum; } } S \= S\_new; M \= M\_new;

for (int i \= 0; i \< buffer\_keys.Count; i++) { double dot \= 0; for (int j \= 0; j \< d\_key; j++) dot \+= buffer\_keys\[i\]\[j\] \* k\_hat\[j\]; if (dot \> 0.9999) { buffer\_vals\[i\] \= (double\[\])new\_val.Clone(); buffer\_surp\[i\] \= severity; } } return severity; }

public int ConsolidateReplay(int top\_k \= 5, double replay\_alpha \= 0.98, double replay\_beta \= 0.5) { if (buffer\_keys.Count \== 0) return 0; var indices \= buffer\_surp.Select((val, idx) \=\> new { val, idx }) .OrderByDescending(x \=\> x.val).Take(top\_k).Select(x \=\> x.idx).ToList();

foreach (int idx in indices) { double\[\] bk \= buffer\_keys\[idx\]; double\[\] bv \= buffer\_vals\[idx\]; double\[\] pred \= MatVec(S, bk, d\_val, d\_key); for (int r \= 0; r \< d\_val; r++) { double err \= pred\[r\] \- bv\[r\]; for (int c \= 0; c \< d\_key; c++) { double grad \= err \ bk\[c\]; S\[r \ d\_key \+ c\] \= replay\_alpha \ S\[r \ d\_key \+ c\] \- replay\_beta \* grad; } } } return indices.Count; }

private void AddBigEndian(List\<byte\> stream, long val) { byte\[\] b \= BitConverter.GetBytes(val); if (BitConverter.IsLittleEndian) Array.Reverse(b); stream.AddRange(b); }

private void AddBigEndian(List\<byte\> stream, double val) { byte\[\] b \= BitConverter.GetBytes(val); if (BitConverter.IsLittleEndian) Array.Reverse(b); stream.AddRange(b); }

public ulong Checksum() { List\<byte\> stream \= new List\<byte\>(); AddBigEndian(stream, (long)d\_key); AddBigEndian(stream, (long)d\_val); AddBigEndian(stream, step\_count); foreach (var v in S) AddBigEndian(stream, v); foreach (var v in M) AddBigEndian(stream, v); AddBigEndian(stream, (long)buffer\_keys.Count);

for (int i \= 0; i \< buffer\_keys.Count; i++) { foreach (var v in buffer\_keys\[i\]) AddBigEndian(stream, v); foreach (var v in buffer\_vals\[i\]) AddBigEndian(stream, v); AddBigEndian(stream, buffer\_surp\[i\]); }

ulong hash \= 0xcbf29ce484222325; ulong prime \= 0x100000001b3; foreach (byte b in stream) { hash ^= b; hash \= hash \* prime; } return hash; } }

3. C Implementation

In pure C, the absence of native garbage collection necessitates precise management of the episodic buffer matrix arrays32. The code models the objects identically to the memory alignment seen in higher-level languages using memmove to safely shift block contexts during buffer capacity evictions32. The system manually inspects architecture byte order via pointer casting to ensure the final FNV-1a checksum receives big-endian bytes32.

C \#include \<stdio.h\> \#include \<stdlib.h\> \#include \<string.h\> \#include \<math.h\> \#include \<stdint.h\>

\#define MAX\_CAPACITY 1024

typedef struct { int64\_t d\_key; int64\_t d\_val; int64\_t capacity; double \S; double \M; int64\_t step\_count;

double \buf\_keys; double \buf\_vals; double \*buf\_surp; int64\_t buf\_len; } StreamingMemoryEngine;

uint64\_t fnv1a\_64(const uint8\_t \data, size\_t len) { uint64\_t hash \= 0xcbf29ce484222325ULL; for (size\_t i \= 0; i \< len; i++) { hash ^= data\[i\]; hash \= 0x100000001b3ULL; } return hash; }

void reverse\_bytes\_if\_little(uint8\_t \bytes, size\_t size) { int x \= 1; if (\(char\*)\&x \== 1) { for (size\_t i \= 0; i \< size / 2; i++) { uint8\_t temp \= bytes\[i\]; bytes\[i\] \= bytes\[size \- 1 \- i\]; bytes\[size \- 1 \- i\] \= temp; } } }

void sme\_init(StreamingMemoryEngine \e, int64\_t d\_key, int64\_t d\_val, int64\_t capacity) { e-\>d\_key \= d\_key; e-\>d\_val \= d\_val; e-\>capacity \= capacity; e-\>S \= calloc(d\_val \ d\_key, sizeof(double)); e-\>M \= calloc(d\_val \ d\_key, sizeof(double)); e-\>step\_count \= 0; e-\>buf\_keys \= calloc(capacity \ d\_key, sizeof(double)); e-\>buf\_vals \= calloc(capacity \* d\_val, sizeof(double)); e-\>buf\_surp \= calloc(capacity, sizeof(double)); e-\>buf\_len \= 0; }

void norm\_k(int64\_t d, const double \k, double \out) { double sum \= 0; for(int i=0; i\<d; i++) sum \+= k\[i\]\*k\[i\]; double norm \= sqrt(sum); if(norm \< 1e-12) { for(int i=0; i\<d; i++) out\[i\] \= 0; } else { for(int i=0; i\<d; i++) out\[i\] \= k\[i\]/norm; } }

double sme\_ingest(StreamingMemoryEngine \e, const double \key, const double \val, double alpha, double beta, double eta) { double \k\_hat \= malloc(e-\>d\_key \* sizeof(double)); norm\_k(e-\>d\_key, key, k\_hat);

double \err \= malloc(e-\>d\_val \ sizeof(double)); double sum\_sq \= 0;

for(int r=0; r\<e-\>d\_val; r++) { double p \= 0; for(int c=0; c\<e-\>d\_key; c++) p \+= e-\>S\[r\e-\>d\_key \+ c\] \ k\_hat\[c\]; err\[r\] \= p \- val\[r\]; sum\_sq \+= err\[r\]\*err\[r\]; } double surprise \= sqrt(sum\_sq);

for(int r=0; r\<e-\>d\_val; r++) { for(int c=0; c\<e-\>d\_key; c++) { int64\_t idx \= r\e-\>d\_key \+ c; double grad \= err\[r\] \ k\_hat\[c\]; e-\>M\[idx\] \= eta \ e-\>M\[idx\] \+ (1.0 \- eta) \ grad; e-\>S\[idx\] \= alpha \ e-\>S\[idx\] \- beta \ e-\>M\[idx\]; } } e-\>step\_count++;

if(e-\>buf\_len \>= e-\>capacity) { memmove(e-\>buf\_keys, e-\>buf\_keys \+ e-\>d\_key, (e-\>capacity\-1)\e-\>d\_key\sizeof(double)); memmove(e-\>buf\_vals, e-\>buf\_vals \+ e-\>d\_val, (e-\>capacity\-1)\e-\>d\_val\sizeof(double)); memmove(e-\>buf\_surp, e-\>buf\_surp \+ 1, (e-\>capacity\-1)\*sizeof(double)); e-\>buf\_len--; }

memcpy(e-\>buf\_keys \+ e-\>buf\_len\e-\>d\_key, k\_hat, e-\>d\_key\sizeof(double)); memcpy(e-\>buf\_vals \+ e-\>buf\_len\e-\>d\_val, val, e-\>d\_val\sizeof(double)); e-\>buf\_surp\[e-\>buf\_len\] \= surprise; e-\>buf\_len++;

free(k\_hat); free(err); return surprise; }

uint64\_t sme\_checksum(StreamingMemoryEngine \e) { size\_t size \= sizeof(int64\_t)\3 \+ sizeof(double)\(e-\>d\_val\e-\>d\_key\2) \+ sizeof(int64\_t) \+ e-\>buf\_len\(sizeof(double)\(e-\>d\_key \+ e-\>d\_val \+ 1)); uint8\_t \stream \= malloc(size); size\_t offset \= 0;

int64\_t header\[3\] \= {e-\>d\_key, e-\>d\_val, e-\>step\_count}; for(int i=0; i\<3; i++) { uint8\_t buf\[8\]; memcpy(buf, \&header\[i\], 8); reverse\_bytes\_if\_little(buf, 8); memcpy(stream \+ offset, buf, 8); offset+=8; } for(int i=0; i\<e-\>d\_val\e-\>d\_key; i++) { uint8\_t buf\[8\]; memcpy(buf, \&e-\>S\[i\], 8); reverse\_bytes\_if\_little(buf, 8); memcpy(stream \+ offset, buf, 8); offset+=8; } for(int i=0; i\<e-\>d\_val\e-\>d\_key; i++) { uint8\_t buf\[8\]; memcpy(buf, \&e-\>M\[i\], 8); reverse\_bytes\_if\_little(buf, 8); memcpy(stream \+ offset, buf, 8); offset+=8; }

uint8\_t buf\_len\[8\]; memcpy(buf\_len, \&e-\>buf\_len, 8); reverse\_bytes\_if\_little(buf\_len, 8); memcpy(stream \+ offset, buf\_len, 8); offset+=8;

for(int i=0; i\<e-\>buf\_len; i++) { for(int j=0; j\<e-\>d\_key; j++) { uint8\_t buf\[8\]; memcpy(buf, \&e-\>buf\_keys\[i\e-\>d\_key \+ j\], 8); reverse\_bytes\_if\_little(buf, 8); memcpy(stream \+ offset, buf, 8); offset+=8; } for(int j=0; j\<e-\>d\_val; j++) { uint8\_t buf\[8\]; memcpy(buf, \&e-\>buf\_vals\[i\e-\>d\_val \+ j\], 8); reverse\_bytes\_if\_little(buf, 8); memcpy(stream \+ offset, buf, 8); offset+=8; } uint8\_t buf\[8\]; memcpy(buf, \&e-\>buf\_surp\[i\], 8); reverse\_bytes\_if\_little(buf, 8); memcpy(stream \+ offset, buf, 8); offset+=8; } uint64\_t hash \= fnv1a\_64(stream, size); free(stream); return hash; }

4. Java Implementation

Java natively handles memory stream generation in big-endian layout via its internal ByteBuffer class, significantly simplifying the deterministic layout verification for the recurrent state32. The arrays map directly onto 2D multidimensional structures for mathematical fidelity while preserving exact object tracking inside the episodic lists32.

Java import java.nio.ByteBuffer; import java.nio.ByteOrder; import java.util.ArrayList; import java.util.Arrays; import java.util.Comparator; import java.util.List;

public class StreamingMemoryEngine { private final int d\_key; private final int d\_val; private final int capacity; private double\[\]\[\] S; private double\[\]\[\] M; private long step\_count \= 0;

private List\<double\[\]\> buffer\_keys \= new ArrayList\<\>(); private List\<double\[\]\> buffer\_vals \= new ArrayList\<\>(); private List\<Double\> buffer\_surp \= new ArrayList\<\>();

public StreamingMemoryEngine(int d\_key, int d\_val, int capacity) { this.d\_key \= d\_key; this.d\_val \= d\_val; this.capacity \= capacity; this.S \= new double\[d\_val\]\[d\_key\]; this.M \= new double\[d\_val\]\[d\_key\]; }

private double\[\] normK(double\[\] k) { double sum \= 0; for (double v : k) sum \+= v \* v; double norm \= Math.sqrt(sum); double\[\] res \= new double\[k.length\]; if (norm \< 1e-12) return res; for (int i \= 0; i \< k.length; i++) res\[i\] \= k\[i\] / norm; return res; }

public double ingest(double\[\] key, double\[\] val, double alpha, double beta, double eta) { double\[\] kHat \= normK(key); double\[\] err \= new double\[d\_val\]; double sumSq \= 0;

for (int r \= 0; r \< d\_val; r++) { double pred \= 0; for (int c \= 0; c \< d\_key; c++) pred \+= S\[r\]\[c\] \ kHat\[c\]; err\[r\] \= pred \- val\[r\]; sumSq \+= err\[r\] \ err\[r\]; } double surprise \= Math.sqrt(sumSq);

for (int r \= 0; r \< d\_val; r++) { for (int c \= 0; c \< d\_key; c++) { double grad \= err\[r\] \ kHat\[c\]; M\[r\]\[c\] \= eta \ M\[r\]\[c\] \+ (1.0 \- eta) \ grad; S\[r\]\[c\] \= alpha \ S\[r\]\[c\] \- beta \* M\[r\]\[c\]; } } step\_count++;

if (buffer\_keys.size() \>= capacity) { buffer\_keys.remove(0); buffer\_vals.remove(0); buffer\_surp.remove(0); } buffer\_keys.add(kHat); buffer\_vals.add(val.clone()); buffer\_surp.add(surprise); return surprise; }

public double handleContradiction(double\[\] key, double\[\] newVal) { double\[\] kHat \= normK(key); double\[\] err \= new double\[d\_val\]; double sumSq \= 0;

for(int r \= 0; r \< d\_val; r++) { double pred \= 0; for(int c \= 0; c \< d\_key; c++) pred \+= S\[r\]\[c\] \ kHat\[c\]; err\[r\] \= pred \- newVal\[r\]; sumSq \+= err\[r\] \ err\[r\]; } double severity \= Math.sqrt(sumSq);

double\[\]\[\] P \= new double\[d\_key\]\[d\_key\]; for(int r \= 0; r \< d\_key; r++) { for(int c \= 0; c \< d\_key; c++) { P\[r\]\[c\] \= (r \== c ? 1.0 : 0.0) \- (kHat\[r\] \* kHat\[c\]); } }

double\[\]\[\] sNew \= new double\[d\_val\]\[d\_key\]; double\[\]\[\] mNew \= new double\[d\_val\]\[d\_key\];

for (int r \= 0; r \< d\_val; r++) { for (int c \= 0; c \< d\_key; c++) { double sSum \= 0, mSum \= 0; for (int k \= 0; k \< d\_key; k++) { sSum \+= S\[r\]\[k\] \ P\[k\]\[c\]; mSum \+= M\[r\]\[k\] \ P\[k\]\[c\]; } sNew\[r\]\[c\] \= sSum \+ newVal\[r\] \* kHat\[c\]; mNew\[r\]\[c\] \= mSum; } } S \= sNew; M \= mNew;

for (int i \= 0; i \< buffer\_keys.size(); i++) { double dot \= 0; for (int j \= 0; j \< d\_key; j++) dot \+= buffer\_keys.get(i)\[j\] \* kHat\[j\]; if (dot \> 0.9999) { buffer\_vals.set(i, newVal.clone()); buffer\_surp.set(i, severity); } } return severity; }

public long checksum() { int bytes \= (3 \ 8) \+ (d\_val \ d\_key \ 8 \ 2) \+ 8 \+ (buffer\_keys.size() \ (d\_key \+ d\_val \+ 1) \ 8); ByteBuffer bb \= ByteBuffer.allocate(bytes); bb.order(ByteOrder.BIG\_ENDIAN);

bb.putLong(d\_key).putLong(d\_val).putLong(step\_count); for (int r \= 0; r \< d\_val; r++) for (int c \= 0; c \< d\_key; c++) bb.putDouble(S\[r\]\[c\]); for (int r \= 0; r \< d\_val; r++) for (int c \= 0; c \< d\_key; c++) bb.putDouble(M\[r\]\[c\]); bb.putLong(buffer\_keys.size());

for (int i \= 0; i \< buffer\_keys.size(); i++) { for (double v : buffer\_keys.get(i)) bb.putDouble(v); for (double v : buffer\_vals.get(i)) bb.putDouble(v); bb.putDouble(buffer\_surp.get(i)); }

long hash \= 0xcbf29ce484222325L; long prime \= 0x100000001b3L; for (byte b : bb.array()) { hash ^= (b & 0xFFL); hash \*= prime; } return hash; } }

5. Rust Implementation

The Rust variant strictly enforces memory safety constraints on the mutable sequential arrays, utilizing flat vector topologies. It avoids unsafe pointer manipulation by leveraging Rust's zero-overhead iterator abstractions to evaluate the mathematical tensors32. Standard big-endian packing is achieved securely using native bitwise f64 conversions before passing the state through the FNV-1a hasher32.

Rust use std::f64;

pub struct StreamingMemoryEngine { d\_key: usize, d\_val: usize, capacity: usize, s\_mat: Vec\<f64\>, m\_mat: Vec\<f64\>, step\_count: i64, buffer\_keys: Vec\<Vec\<f64\>\>, buffer\_vals: Vec\<Vec\<f64\>\>, buffer\_surp: Vec\<f64\>, }

impl StreamingMemoryEngine { pub fn new(d\_key: usize, d\_val: usize, capacity: usize) \-\> Self { Self { d\_key, d\_val, capacity, s\_mat: vec\!\[0.0; d\_val \ d\_key\], m\_mat: vec\!\[0.0; d\_val \ d\_key\], step\_count: 0, buffer\_keys: Vec::new(), buffer\_vals: Vec::new(), buffer\_surp: Vec::new(), } }

fn norm\_k(&self, k: &\[f64\]) \-\> Vec\<f64\> { let sum: f64 \= k.iter().map(|v| v \* v).sum(); let norm \= sum.sqrt(); if norm \< 1e-12 { vec\!\[0.0; self.d\_key\] } else { k.iter().map(|v| v / norm).collect() } }

pub fn ingest(&mut self, key: &\[f64\], val: &\[f64\], alpha: f64, beta: f64, eta: f64) \-\> f64 { let k\_hat \= self.norm\_k(key); let mut err \= vec\!\[0.0; self.d\_val\]; let mut sum\_sq \= 0.0;

for r in 0..self.d\_val { let mut pred \= 0.0; for c in 0..self.d\_key { pred \+= self.s\_mat\[r \ self.d\_key \+ c\] \ k\_hat\[c\]; } err\[r\] \= pred \- val\[r\]; sum\_sq \+= err\[r\] \* err\[r\]; } let surprise \= sum\_sq.sqrt();

for r in 0..self.d\_val { for c in 0..self.d\_key { let idx \= r \ self.d\_key \+ c; let grad \= err\[r\] \ k\_hat\[c\]; self.m\_mat\[idx\] \= eta \ self.m\_mat\[idx\] \+ (1.0 \- eta) \ grad; self.s\_mat\[idx\] \= alpha \ self.s\_mat\[idx\] \- beta \ self.m\_mat\[idx\]; } } self.step\_count \+= 1;

if self.buffer\_keys.len() \>= self.capacity { self.buffer\_keys.remove(0); self.buffer\_vals.remove(0); self.buffer\_surp.remove(0); } self.buffer\_keys.push(k\_hat); self.buffer\_vals.push(val.to\_vec()); self.buffer\_surp.push(surprise);

surprise }

pub fn handle\_contradiction(&mut self, key: &\[f64\], new\_val: &\[f64\]) \-\> f64 { let k\_hat \= self.norm\_k(key); let mut sum\_sq \= 0.0;

for r in 0..self.d\_val { let mut pred \= 0.0; for c in 0..self.d\_key { pred \+= self.s\_mat\[r \ self.d\_key \+ c\] \ k\_hat\[c\]; } let diff \= pred \- new\_val\[r\]; sum\_sq \+= diff \* diff; } let severity \= sum\_sq.sqrt();

let mut p\_mat \= vec\!\[0.0; self.d\_key \ self.d\_key\]; for r in 0..self.d\_key { for c in 0..self.d\_key { let i\_val \= if r \== c { 1.0 } else { 0.0 }; p\_mat\[r \ self.d\_key \+ c\] \= i\_val \- (k\_hat\[r\] \* k\_hat\[c\]); } }

let mut s\_new \= vec\!\[0.0; self.d\_val \ self.d\_key\]; let mut m\_new \= vec\!\[0.0; self.d\_val \ self.d\_key\];

for r in 0..self.d\_val { for c in 0..self.d\_key { let mut s\_sum \= 0.0; let mut m\_sum \= 0.0; for k in 0..self.d\_key { s\_sum \+= self.s\_mat\[r \ self.d\_key \+ k\] \ p\_mat\[k \ self.d\_key \+ c\]; m\_sum \+= self.m\_mat\[r \ self.d\_key \+ k\] \ p\_mat\[k \ self.d\_key \+ c\]; } s\_new\[r \ self.d\_key \+ c\] \= s\_sum \+ new\_val\[r\] \ k\_hat\[c\]; m\_new\[r \* self.d\_key \+ c\] \= m\_sum; } } self.s\_mat \= s\_new; self.m\_mat \= m\_new;

for i in 0..self.buffer\_keys.len() { let dot: f64 \= self.buffer\_keys\[i\].iter().zip(k\_hat.iter()).map(|(a, b)| a \* b).sum(); if dot \> 0.9999 { self.buffer\_vals\[i\] \= new\_val.to\_vec(); self.buffer\_surp\[i\] \= severity; } } severity }

pub fn checksum(&self) \-\> u64 { let mut stream \= Vec::new(); stream.extend\_from\_slice(&(self.d\_key as i64).to\_be\_bytes()); stream.extend\_from\_slice(&(self.d\_val as i64).to\_be\_bytes()); stream.extend\_from\_slice(&self.step\_count.to\_be\_bytes());

for \&v in &self.s\_mat { stream.extend\_from\_slice(\&v.to\_bits().to\_be\_bytes()); } for \&v in &self.m\_mat { stream.extend\_from\_slice(\&v.to\_bits().to\_be\_bytes()); }

stream.extend\_from\_slice(&(self.buffer\_keys.len() as i64).to\_be\_bytes());

for i in 0..self.buffer\_keys.len() { for \&v in &self.buffer\_keys\[i\] { stream.extend\_from\_slice(\&v.to\_bits().to\_be\_bytes()); } for \&v in &self.buffer\_vals\[i\] { stream.extend\_from\_slice(\&v.to\_bits().to\_be\_bytes()); } stream.extend\_from\_slice(&self.buffer\_surp\[i\].to\_bits().to\_be\_bytes()); }

let mut hash: u64 \= 0xcbf29ce484222325; let prime: u64 \= 0x100000001b3; for \&b in \&stream { hash ^= b as u64; hash \= hash.wrapping\_mul(prime); } hash } }

Conclusion

The engineering breakthroughs facilitating memory-native sequence models—including chunkwise parallelism, unrolled matrix optimizations, and mathematically rigorous memory editing—guarantee that the theoretical elegance of Test-Time Training and Neural Long-Term Memory scales robustly onto current-generation GPU and TPU hardware architectures9. As models systematically transition from restrictive 8K context limits to massive 10-million token horizons, the capability to retroactively edit facts via orthogonal projections and consolidate knowledge dynamically through momentum-driven surprise metrics represents a foundational shift in computational intelligence7. Memory-native sequence intelligence proves definitively that network memory is not a passive data repository but an active, continuous, and dynamic computational engine. Sequence processing is transitioning from being purely "spatial"—reliant solely on scaling parameters and network depth during pre-training—to being "temporal", effectively scaling test-time compute by embedding optimization loops directly within inference. Systems equipped with these algorithms can sustain coherent, highly detailed streams of reasoning indefinitely, ensuring that the intelligent agents of the future are no longer bound by the rigid constraints of a frozen inference window, but instead learn, forget, and adapt synchronously with the flow of time.

Works cited

1. \[2501.00663\] Titans: Learning to Memorize at Test Time \- arXiv, https://arxiv.org/abs/2501.00663

2. Learning to (Learn at Test Time): RNNs with Expressive Hidden States, https://www.alphaxiv.org/abs/2407.04620

3. Test-Time Training for RNNs | PDF | Applied Mathematics \- Scribd, https://www.scribd.com/document/781697980/Learning-to-Learn-at-Test-Time-RNNs-with-Expressive-Hidden-States

4. Titans: Learning at Test Time \- Emergent Mind, https://www.emergentmind.com/topics/titans-learning-at-test-time

5. Titans: Learning to Memorize at Test Time \- arXiv, https://arxiv.org/html/2501.00663v1

6. Learning to (Learn at Test Time): RNNs with Expressive Hidden States, https://sophon.at/papers/learning-to-learn-at-test-time-rnns-with-expressive-hidden-states

7. Titans: Learning to Memorize at Test Time \- alphaXiv, https://www.alphaxiv.org/abs/2501.00663

8. \[Literature Review\] Learning to (Learn at Test Time) \- Moonlight, https://www.themoonlight.io/en/review/learning-to-learn-at-test-time-rnns-with-expressive-hidden-states

9. Learning to (Learn at Test Time): RNNs with Expressive Hidden States, https://arxiv.org/pdf/2407.04620

10. ml-zettelkasten/linear-attention.md at main \- GitHub, https://github.com/vishar0/ml-zettelkasten/blob/main/linear-attention.md

11. Transformer based on a variant of attention that is linear ... \- GitHub, https://github.com/lucidrains/linear-attention-transformer

12. Linear Transformers as Fast Weight Programmers \- Emergent Mind, https://www.emergentmind.com/topics/linear-transformers-as-fast-weight-programmers

13. A Swap Is a Reflection: The Delta Rule Behind Qwen's Efficient VLMs, https://pub.towardsai.net/a-swap-is-a-reflection-the-delta-rule-behind-qwens-efficient-vlms-0634540426d6

14. DeltaNet并行化处理的部分关键推导 \- CSDN博客, https://blog.csdn.net/m0\_73678693/article/details/145387311

15. Curvature-aware Sequence Modeling for Linear Recurrences, https://www.alphaxiv.org/resources/2604.21100

16. Gated DeltaNet: The “Surgical Eraser” Solving Linear Attention's, https://pub.towardsai.net/gated-deltanet-the-surgical-eraser-solving-linear-attentions-memory-problem-1e50ca3e42ab

17. Comba: Improving Bilinear RNNs with Closed-loop Control \- alphaXiv, https://www.alphaxiv.org/abs/2506.02475

18. Short Convolution (ShortConv) | Sebastian Raschka, PhD, https://sebastianraschka.com/llm-architecture-gallery/shortconv/

19. State Space Duality (Mamba-2) Part III \- The Algorithm | Tri Dao, https://tridao.me/blog/2024/mamba2-part3-algorithm/

20. Gated Delta Networks: Improving Mamba2 with Delta Rule \- Liner, https://liner.com/review/gated-delta-networks-improving-mamba2-with-delta-rule

21. Gated DeltaNet: Adaptive Fast-Weight Model \- Emergent Mind, https://www.emergentmind.com/topics/gated-deltanet

22. Test-Time Training Layers \- Emergent Mind, https://www.emergentmind.com/topics/test-time-training-ttt-layers

23. Test-Time Training (TTT): A New Approach to Sequence Modeling, https://keyurramoliya.com/posts/Test-Time-Training/

24. Adaptive Compute Allocation for Code Generation via Test-Time, https://arxiv.org/pdf/2601.00894

25. Rethinking Expressivity and Efficiency in Test-Time Training \- arXiv, https://arxiv.org/html/2608.21308v1

26. Titans: Learning to Memorize at Test Time | by Anbu Valluvan, https://medium.com/@avd.sjsu/titans-learning-to-memorize-at-test-time-271e9444e049

27. Titans: Learning to Memorize at Test Time | HP AI Creator Community, https://community.datascience.hp.com/ideas/titans-learning-to-memorize-at-test-time-280

28. Titans: Learning to Memorize at Test Time \- The AI Timeline, https://mail.bycloud.ai/p/titans-memorize-at-test-time

29. Titans: Learning to Memorize at Test Time | HP AI Creator Community, https://community.datascience.hp.com/artificial-intelligence-62/titans-learning-to-memorize-at-test-time-281

30. Titans: Learning to Memorize at Test Time \- OpenReview, https://openreview.net/forum?id=8GjSf9Rh7Z

31. PyTorch implementation of Google Titans: Learning to Memorize at, https://github.com/Aedelon/titans-pytorch-mlx

32. unknown\_url