Out-of-Core Inference and Expert Streaming: An Architectural Analysis of Colibri for TinyRustLM

Full report

On this page

1. Public Colibri Observation and Revision Ledger

The Colibri project, maintained under the JustVugg/colibri repository, represents a highly specialized systems engineering endeavor aimed at executing the 744-billion-parameter GLM-5.2 Mixture-of-Experts (MoE) model on heavily memory-constrained consumer hardware1. The public repository operates as a dependency-free, pure C codebase comprising approximately 2,400 lines of implementation logic (c/glm.c) accompanied by minimal structural headers1. Establishing an immutable observation of the project reveals several critical discrepancies across public records, secondary technical publications, and the primary source repository, all of which must be reconciled before evaluating the architecture for potential integration into the TinyRustLM ecosystem. A central discrepancy exists regarding the software license governing the project. Several secondary technical publications and community analyses assert that the Colibri codebase operates under the MIT license4. However, primary source evaluation of the repository indicates the application of the Apache 2.0 license6. For an enterprise or pre-release product like TinyRustLM, this distinction is paramount, as the Apache 2.0 license includes explicit patent retaliation clauses and specific derivative work distribution conditions that govern how the codebase can be adapted or commercially distributed. The reported hardware footprint consistently demands a baseline of approximately 25 GB of system RAM to hold the dense model backbone, while requiring roughly 370 GB of fast NVMe storage to house the offline routed experts1. The repository’s evolutionary history indicates a pivotal transition in memory management logic that occurred around a commit dated July 10, 2026\. Prior to this revision, the expert Least Recently Used (LRU) cache failed to scale dynamically with available system memory, effectively capping performance on 128 GB machines to match the throughput of 16 GB machines by restricting the cache to 8 experts per layer1. The post-July 10 revision introduced an auto-sizing capability that calculates the \--ram budget by reading operating system availability metrics, fundamentally altering the performance scaling curve1. Furthermore, this same revision period introduced persistent conversational state (.coli\_kv) and learned usage caching (.coli\_usage). These additions transformed the runtime from a static, stateless matrix evaluator into a stateful, learning inference engine capable of incrementally improving its storage hit rate based on historical user query distributions1. The engine currently focuses on Phase 1 completeness (compiled, statically linked correctness), with Phase 2 dedicated to integrating O\_DIRECT for asynchronous file I/O, and subsequent phases targeting GPU offloading via LoadLibrary mechanisms1.

2. Exact Architecture and Data-Flow Reconstruction

Colibri fundamentally reconceptualizes the physical hardware profile of a consumer machine, treating physical RAM, optional VRAM, and solid-state storage as a unified, tightly managed memory hierarchy. This is achieved by exploiting the extreme sparsity inherent in the GLM-5.2 architecture. The model graph is strictly partitioned into two domains: a permanently resident dense backbone and a transient, dynamically loaded pool of routed experts. The dense backbone, comprising approximately 17 billion parameters, encompasses the attention layers, shared experts, token embeddings, and the Multi-Token Prediction (MTP) projection heads3. At int4 quantization, this backbone consumes approximately 9.9 GB of memory, dictating the absolute minimum memory floor for the runtime process3. The attention mechanism relies on a heavily compressed Multi-Head Latent Attention (MLA) implementation that drastically reduces Key-Value (KV) cache memory pressure. It stores only 576 floating-point values per token as opposed to the traditional 32,768 values1. This translates to roughly 182 KB of state per token1, an optimization critical for out-of-core engines because it prevents context growth from evicting the dense weights from the available RAM budget. The routed expert pool consists of 21,504 distinct parameter blocks, mathematically derived from 75 MoE layers containing 256 experts each, plus the speculative MTP heads1. Each expert occupies approximately 19 MB at int4 precision, aggregating to the 370 GB disk footprint1. The execution flow per token is rigidly structured into a sequential five-step pipeline: Route, Union, Place, Overlap, and Learn1. The pipeline begins with the Route phase, where the resident dense routing matrix analyzes the token's hidden state to determine the top-k experts required for the forward pass. This behavior is strictly model-trained; the runtime merely evaluates the probabilities encoded in the trained weights. The Union phase follows, specifically targeting the prefill stage. It deduplicates expert requirements across batched prompt tokens, ensuring that a unique expert required by multiple tokens in the sequence is read from disk only once1. The Placement phase acts as the storage orchestrator, consulting the live LRU cache and the .coli\_usage heatmap to determine which required experts are already resident in memory and which must be fetched from the NVMe drive1. The Overlap phase is the critical latency-hiding mechanism. It dispatches asynchronous I/O requests (PIPE=1) for the missing experts while the CPU concurrently executes the matrix multiplications for the experts that are already resident1. Finally, the Learn phase updates the persistent usage file, effectively shifting the cache admission policy toward the user's specific linguistic or domain distribution over time, allowing the system to achieve higher cache hit rates in subsequent sessions. The MTP speculative decoding logic introduces an additional data-flow loop, leveraging the native MTP head to draft multiple tokens ahead of the current state, which are then verified in a single batched forward pass to accelerate throughput1.

3. Exact-Versus-Lossy Mode Table

For deep systems integration, separating runtime operations that strictly preserve the original transformer mathematics from those that introduce lossy, quality-degrading approximations is paramount. Colibri explicitly defaults to exact routing semantics, avoiding the silent precision degradation frequently observed in high-throughput enterprise inference servers.

The MTP speculative decoding implementation represents a highly sensitive boundary condition. The engine drafts multiple tokens using the MTP head while the current token computes2. However, the design rules explicitly mandate that the draft and verification steps must calculate using the identical internal functions and precision1. Using int4 quantization for the MTP heads drastically reduces the acceptance rate of drafted tokens. When drafts are rejected, the compute cycles spent generating them are wasted, effectively converting a designed performance enhancement into a severe computational and thermal penalty1.

4. Storage, Memory, and I/O Formulas

The fundamental viability of out-of-core inference relies entirely on the underlying physics and capabilities of the storage medium. To comprehend the baseline constraints and establish mathematical limits for a TinyRustLM deployment, rigorous modeling of the exact disk access patterns per token is required. Let [Figure omitted from source export] represent the total number of MoE layers in the model, and [Figure omitted from source export] represent the number of active experts routed per token per layer. Let [Figure omitted from source export] denote the byte size of a single quantized expert. The absolute maximum data fetched from disk for a single token in a completely cold cache state ([Figure omitted from source export]) is given by the formula: [Figure omitted from source export] For the GLM-5.2 implementation operating within Colibri, [Figure omitted from source export], [Figure omitted from source export], and [Figure omitted from source export]. Consequently, the theoretical maximum cold read requirement per token is: [Figure omitted from source export] Assuming an ideal, sequential read profile on a premium PCIe Gen4 NVMe drive with a theoretical bandwidth ([Figure omitted from source export]) of 7 GB/s, the absolute minimum latency boundary ([Figure omitted from source export]) for a cold token is defined as: [Figure omitted from source export] However, observed cold start metrics in Colibri show generation speeds of 0.05 to 0.1 tokens per second, which equates to 10 to 20 seconds per token6. This massive discrepancy between the theoretical sequential bandwidth and the observed throughput indicates severe read amplification and queue depth limitations at the hardware and operating system levels. Because the necessary experts are scattered throughout the massive 370 GB storage file, fetching 8 experts per layer results in [Figure omitted from source export] distinct random read operations per token. On NVMe storage, shallow queue depth random reads do not achieve peak sequential bandwidth. Furthermore, OS page sizes and filesystem block alignments can induce read amplification, where requesting a 19 MB logical block forces the SSD controller to read and discard surrounding data, silently degrading effective throughput. Regarding storage endurance, the primary concern is the physical wear on NAND flash memory cells. Drive endurance is standardized as Terabytes Written (TBW). The formula for daily wear percentage is: [Figure omitted from source export] Because Colibri's 11.4 GB per-token transfer is strictly a read operation, it does not meaningfully degrade SSD lifespan or consume the TBW rating1. The only inherent write amplification originates from the KV cache state persistence (.coli\_kv), which appends approximately 182 KB per token1. Over an extensive 10,000-token generation session, this results in merely 1.82 MB of writes, making the write-wear virtually negligible. The genuine physical danger is sustained thermal output; continuous heavy random read cycles over several hours will heat cheaper DRAM-less SSD controllers, potentially triggering hardware-level thermal throttling that abruptly degrades [Figure omitted from source export] mid-generation1. Furthermore, if the system runs out of physical RAM and the operating system initiates swap thrashing, the heavy, unoptimized write traffic to the disk's swap partition will rapidly consume the drive's TBW rating, leading to premature drive failure1.

5. Cache and Prefetch Design Comparison

To mitigate the catastrophic 11.4 GB per-token I/O bottleneck, Colibri relies on a multi-tiered caching and prefetch architecture, which can be analyzed against leading academic MoE offloading frameworks to isolate optimal strategies for TinyRustLM. The foundational layer of defense is the innate Operating System page cache, which operates transparently as an L2 cache1. However, OS page caches rely on standard Least Recently Used (LRU) algorithms optimized for spatial and temporal locality in traditional file systems. The semantic, data-dependent routing of a neural network violates standard spatial locality; reading expert [Figure omitted from source export] at layer [Figure omitted from source export] provides zero statistical guarantee that expert [Figure omitted from source export] will be required next. To resolve this, Colibri implements an explicit user-space usage cache based on a persistent .coli\_usage heatmap1. By tracking the historical frequency of expert activations across previous conversational turns, the engine mathematically clusters and pins the hottest experts directly into the spare RAM budget1. Because language distributions exhibit power-law characteristics, a small subset of experts absorbs the vast majority of routine syntax and semantic routing, allowing the cache hit rate to climb rapidly from 21% to over 70% during a sustained session9. A more aggressive optimization is the experimental router-lookahead prefetch system, activated via the PILOT=1 flag1. The fundamental insight is that passing the post-attention state of layer [Figure omitted from source export] through the routing function of layer [Figure omitted from source export] can predict the required experts one layer in advance with 71.6% accuracy1. The engine dispatches these predictions to a dedicated asynchronous I/O thread pool (PIPE=1), loading the weights from disk while the main CPU thread computes the current layer's matrix multiplications1. This predictive prefetch aligns with concurrent academic systems engineering. For example, the MoE-Infinity architecture utilizes sequence-level activation tracing to capture temporal locality, enabling activation-aware prefetching that drastically reduces latency compared to naive demand paging10. The PROBE system expands on this via Continuous Lookahead Pipelining, utilizing a Gate-Initialized Lookahead Predictor to distill target routers and hide PCIe transfer latency from the critical path, balancing the workload using a Hardware-Aware Balance Planning solver13. Similarly, the MoE-SpAc framework repurposes speculative decoding as an informative lookahead sensor to drive a Heterogeneous Workload Balancer, effectively turning the memory wall into an asynchronous scheduling optimization problem15. MoE-SpeQ utilizes a heavily quantized, on-device draft model to populate an Expert Lookahead Buffer (ELB), orchestrating entropy-aware caching and near-optimal eviction16. Furthermore, fMoE introduces an "expert map" data structure to track iteration-level probability distributions, leveraging input semantic embeddings to augment the prefetch search, achieving up to a 36% improvement in expert hit rates over standard methods17. At the OS interface level, how these bytes are fetched fundamentally dictates performance. Standard memory mapping (mmap) relies on demand paging, which blocks the execution thread on page faults. This is fatal to deterministic inference latency, as the CPU stalls waiting for the kernel to fulfill the disk interrupt. Buffered reads copy data from disk to kernel space, then to user space, wasting memory bandwidth and CPU cycles on redundant data movement. For maximum performance on Linux, systems must bypass the page cache entirely using Direct I/O (O\_DIRECT), reading straight into aligned user-space buffers via io\_uring or asynchronous pread. Colibri acknowledges this in its roadmap, listing O\_DIRECT integration as Phase 21. Windows environments require similar careful management using FILE\_FLAG\_NO\_BUFFERING combined with overlapped (asynchronous) I/O to achieve zero-copy disk streaming.

6. Failure, Corruption, and Restart State Machine

Out-of-core inference systems are inherently susceptible to hardware state mutations, requiring robust finite state machines for error recovery. When an MoE model streams gigabytes of data continuously, the probability of encountering a short read, a corrupted block, or a device failure increases substantially. If an expert file block is missing or corrupt on disk, the system cannot silently bypass it or substitute zeros. Executing a neural network forward pass with zeroed, NaN, or randomized weights produces catastrophic cascading activation failures, destroying the generated token and poisoning the residual stream. In a strictly deterministic graph, a failure to map or read an expert must immediately halt the generation loop, flush the current token, and return a specific hardware fault code to the orchestrator. If a partial read or short read occurs—where the operating system returns fewer bytes than the requested 19 MB logical chunk—the asynchronous I/O pipeline must implement a robust retry mechanism. The state machine must loop the pread syscall with adjusted offsets until the buffer is entirely filled. If the retry loop exceeds a temporal timeout threshold, it must trigger a controlled process abort. The .coli\_kv file, which persists the compressed MLA KV-cache to disk incrementally, introduces severe write-path vulnerabilities1. Because it appends approximately 182 KB per token to the end of the file, a sudden process crash, memory envelope violation, or power loss during the write system call could corrupt the tail of the cache, rendering the entire conversation history unreadable1. Colibri mitigates this by enforcing crash-safe append operations1. If the disk fills up (ENOSPC error), the write syscall fails. The state machine must intercept this specific error, cease appending to the KV cache, and gracefully degrade functionality. It can either halt inference with an out-of-storage error or dynamically truncate the context window in memory, prioritizing the preservation of the dense backbone memory mappings over historical state retention. Device removal represents a critical failure mode. If an external Thunderbolt NVMe drive containing the model is abruptly unplugged, any active mmap regions backed by the removed device will generate a SIGBUS (Bus Error) signal on Linux upon memory access. Without a specialized signal handler, this instantly terminates the process without cleanup. Therefore, asynchronous direct I/O with explicit bounds checking and error code handling is inherently safer for transient deployment than relying on the kernel's virtual memory subsystem. Finally, if the model files are changed or updated between runs, the .coli\_usage and .coli\_kv structures become instantly stale and incompatible. The engine must verify file integrity, typically via cryptographic content-addressable hashes, before blindly applying a stale placement cache to newly downloaded tensors.

7. Transferability to Champion, Adapters, and Specialists

The TinyRustLM product hypothesis envisions a dual-tier deployment paradigm: a single general "champion" model operating alongside several small "specialist" models, where some artifacts may share exact content-addressed pieces. Evaluating whether Colibri's MoE expert streaming transfers to this paradigm requires distinguishing between mathematically intertwined model structures and standalone executable graphs. An MoE model like GLM-5.2 is monolithic in its design. The 21,504 experts are deeply entangled within the 75 transformer layers. A single token passes through the dense attention mechanisms, hits a layer router, dynamically branches to a specific expert, and then merges back into the residual stream, repeating this 75 times per token generation1. The experts are not independent chatbots, distinct personalities, or standalone modules; they are fragmented feed-forward networks lacking their own tokenizers, positional embeddings, or attention heads. They have absolutely no utility outside the strict confines of the specific MoE layer and specific neural network that trained them. In stark contrast, a specialist model in a portfolio paradigm is a complete, independent computational graph. It contains its own embeddings, attention blocks, FFNs, and output projections. The routing between a champion and a specialist is not performed inside a feed-forward layer at the microsecond level; it is performed at the macro level via user intent classification or meta-prompting before the prompt is processed. Because of this fundamental architectural difference, Colibri's per-token, per-layer streaming logic is entirely inapplicable to the champion-plus-specialist design. The runtime does not need to rapidly fetch 11 GB of 19 MB chunks during a generation loop. Instead, transitioning from a champion to a specialist requires a complete context switch: evicting the champion's unique weights and loading the specialist's weights in bulk. However, the underlying mechanism of content-addressed shared immutable tensor chunks is highly transferable. If the champion and the specialist share foundational layers—similar to how LoRA (Low-Rank Adaptation) sidecars attach to frozen base models—the inference runtime can leverage an architecture similar to the fMoE or MoE-Infinity expert maps to keep shared tensors resident in RAM while only swapping the task-specific projections11. The runtime can treat the specialist models as extremely large, single-layer experts. The caching logic, the auto-sizing RAM budget constraints, and the asynchronous overlap of loading a specialist from disk while the champion finalizes its last output token are direct intellectual inheritances from Colibri's design that can drastically improve TinyRustLM's context-switching latency.

8. Native Companion and Browser Constraint Analysis

Deploying a model runtime directly within a web browser introduces draconian limitations on memory access, file I/O, and concurrent execution. TinyRustLM's core proposition as a Rust/WASM browser-local runtime directly collides with the physical requirements of out-of-core inference. WebAssembly (WASM) operates within a contiguous linear memory sandbox. For standard 32-bit WASM deployments, this linear memory is strictly limited to 4 GB. The dense backbone of GLM-5.2 alone requires 9.9 GB of resident memory3. It is mathematically impossible to load the resident backbone into a standard WASM32 environment, rendering the model categorically incompatible. While WASM64 expands this address space, browser support remains fragmented, and the underlying JavaScript engine's garbage collector and memory allocator impose severe performance penalties on multi-gigabyte contiguous array buffers. Furthermore, the browser lacks the native POSIX semantics required for high-performance disk streaming. There is no mmap, no pread, and no O\_DIRECT. The modern browser storage hierarchy relies on IndexedDB, Cache Storage, and the Origin Private File System (OPFS). IndexedDB is transaction-heavy, relies on structured cloning, and is entirely unsuitable for tens of gigabytes of raw matrix weights. Cache Storage is optimized for HTTP request/response pairs, not arbitrary binary tensor reads. OPFS provides the File System Access API, which exposes a FileSystemSyncAccessHandle. This handle allows synchronous read operations from a Web Worker, which is the closest approximation to pread available in the browser. However, every read operation in OPFS requires copying bytes from the browser's internal storage sandbox into JavaScript memory, and then bridging those bytes into the WASM linear memory heap. This memory-copy overhead scales linearly with the model size. Attempting to stream 11.4 GB of data per token through the OPFS bridging layer would result in latency measured in minutes per token, rather than seconds. Additionally, browser eviction policies treat local storage as ephemeral. If the operating system runs low on storage, the browser may autonomously purge the model weights without warning based on its permission lifecycle, destroying the user experience. Web Workers allow for multithreading, and SharedArrayBuffer allows multiple workers to access the same memory, enabling a browser-based equivalent to Colibri's PILOT=1 prefetch thread. Yet, the security context requirements (Cross-Origin-Opener-Policy and Cross-Origin-Embedder-Policy) make deploying SharedArrayBuffer hostile to casual web embedding, severely limiting its utility for a frictionless product.

9. Design of the Optional Companion Boundary

Given the insurmountable constraints of the browser sandbox, TinyRustLM's optional native companion is the only viable pathway for advanced out-of-core capabilities. The architectural boundary between the browser and the native companion must be designed with explicit consent, strict scoping, and zero implicit trust. The native companion should function as an isolated, local HTTP or WebSocket daemon, exposing a narrow loopback API bound strictly to 127.0.0.1 or ::1. The browser must explicitly request connection to the companion via user opt-in UI elements. Automated port scanning or blind probing from the web context is a security violation, constitutes malicious behavior, and will inevitably trigger browser ad-blocker heuristics and security flags. The companion manages the physical storage hierarchy, executing the high-performance memory mappings, O\_DIRECT reads, and asynchronous I/O that the browser cannot legally perform. It maintains a read-only content-addressed storage pool, where model chunks are identified by cryptographic hashes (e.g., SHA-256) rather than arbitrary file names. This structural design prevents directory traversal attacks and ensures that the browser can verify the integrity of the data it requests before executing it. The API contract must be strictly demarcated to prevent capability creep. The browser orchestrates the user interface, the tokenization, the conversation history, and potentially the execution of small specialist models that fit safely within the WASM memory limit. When a heavy champion model or an out-of-core MoE operation is required, the browser delegates the raw tensor math to the native companion. The native companion returns the computed logits or the finished tokens back over the local socket. Under no circumstances does the native companion upload prompts, telemetry, conversation history, or model files to external project sites; it operates entirely offline. If the native companion is abruptly terminated, disabled by the OS, or the WebSocket connection drops, the browser must gracefully fallback to a truthful error state, relying on a lightweight WASM specialist model to explain the failure to the user without crashing the application tab.

10. Evaluation of an Admitted Small MoE for TinyRustLM

While a 744-billion parameter model is entirely non-viable for browser execution, admitting a heavily distilled, small MoE model (e.g., in the 3B to 8B total parameter range) into the TinyRustLM ecosystem is possible, provided it passes strict, sequential adoption gates. First, the Artifact-Size Gate: The total serialized footprint of the MoE model must fit within the user's acceptable download threshold for web applications. This is typically under 2 GB, necessitating aggressive 3-bit or 4-bit quantization. If the model exceeds the browser's storage quota or memory limit, it fails this gate. Second, the Tokenizer and Context Gate: The small MoE must share the tokenizer with the existing champion models. Shipping multiple megabytes of duplicate Byte-Pair Encoding (BPE) dictionaries and parsing logic to the client wastes bandwidth. Furthermore, the KV cache must be highly compressible, utilizing architectures like MLA, to prevent context explosion from overwhelming the limited WASM sandbox during long sessions. Third, the Expert-Kernel Gate: The computational kernels must compile natively to SIMD-accelerated WASM. If the MoE relies on highly specialized, fused CUDA or Triton kernels to achieve its speed (such as those seen in MoE-Infinity10), it will collapse to scalar fallback speeds in the browser. The architectural complexity of MoE routing destroys performance if the underlying dot-product kernels cannot be vectorized effectively on the CPU. Fourth, the Router and Source-Quality Gate: The router must exhibit extreme sparsity. A small MoE that activates 4 out of 8 experts per token yields only a theoretical 2x compute reduction. In practice, this reduction is entirely erased by the overhead of branching memory accesses and fragmented tensor loads. The routing mechanism must allow for highly deterministic prefetching, similar to Colibri's lookahead1, to ensure the CPU pipeline is not starved waiting for weights to be copied into the active cache line. If a small MoE passes all these gates, it could theoretically serve as a highly efficient champion model within the browser. However, deploying a small MoE natively inside WASM relies entirely on fitting the entire model into RAM. The core Colibri paradigm of disk-streaming experts does not translate; the MoE architecture's sole benefit in the browser is CPU compute reduction (FLOPs), not RAM expansion. Streaming 100 MB chunks per token via OPFS is physically too slow for interactive chat latency.

11. Benchmark and Fault-Injection Plan

To scientifically validate any storage-heavy integration, benchmarks must rigorously isolate the physical I/O constraints from the CPU matrix multiplication speeds. Reporting only the peak tokens per second (tok/s) during a hot-cache generation loop artificially inflates performance and masks the true systems cost. A rigorous local benchmark must report p50, p95, and p99 latencies, cleanly separating the following operational states:

1. Cold Placement: Measure the Time-To-First-Token (TTFT) when the OS page cache is entirely flushed (e.g., via executing echo 3 \> /proc/sys/vm/drop\_caches on Linux) and the explicit .coli\_usage LRU is empty. The benchmark must record the total exact bytes read from the block device to verify mathematical limits.

2. Warm Placement: Measure TTFT and decode latency when the .coli\_usage heat map is fully populated, but the physical RAM cannot hold all experts, forcing a steady state of cache eviction and disk reloading.

3. Prefetch Hit/Miss: Calculate the latency delta between a correctly predicted lookahead route (PILOT=1) and a branch misprediction that stalls the CPU pipeline waiting for synchronous I/O.

4. RAM Tiers: Plot performance degradation curves across strict 16 GB, 24 GB, 32 GB, and 64 GB system memory constraints to observe swap behavior.

5. Storage Classes: Isolate the storage bottleneck by testing across premium NVMe Gen4 SSDs, older SATA SSDs, and mechanical HDDs (to verify the total failure of the streaming paradigm on rotational media).

6. Exact/Lossy Modes: Quantify the latency improvements versus the perplexity degradation when enabling int8 MTP speculative decoding or altering the quantization boundaries.

Fault injection must deliberately trigger OS-level storage failures to validate the state machine. This includes artificially limiting file descriptor counts (ulimit \-n), removing read permissions on expert files mid-generation, simulating ENOSPC during KV cache appending by mapping a heavily constrained loopback filesystem, and throttling the block device queue depth using Linux cgroups to observe how gracefully the engine handles asynchronous I/O stalls without crashing.

12. Adoption Gates and Likely Dead Ends

The fundamentally independent architectural paths of the browser and the native companion necessitate separate, non-overlapping adoption rules for TinyRustLM. For the Browser Lane, the adoption gate is uncompromising: if a generative operation cannot execute entirely within the WASM sandbox using pre-loaded, strictly RAM-resident tensors, it must be rejected. The concept of streaming neural network weights from OPFS on a per-token basis is a physical dead end. The latency introduced by JavaScript-to-WASM memory copy overhead, combined with the lack of asynchronous direct memory mapping (mmap), makes it impossible to achieve real-time performance. Therefore, the browser lane must focus exclusively on aggressively quantized dense models or fully resident small MoEs. For the Native Lane, the adoption gate revolves around software complexity, storage bandwidth, and user expectation management. If the native companion cannot secure raw O\_DIRECT or unbuffered file handles to the local disk, the performance will collapse under OS page-fault overhead and memory thrashing. Colibri successfully proves that out-of-core inference functions correctly if the SSD can sustain massive random-read throughput1. However, the likely dead end here is expecting interactive, real-time generation on typical consumer hardware. Generating 0.1 tokens per second is acceptable for batch processing or offline agentic background tasks6, but it is catastrophic for a consumer-facing chat product. The native companion should adopt expert caching and content-addressed storage for swapping macro-specialists, but should abandon the per-token expert streaming paradigm unless the target hardware profile specifically guarantees unified memory (e.g., Apple Silicon) or extreme PCIe bandwidth.

13. Unknowns Requiring Local Hardware and Source Access

While extensive architectural deductions can be derived from the public repository and supporting academic literature, several critical unknowns remain that require local hardware profiling, compilation, and direct source inspection to resolve for TinyRustLM. First, the exact thermal throttling profiles of consumer SSDs under sustained 100% duty cycle random reads are highly variable. While the math proves NAND wear is negligible for read-heavy workloads, the thermal dissipation capabilities of DRAM-less NVMe drives in thin-and-light laptops are undocumented. Hardware testing is required to measure how quickly the drive drops from PCIe Gen4 speeds to base SATA speeds due to heat saturation, which would exponentially degrade token generation speed over a long conversation. Second, the hidden memory copy overheads in the exact SIMD int4 dequantization routines require instruction-level profiling (perf on Linux). Public benchmarks do not adequately separate the time spent decoding the 4-bit integer weights into fp16/fp32 registers from the time spent performing the actual matrix multiplication. Third, the presence of undocumented memory leaks in the pure C implementation of Colibri cannot be ruled out without executing Valgrind or AddressSanitizer over a prolonged inference session. Because the engine rapidly allocations and frees memory for dynamic experts, heap fragmentation is a significant risk that could artificially inflate the 25 GB memory requirement over time, eventually triggering an OS Out-Of-Memory (OOM) killer event.

14. Annotated Primary-Source Bibliography

This section isolates the critical primary and peer-reviewed materials referenced to deduce the capabilities and limitations of Colibri and related out-of-core inference frameworks, providing immutable context for the architectural analysis.

Works cited

1. colibri/README.md at main \- GitHub, https://github.com/JustVugg/colibri/blob/main/README.md

2. Run GLM 5.2 in Just 25 GB RAM : Colibri | by Mehul Gupta | Data Science in Your Pocket, https://medium.com/data-science-in-your-pocket/run-glm-5-2-in-just-25-gb-ram-colibri-6e2d5b7bb51d

3. Colibri: Running GLM 5.2 on a 32GB Laptop with Disk Streaming and Expert Offloading, https://www.developersdigest.tech/blog/colibri-glm-52-slow-computer-local-inference

4. Colibri Inference: Run 744B MoE Locally \[14.7K Stars, Pure C, 2026 Guide\] | Dailyaiworld, https://dailyaiworld.com/blogs/colibri-glm-52-local-inference-workflow-2026

5. GLM-5.2 Local Deployment: Running Z.ai's 744B Model on Consumer Hardware, https://www.developersdigest.tech/blog/glm-5-2-local-deployment-unsloth-quantization

6. Colibri: The Revolutionary AI Engine Running 744B-Parameter Models on Just 25GB RAM, https://www.alphamatch.ai/blog/colibri-ai-engine-glm-5-2-25gb-ram-2026

7. Colibri Runs GLM-5.2 on Consumer Hardware. Here Is the Catch. \- Wavect, https://wavect.io/blog/colibri-glm-5-2-consumer-hardware/

8. tried predicting which MoE experts get used next token to speed up cpu/gpu offload, got some real numbers, is this actually implementable or am i wasting my time (30tg/s \-\> 150-200tg/s) : r/LocalLLaMA \- Reddit, https://www.reddit.com/r/LocalLLaMA/comments/1uybm8y/tried\_predicting\_which\_moe\_experts\_get\_used\_next/

9. Colibri Hands-on: Running GLM 5.2 (744B) Locally without GPU : r/LocalLLaMA \- Reddit, https://www.reddit.com/r/LocalLLaMA/comments/1uwkz1z/colibri\_handson\_running\_glm\_52\_744b\_locally/

10. EfficientMoE/MoE-Infinity: PyTorch library for cost-effective, fast and easy serving of MoE models. \- GitHub, https://github.com/EfficientMoE/MoE-Infinity

11. MoE-Infinity: Offloading-Efficient MoE Model Serving \- arXiv, https://arxiv.org/html/2401.14361v2

12. MoE-Infinity: Activation-Aware Expert Offloading for Efficient MoE Serving \- Hugging Face, https://huggingface.co/papers/2401.14361

13. PROBE: Co-Balancing Computation and Communication in MoE Inference via Real-Time Predictive Prefetching \- ResearchGate, https://www.researchgate.net/publication/400369736\_PROBE\_Co-Balancing\_Computation\_and\_Communication\_in\_MoE\_Inference\_via\_Real-Time\_Predictive\_Prefetching

14. \[2602.00509\] PROBE: Co-Balancing Computation and Communication in MoE Inference via Real-Time Predictive Prefetching \- arXiv, https://arxiv.org/abs/2602.00509

15. MoE-SpAc: Efficient MoE Inference Based on Speculative Activation Utility in Heterogeneous Edge Scenarios \- arXiv, https://arxiv.org/html/2603.09983v1

16. MoE-SpeQ: Speculative Quantized Decoding with Proactive Expert Prefetching and Offloading for Mixture-of-Experts \- arXiv, https://arxiv.org/html/2511.14102v1

17. \[Literature Review\] fMoE: Fine-Grained Expert Offloading for Large Mixture-of-Experts Serving \- Moonlight, https://www.themoonlight.io/en/review/fmoe-fine-grained-expert-offloading-for-large-mixture-of-experts-serving

18. (PDF) fMoE: Fine-Grained Expert Offloading for Large Mixture-of-Experts Serving, https://www.researchgate.net/publication/388884126\_fMoE\_Fine-Grained\_Expert\_Offloading\_for\_Large\_Mixture-of-Experts\_Serving

19. fMoE: Fine-Grained Expert Offloading for Large Mixture-of-Experts Serving \- arXiv, https://arxiv.org/html/2502.05370v1

20. Unlocking Local AI Potential: Running a 744 Billion Parameter Model on Your Laptop, https://www.franksworld.com/2026/07/16/unlocking-local-ai-potential-running-a-744-billion-parameter-model-on-your-laptop/

21. CoX-MoE: Coalesced Expert Execution for High-Throughput MoE Inference with AMX-Enabled CPU–GPU Co-Execution \- arXiv, https://arxiv.org/html/2605.17889v1

22. CoX-MoE: Coalesced Expert Execution for High-Throughput MoE Inference with AMX-Enabled CPU-GPU Co-Execution \- arXiv, https://arxiv.org/pdf/2605.17889

23. VisMMOE: Exploiting Visual-Expert Affinity for Efficient Visual-Language MoE Offloading, https://arxiv.org/html/2605.05899v1