Architectural Review and Feasibility Analysis of Sub-20M Parameter Language Models for TinyRustLM

Full report

On this page

Introduction: The Convergence of Tiny Models and WebAssembly Environments

The landscape of on-device machine intelligence in 2026 is defined by a rigorous push toward extreme resource efficiency, decoupled architectures, and verifiable local execution. While the broader industry continues to scale large language models into the multi-trillion parameter regime, a parallel and equally vital research vector has matured: the exploration of minimal-parameter neural networks capable of coherent language generation, logic synthesis, and local action evaluation. This paradigm, encapsulated by the MiRust project and the TinyRustLM workspace, focuses on teleodynamic and resource-bounded systems designed to operate safely within severely constrained computing environments.1 In this specific context, teleodynamics refers to the implementation of two-timescale dynamics, endogenous resource coupling, and explicit no-op behaviors that prevent runaway resource consumption during inference cycles.1 The integration of these microscopic transformer models—specifically those positioned under the 20-million parameter threshold—into WebAssembly (WASM) browser environments demands a uniquely high degree of architectural scrutiny. The TinyRustLM 0.1.0 workspace provides a source-grounded implementation map for this endeavor, emphasizing Rust-oriented engineering practices such as the utilization of traits as ports, typestate encapsulation, focused adapters, explicit error handling, and the bounded use of dynamic dispatch.1 To successfully deploy generative models within this strict framework, the resulting compilation artifacts must align with stringent memory boundaries, highly optimized execution pathways, and the custom .slm (SLM1) binary container format.1 This comprehensive research report provides an exhaustive evaluation of viable candidate language models residing strictly within the sub-20M parameter space. The analysis will dissect their architectural topologies, embedding efficiencies, massive activation trajectories, and licensing structures. Furthermore, the report maps these models against the explicit execution limits of WebAssembly 3.0 browser budgets, calculating anticipated SLM1 artifact sizes across 32-bit floating-point (f32), 8-bit quantized (q8), and 4-bit quantized (q4) representations to satisfy the TinyRustLM selector guidelines.

Teleodynamic Systems and the MiRust Framework

Before evaluating specific neural architectures, it is necessary to establish the theoretical and practical governance of the execution environment. The MiRust project operates under a philosophy of resource-bounded machine intelligence.2 The project maintains a strict boundary between its public documentation presence and its executable implementation. The primary web presence, hosted as a WordPress site, ships no Rust runtime, no WebAssembly inference engine, no WebGPU execution pathways, and no model weights.1 Instead, the source-of-truth executable boundary is delegated to a separate implementation project identified as GGUF.MiRust.com, where runtime releases, model artifacts, and measured compatibility are independently governed.1 This separation is critical for enterprise delivery and governance. The core of the TinyRustLM methodology is the implementation of teleodynamic systems.1 A teleodynamic system in this context relies on two-timescale dynamics, meaning the inference loop operates on a rapid, localized timescale while overarching resource management, such as memory cleanup or state transition, operates on a slower, system-wide timescale.1 Endogenous resource coupling ensures that the model cannot attempt to allocate memory or compute cycles that the host environment has not explicitly authorized. Most importantly, the framework demands explicit no-op behavior.1 If a local action evaluation determines that a model's confidence threshold for a given prompt is insufficient, the system must cleanly exit (no-op) without triggering cascading fallback failures. This requires models that exhibit highly predictable activation patterns and strict adherence to defined output schemas.

WebAssembly 3.0 Resource Budgets and Browser Execution Profiles

To determine the viability of a sub-20M parameter model for edge deployment, one must first deeply analyze the WebAssembly execution environment. The release of WebAssembly 3.0 in 2025 introduced foundational shifts in browser-based compute, most notably the integration of the memory64 proposal, native Wasm Garbage Collection (Wasm GC), and WASI Preview 2, which allows for standardized networking and system interfaces outside the browser.4 By 2026, Rust has solidified its position as the de facto language for writing WebAssembly modules due to its lack of an implicit garbage collector, its ability to produce highly compact binary sizes, and its near-native execution speed.4 However, theoretical specification capabilities often diverge significantly from practical browser implementations, requiring careful architectural navigation. The most critical constraint in browser-based machine learning is memory management. While the WebAssembly memory64 specification allows a 64-bit pointer to theoretically address up to 16 exabytes of memory, modern web browsers universally enforce a hard limit of 16 GB for WebAssembly memory allocations to maintain security sandbox integrity and prevent malicious resource monopolization.6 More critically, browser engines, such as V8 in Google Chrome, have spent years heavily optimizing their Just-In-Time (JIT) compilation around 32-bit pointers. Transitioning an application to 64-bit pointers introduces a documented and potentially severe performance penalty depending on the workload, as the engines cannot leverage historical 32-bit optimization heuristics or fast-path memory addressing logic.6 For TinyRustLM, which operates within highly constrained teleodynamic loops, avoiding this 64-bit performance penalty is an absolute necessity. Consequently, the optimal execution pathway targets the standard 32-bit Wasm heap, which imposes a strict maximum of 4 GB.6 A neural network that fits comfortably within a few dozen megabytes—including its key-value (KV) cache, runtime execution arena, and shadow stack—avoids the necessity of crossing into the memory64 domain entirely, thereby preserving maximum inference speed and avoiding JIT de-optimization.1 Furthermore, the WebAssembly memory model consists of distinct architectural regions: a runtime-managed stack (which is strictly isolated from linear memory and managed by the runtime engine), a growable linear memory array, tables for holding function and object references, and a GC heap that is shared with the JavaScript host environment.7 WebAssembly linear memory is an array of bytes that can be read and written via index addressing, and while its size can grow dynamically via the memory.grow instruction, it currently cannot shrink.7 Because WebAssembly modules cannot directly interact with the Document Object Model (DOM), all user interface updates require JavaScript as an intermediary.8 This inherent indirection adds execution complexity and can severely degrade performance in applications that require frequent, high-bandwidth communication across the JS-Wasm boundary.8 The TinyRustLM framework mitigates this boundary friction through a Rust-oriented architecture utilizing asynchronous message passing and focused adapters rather than monolithic synchronous calls.1 In this environment, a sub-20M parameter language model represents the perfect computational artifact. Its entire weight matrix, forward-pass execution graph, and context window buffers can be pinned contiguously in a single linear memory allocation. This maximizes cache-line hits on the host CPU and minimizes the need to copy data structures back and forth across the JavaScript boundary.7

The SLM1 Binary Format and Custom Tensor Layouts

While standard cloud deployment mechanisms often rely on comprehensive container formats like GGML, GGUF, or Safetensors, the TinyRustLM project defines a separate, highly specialized implementation track involving the custom .slm (SLM1) binary format.1 This format is engineered specifically to cater to small-model runtime architectures, focusing heavily on reproducible measurement, strict tensor metadata alignment, and zero-copy loading within WebAssembly environments.1 The design principles of the SLM1 container diverge from traditional machine learning formats by prioritizing deterministic memory initialization. A core feature of the SLM1 format is ahead-of-time (AOT) memory planning. The required activation buffers, scratch spaces, and maximum KV cache sizes are calculated during the model conversion process and baked directly into the binary file header. This allows the host Wasm module to allocate the exact required memory pages upon instantiation via the new WebAssembly.Memory(memoryDescriptor) constructor 9, preventing runtime allocation latency and fragmentation. The memoryDescriptor object utilizes properties such as initial and maximum to hint to the browser engine exactly how many WebAssembly pages (which possess a constant size) the module will require.9 Furthermore, the SLM1 format enforces strict tokenizer boundaries.1 Unlike conventional model deployments that require a separate, computationally heavy Python-based Byte-Pair Encoding (BPE) or SentencePiece execution environment to process text, SLM1 embeds the tokenizer logic directly as static finite state machines mapped to Rust enums (closed domains).1 This architectural choice enables pure-WASM tokenization, eliminating the need for external dependencies and further securing the teleodynamic execution loop.

Edge Hardware Convergence: Optical Neural Networks and SLM1 Hardware Modulators

When evaluating the future viability of sub-20M parameter models and their associated .slm (SLM1) file formats, it is critical to acknowledge the hardware convergence occurring at the extreme edge of machine learning inference. The terminology "SLM1" not only refers to the custom file format utilized by TinyRustLM 1, but it is also the standard nomenclature for Spatial Light Modulators (SLM1, SLM2) used in the physical construction of optical neural networks and photonic Arithmetic Logic Units (ALUs).10 The physics of optical computing present a fascinating parallel to the mathematical requirements of tiny language models. In computing, an optical ALU utilizes longitudinal Orbital Angular Momentum (OAM) light to perform arithmetic and bitwise operations on integer binary numbers without the thermal bottlenecks of silicon electron transit.12 These systems are constructed using an optoelectronic architecture where light passes sequentially through an initial Spatial Light Modulator (SLM1) and a secondary Spatial Light Modulator (SLM2).13 The SLM1 hardware operates in the front focal plane of a Fourier lens, while SLM2 is positioned in the front focal plane of a secondary lens, establishing an object-image conjugation relationship that performs matrix dot multiplication operations sequentially at the speed of light.14 In these physical systems, a circular binary phase pattern is displayed on the SLM1 hardware, utilizing binary modulation where each pixel applies a phase shift corresponding to logical states of 0 and 1 (black and white).13 This physical hardware constraint maps perfectly to the extreme quantization trajectories of the TinyRustLM ecosystem. A sub-20M parameter language model, when quantized down to 1-bit binary weights or specialized integer formats, can theoretically have its entire weight matrix loaded directly onto the high-resolution pixel arrays of physical SLM1 and SLM2 optical hardware panels. While the TinyRustLM workspace currently targets WebAssembly browser environments via its custom .slm binary file container, the architectural choices—specifically the push toward highly compact, low-precision, deterministic matrices—inadvertently prepare these models for near-zero power inference on emerging photonic hardware platforms.1

The Pythia-14M Architecture: Geometric Manifolds and Predictive Trajectories

Transitioning to specific model evaluations, the Pythia suite, developed by EleutherAI, represents a foundational cornerstone for understanding model scaling and interpretability. Engineered specifically to facilitate rigorous research across identical datasets (The Pile), the Pythia suite maintains identical training data orders and architectural philosophies from its smallest to its largest iterations.17 The Pythia-14M model serves as the absolute floor of this scaling suite and provides an incredibly well-documented baseline for TinyRustLM integration. The architectural dimensions of Pythia-14M are highly classical. It features approximately 14 million parameters distributed across 6 hidden layers, utilizing 4 attention heads per layer.18 The hidden dimension size is set to 128, while the intermediate size for the feed-forward network (FFN) expands to 512\.18 The context length, governed by its maximum position embeddings, is generally configured for 512 tokens.20 The architecture employs the GELU activation function and utilizes a standard vocabulary size of 50,257.18 What makes Pythia-14M exceptionally valuable for the teleodynamic framework is the depth of research regarding its internal geometric manifolds and massive activation (MA) patterns. Recent academic studies have successfully mapped the geometric projection of linguistic inputs within Pythia-14M's neural pathways. This research reveals a consistent pattern: the early transformer layers operate on low-dimensional manifolds, the middle layers expand this dimensional space to process complexity and feature interaction, and the later layers compress the data back into a structured, low-dimensional manifold aligned with task-specific decision outputs.22 Furthermore, Pythia-14M is uniquely valuable for analyzing "massive activations." During the course of its training cycle, Pythia-14M demonstrates a top-1 to median activation ratio of 83\.24 Behaviorally, rigorous analysis across more than 1,400 checkpoints reveals that up to 98% of the variance in the model's word-level generation can be mathematically explained by basic unigram frequency and [Figure omitted from source export]\-gram heuristics after the initial training steps (specifically, any step greater than 128).17 The deep reliance on [Figure omitted from source export]\-gram heuristics indicates that Pythia-14M inherently lacks the deep semantic abstraction found in language models that scale past the 100M parameter threshold. However, for TinyRustLM's focus on local action evaluation—such as parsing basic system commands, executing deterministic control flows, or formatting sensor data—deep semantic abstraction is not strictly necessary. Pythia-14M's predictable, mathematically definable activation trajectories make it highly suitable for safety-critical edge constraints where explicit no-op behavior, strict bounds-checking, and diagnostic phase evaluations must be verifiable at runtime.1 From a licensing and governance perspective, Pythia-14M is distributed under the Apache License 2.0.25 This provides robust legal coverage, an explicit grant of patent rights, and a shield against downstream litigation, making it a gold standard for enterprise deployment and redistribution within a compiled .slm format.25

The TinyStories Family: Evaluating GPT-Neo Adaptations and Vocabulary Bloat

The TinyStories dataset and its associated language models, developed by Eldan and Li (2023), caused a paradigm shift in the understanding of parameter efficiency. Their research conclusively demonstrated that language models with fewer than 10 million parameters could generate highly fluent, grammatically perfect English, provided they were trained on a synthetically generated vocabulary heavily restricted to words typically understood by a 3-to-4-year-old child.27 The models hosted under the roneneldan/TinyStories repository, specifically the 1M, 3M, and 8M variants, achieved coherent generation and rudimentary reasoning capabilities.28 The economics of training these models highlight the accessibility of extreme-edge AI. Training the 5M parameter variant on a single H100 80GB GPU requires approximately 6 hours at a cost of roughly $12 USD.29 Scaling up to a 54M parameter variant requires only 8 hours of compute time (\~$16 USD), while a 157M variant requires 16 hours (\~$32 USD).29 Because the training procedure for these highly compact networks is heavily VRAM-bound rather than FLOP-bound, older hardware configurations often present optimal cost efficiency per gigabyte of memory.29 However, when evaluating the TinyStories models for deployment within the highly constrained WebAssembly memory budgets of TinyRustLM, a severe architectural inefficiency becomes glaringly apparent: vocabulary bloat. The models utilize the older GPT-Neo architecture, configured with 8 layers and 16 attention heads.30 The TinyStories-1M variant utilizes a hidden size of 64, while the TinyStories-3M variant utilizes a hidden size of 128\.30 Despite the "1M" designation in its repository title, a programmatic inspection of the TinyStories-1M parameter count reveals an actual size of 3,745,984 parameters.33 This discrepancy stems entirely from the token embedding matrix. The researchers repurposed the EleutherAI/gpt-neo-125M tokenizer, which maintains a massive vocabulary size of 50,257 tokens.33 At a hidden dimension of 64, the embedding matrix alone consumes over 3.2 million parameters (calculated as [Figure omitted from source export]). In TinyStories-1M, approximately 86% of the model's total trained weights are dedicated strictly to the vocabulary embedding lookup table, leaving a mere 500,000 parameters dedicated to logical reasoning, attention mechanics, and sequence generation.33 In a rigid WebAssembly memory budget, loading 3.2 million embedding parameters to output a functional vocabulary that is synthetically restricted to roughly 3,000 common words is a catastrophic misallocation of memory bandwidth and cache space. If this model were to be deployed in the TinyRustLM ecosystem, the SLM1 converter pipeline would need to heavily prune the vocabulary matrix, excising tens of thousands of unused token rows to prevent wasted linear memory allocation.1 Despite these architectural flaws, the TinyStories models and their associated training datasets are distributed under highly permissive licenses (MIT for the codebase wrappers, CDLA-sharing-1.0 for the datasets), making them enterprise-compliant for the MiRust composer ecosystem.34

The stories15M Implementation: Native C Integration and Embedded Space Systems

The stories15M model gained significant prominence through Andrej Karpathy's llama2.c project, which sought to strip away the massive complexity of Python-based machine learning frameworks (such as PyTorch) to execute a standard Llama 2 architecture in pure C with zero external dependencies.37 This minimalist approach perfectly mirrors the Rust-oriented philosophy of TinyRustLM, demonstrating that high-level abstractions are not strictly necessary for capable local generation. The architectural dimensions of stories15M mirror the modern standards set by Meta's Llama series, albeit scaled down drastically. The model features 15.2 million parameters distributed across 6 layers.39 The architecture utilizes advanced normalizations such as RMSNorm, modern positional encodings via Rotary Position Embeddings (RoPE), and SwiGLU feed-forward networks.39 The integration of this architecture into the TinyRustLM ecosystem requires the Rust runtime to natively support complex sinusoidal position calculations for RoPE. As the MiRust workspace explicitly documents scalar transformer paths, standard library implementations of trigonometric functions must be rigorously profiled for Wasm CPU execution overhead, as non-vectorized WebAssembly math operations can introduce pipeline stalls.1 The viability of stories15M has been proven across an astonishing variety of highly constrained devices and esoteric platforms. It has been successfully compiled and executed on legacy Intel Itanium architectures, achieving a throughput of 39 tokens per second when utilizing OpenMP multi-threading.41 More exotically, the 15M architecture was converted into a vanilla Minecraft datapack, allowing inference to run entirely within the game's native tick engine without external server access.42 The most compelling evidence for stories15M as a candidate for teleodynamic evaluation comes from a documented case study regarding AIfES Intelligent Sensor Fault Detection for Embedded Space Systems.43 In this real-world deployment, the language model is gated by an AIfES confidence threshold; it is only invoked when the sensor suite reports a non-normal class with a confidence greater than or equal to 85%.43 This gating logic ensures the LLM fires less than 5% of the time, perfectly encapsulating the teleodynamic principle of explicit no-op behavior.1 The hardware profiling baseline in this study demonstrates that the 60 MB unquantized f32 binary of stories15M requires approximately 100 MB of total RAM at runtime.43 Running on a deeply constrained Raspberry Pi Zero 2W (which possesses a 512MB RAM envelope comparable to an aggressive browser tab budget), the model achieves inference speeds of roughly 200 milliseconds per token.43 Translated to a WebAssembly module executing on a modern desktop client CPU, this translates to real-time interactive generation (10–30 TPS), comfortably fitting within the requirements for rapid local action evaluation.

The StentorLabs Family: Grouped Query Attention and the Depth Delusion

StentorLabs has produced a sequence of neural models targeting the ultra-small regime, culminating in architectures that are heavily optimized based on modern scaling laws rather than legacy configurations. The first generation, Stentor-12M, established a strong baseline. The model features 12,047,040 parameters arranged in 9 layers, utilizing 3 attention heads, 3 key-value (KV) heads, and a head dimension of 64\.44 With a context length of 512 tokens and a vocabulary size of 32,768 (padded to a multiple of 128 for tensor alignment), the model was trained in a remarkably short 4,698 seconds (\~1.3 hours) on a dual Tesla T4 GPU setup, achieving an average throughput of 43,000 tokens per second during training and hitting a best perplexity score of 89.01.44 While impressive, over 52% of Stentor-12M's parameters (approximately 6.29 million) remained locked in the embedding layer due to the 32k vocabulary.44 The subsequent iteration, Stentor3-20M, solves the architectural bottlenecks of its predecessors and represents arguably the most advanced sub-20M architecture available for TinyRustLM mapping. The model features 20,324,160 parameters distributed across 12 layers.46 It utilizes 10 Query (Attention) Heads and 2 KV Heads, an expanded hidden size of 320, a head dimension of 32, and an intermediate FFN size of 1,280 utilizing SwiGLU activations.46 It was trained on advanced TPU v5e-8 hardware for approximately 8.16 hours, achieving a vastly improved perplexity score of 14.02.46 The architectural design of Stentor3-20M is heavily informed by recent scaling research, specifically referencing the "Depth Delusion" (Wu et al., 2026\) and the "Depth Myth" (Izumoto, 2026). This body of research synthesizes findings from over 30 papers to argue that modern LLMs are systematically designed too deep and too narrow, resulting in diminishing returns as depth increases.46 Stentor3-20M implements a "Width over Depth" philosophy, balancing a shallow 12-layer stack with a wide 320-hidden dimension to maximize overall parallel efficiency.46 From a WebAssembly perspective, wide, shallow networks are technically superior to deep, narrow networks. Deep networks force sequential matrix multiplications that cannot be dispatched asynchronously, stalling the runtime pipeline. Conversely, a wide intermediate FFN layer allows Single Instruction, Multiple Data (SIMD) intrinsics in Wasm to process parallel chunks of the 1280-dimension matrix simultaneously, drastically improving tokens-per-second output. Furthermore, Stentor3-20M implements Grouped Query Attention (GQA) with a 5:1 Query-to-KV head ratio.46 While GQA is standard in massive models (such as Llama 3\) to conserve VRAM, applying it at the 20M parameter scale is a highly novel structural decision. By utilizing only 2 KV heads, the memory footprint required for the Key-Value cache during autoregressive generation is reduced by 80%.46 In a WebAssembly browser context, the KV cache grows dynamically in the linear memory with each generated token. A smaller KV cache directly mitigates the need for expensive memory.grow operations, prevents Wasm Garbage Collection stalls, and vastly reduces the likelihood of triggering out-of-memory browser panics. This efficiency is what allows Stentor3-20M to support an unprecedented 4,096-token context window at the 20M scale.46 Finally, Stentor3-20M solves the vocabulary bloat issue that plagued the TinyStories models. By migrating to a highly compact TokenMonster vocabulary of exactly 4,096 tokens (english-4096-strict-nocapcode-v1), the embedding matrix requires a mere 1.3 million parameters (calculated as [Figure omitted from source export]).46 This shifts the dense parameter weight directly into the 12 intermediate hidden layers, allowing the network to rely on complex internal representations rather than surface-level embedding lookups. From a licensing perspective, StentorLabs base models frequently operate under the Open Data Commons Attribution License (ODC-By) v1.0 47, which requires attribution but remains highly suitable for open composer integration within the MiRust enterprise framework.

DataDecide-c4-20M: The Falcon Architecture

A final candidate in the sub-20M parameter space is the DataDecide-c4-20M model. This model utilizes the Falcon architecture and is heavily focused on dataset quality control.48 The DataDecide ecosystem explores the impact of varying levels of quality filtering on the Common Crawl (CC) corpus, providing checkpoints trained on data subjected to 10% and 20% quality filtering thresholds.48 While the Falcon architecture brings robust rotational embeddings and parallel attention mechanisms, the lack of deeply documented edge-deployment case studies compared to stories15M or the structural WASM-aligned innovations of Stentor3-20M makes it a secondary candidate for the highly specific TinyRustLM SLM1 converter pipeline.

Artifact Size Projections and Quantization Mapping

To definitively evaluate the fitness of these models against the TinyRustLM browser memory budgets, we must project the actual physical file size of the compiled .slm (SLM1) artifacts. The final file size is a direct mathematical product of the parameter count and the byte depth of the chosen precision format. The TinyLM-16M reference workload defines standard paths for three precision modalities 1:

• f32 (32-bit floating point): Requires 4 bytes per parameter. This represents the baseline scalar transformer path and provides the highest fidelity, though it consumes the most memory bandwidth.
• q8 (8-bit quantization): Requires 1 byte per parameter. This offers an excellent balance of execution speed and retention of the model's internal geometric manifolds.
• q4 (4-bit quantization): Requires approximately 0.5 bytes per parameter. This format is ideal for edge-constrained memory bandwidth, though it incurs minor unpacking overhead during CPU execution as the 4-bit values must be expanded into working registers for arithmetic calculation.

Note: In all models, a small constant overhead consisting of tensor metadata, hyperparameter headers, and Rust struct padding (approximately 100–200 KB) is added to the SLM1 format to facilitate zero-copy loading.1 The calculations in the matrix below represent pure parameter approximations based on the mathematical sizing formulas.

SLM1 Container Size Matrix

\ Estimated based on 128 hidden dimension matrix interacting with a 50,257 token vocabulary.*

Analysis of WebAssembly Budget Fit

The TinyLM-16M reference workload 1 implicitly sets a target profile around the 15 to 16 million parameter mark for optimal teleodynamic operation. When evaluating the projected SLM1 sizes within the context of WebAssembly browser memory: A q4 artifact of any of these candidates ranges between 1.9 MB and 10.2 MB. This footprint is astonishingly small—often smaller than modern web-delivered hero images, embedded video players, or standard unminified JavaScript library bundles. Loading a 10 MB file into Wasm linear memory leaves nearly the entirety of the 4 GB standard WebAssembly allocation available for the heap and runtime stacks. An f32 artifact of Stentor3-20M (\~81 MB) is comparatively large for a standard web payload, requiring progressive fetching algorithms and persistent caching strategies (such as IndexedDB storage) to prevent unacceptable latency on subsequent page loads. However, once loaded into the WebAssembly linear memory, it requires only a fraction of the theoretical 4 GB (WASM32) or 16 GB (WASM64) browser limit.6 The q8 artifacts present the most logical target for TinyRustLM's deployment via the Web. At sizes ranging from 12 MB to 20 MB, these models fit completely inside the L3 Cache of modern client CPUs (which typically range from 16 MB to 36 MB on consumer hardware). Pinning the memory execution arena directly within the L3 Cache bypasses system RAM bandwidth bottlenecks entirely, dramatically accelerating local action evaluation and satisfying the two-timescale dynamics required by the MiRust framework.1

Architectural Trade-Offs and Rust Typestate Alignment

Evaluating the candidate models against the technical promises of the MiRust framework requires deeply aligning their mathematical topologies with Rust-specific software capabilities. The MiRust project advocates for a strict Rust-oriented architecture, defining traits as ports, utilizing enums for closed domains, and employing typestates and explicitly bound dynamic dispatch.1 In idiomatic Rust, Resource Acquisition Is Initialization (RAII) and the typestate pattern allow developers to guarantee at compile time that a language model transitions linearly through its lifecycle states: Unloaded [Figure omitted from source export] MemoryMapped [Figure omitted from source export] Ready [Figure omitted from source export] Inferencing. The simplistic, uniform architecture of Pythia-14M (a standard transformer with GELU activations and standard multi-head attention) makes it the easiest model to map to rigid Rust typestates.18 Its attention mechanics do not feature complex tensor grouping, meaning the tensor dimensionality remains static and predictable throughout the entire sequence length. Conversely, Stentor3-20M's implementation of Grouped Query Attention (10 Q to 2 KV) 46 requires much more complex tensor broadcasting during the forward pass. In a strict Rust runtime, this broadcast logic must be handled explicitly through zero-cost abstractions to prevent severe matrix-copy overhead. If not carefully designed, the mismatch between query and KV heads during the attention multiplication step could force the Rust compiler to insert dynamic dispatch checks or boundary panics, which violates the TinyRustLM teleodynamic goals.1

The KV Cache WebAssembly Memory Squeeze

As a language model generates text autoregressively, the Key-Value (KV) cache grows linearly. The equation for calculating the KV cache memory size per generated token in standard unquantized precision is: [Figure omitted from source export] To illustrate the architectural superiority of GQA in memory-constrained environments, we must compare the KV cache expansion rates at a 512-token sequence length:

KV Cache Memory Expansion Profiles

Despite Stentor3-20M being 45% larger in raw parameter count than Pythia-14M (20.3M vs 14M), its implementation of Grouped Query Attention cuts its runtime KV cache memory requirement in half. For WebAssembly runtimes where contiguous linear memory allocations can cause fragmentation or require expensive memory resizing operations via memory.grow, Stentor3-20M is vastly superior for sustained local generation.7

Tokenizer Portability and Closed Domains

As established, TinyStories-1M/3M and Pythia-14M utilize the massive 50,000+ GPT-Neo vocabularies.18 In the MiRust paradigm, tokenizers are meant to be compiled into WebAssembly as self-contained finite state machines leveraging Rust enums to represent closed domains.1 A vocabulary of 50,000 tokens results in a massive trie or regex-based tree structure that swells the binary size of the WASM file itself. The TokenMonster 4,096 vocabulary utilized by Stentor3-20M 46 fits elegantly into small Rust enum structures. This alignment drastically shrinks the Wasm binary payload, parsing overhead, and compilation time, leaving more browser budget for the model weights themselves.

Strategic Conclusions and Teleodynamic Recommendations

The objective of identifying viable sub-20M parameter language models for the TinyRustLM workspace requires satisfying a complex intersection of variables: WebAssembly browser memory constraints, 32-bit execution penalties, teleodynamic runtime loops, optical ALU-adjacent binary architectures, and the strict SLM1 container specifications.1 Based on exhaustive architectural evaluation and memory profiling, the field of candidates stratifies into distinct utility tiers, providing a clear roadmap for the GGUF.MiRust.com implementation boundary.

1. The Superior Architecture: Stentor3-20M At 20.3 million parameters, the Stentor3-20M model presses directly against the upper bound of the designated parameter ceiling. However, its architectural decisions are flawlessly aligned with WebAssembly physical realities. By aggressively utilizing Grouped Query Attention (a 5:1 ratio), it drastically minimizes the runtime KV cache footprint 46, allowing for deep context generation without triggering Wasm memory grow stalls. By compressing its vocabulary to a 4,096-token TokenMonster configuration, it reallocates parameter weight from static embedding lookup tables into the active 12-layer feed-forward networks, maximizing semantic logic without bloating memory. Furthermore, its "Width over Depth" philosophy is perfectly suited for SIMD vectorization in Rust. For the TinyLM-16M reference workload, scaling this structure into the SLM1 format as a 10.2 MB q4 artifact or a 20.3 MB q8 artifact provides the absolute best synergy of capability, context length, and performance.
2. The Enterprise Baseline: Pythia-14M At exactly 14 million parameters, Pythia-14M is the most structurally transparent and legally secure option available. The Apache 2.0 license offers enterprise-grade safety 25, providing an explicit grant of patent rights that protects the MiRust ecosystem from downstream litigation. Its reliance on basic low-dimensional manifolds makes its reasoning pathways highly predictable and quantifiable.17 For environments strictly requiring diagnostic predictability over generation creativity—such as parsing explicit no-op commands in teleodynamic loops or gating logic similar to the AIfES sensor space deployment 43—a 7.0 MB q4 or 14.0 MB q8 SLM1 file of Pythia-14M is a highly resilient, enterprise-ready fallback option.
3. The Proven Embedded Pathway: stories15M The 15.2 million parameter stories15M model 40 has already proven its viability in the harshest computing environments, running on legacy Intel Itanium systems, embedded space-system hardware, and game engines.41 Its utilization of the Llama 2 architecture provides modern normalization and positional encodings. While it lacks the GQA efficiency of Stentor3-20M, its proven C/Rust execution history makes it an incredibly safe bet for the SLM1 converter pipeline.
4. The Deprecated Pathway: The TinyStories Family While groundbreaking in 2023 for proving the concept of sub-10M parameter language generation, the GPT-Neo architecture utilized in the TinyStories family (1M, 3M, 8M) is fundamentally misaligned with edge deployment realities. Forcing a 50,257-token vocabulary onto a 3.7 million parameter model dictates that 86% of the parameters are functionally inert during reasoning.33 The memory bandwidth spent moving this massive lookup table across the WebAssembly linear memory provides no teleodynamic value, and these models should be deprecated in favor of Stentor3-20M or Pythia-14M.

To align with the MiRust commitments of resource-bounded systems 2, the execution track should prioritize a parsing and quantization pipeline focused heavily on Stentor3-20M. The target q8 quantization yields an approximately 20.3 MB SLM1 file. This size perfectly navigates the browser's 32-bit execution optimizations, fits gracefully inside modern CPU L3 caches for immediate zero-copy execution, and operates entirely independently of the unstable 64-bit memory penalties and Wasm GC bottlenecks present in modern browser environments.

Works cited

1. Research \- MiRust, accessed June 30, 2026, https://mirust.com/research/
2. The five commitments of Teleodynamic Learning \- MiRust, accessed June 30, 2026, https://mirust.com/five-commitments-of-teleodynamic-learning/
3. MiRust site layout hotfix 1.1.1, accessed June 30, 2026, https://mirust.com/mirust-site-layout-hotfix-1-1-1/
4. Rust \+ WebAssembly in 2026 — Complete Guide (Browser, WASI, Edge, Optimisation), accessed June 30, 2026, https://www.youtube.com/watch?v=N25oMCsyaZ0
5. Making a budget Pascal compiler to WebAssembly | Hacker News, accessed June 30, 2026, https://news.ycombinator.com/item?id=30298818
6. The State of WebAssembly – 2025 and 2026 \- Uno Platform, accessed June 30, 2026, https://platform.uno/blog/the-state-of-webassembly-2025-2026/
7. WebAssembly Limitations | qouteall notes, accessed June 30, 2026, https://qouteall.fun/qouteall-blog/2025/WebAsembly%20Limitations
8. WebAssembly Limitations: What Developers Need to Know in 2026 | IBK Technet Hub, accessed June 30, 2026, https://ibktechnethub.synergize.co/blog/webassembly-limitations/
9. WebAssembly.Memory() constructor \- MDN Web Docs \- Mozilla, accessed June 30, 2026, https://developer.mozilla.org/en-US/docs/WebAssembly/Reference/JavaScript\_interface/Memory/Memory
10. Digital phase-shift mask projection lithography enabling sub-diffraction-limit resolution for dense nanoscale patterning | Light, accessed June 30, 2026, https://www.light-am.com/article/pdf/preview/LAM2026020051.pdf
11. Limitations of a class of binary phase-only filters \- Optica Publishing Group, accessed June 30, 2026, https://opg.optica.org/abstract.cfm?uri=ao-31-26-5681
12. Arithmetic with spatiotemporal optical vortex of integer and fractional topological charges \- arXiv, accessed June 30, 2026, https://arxiv.org/pdf/2512.25049
13. In vivo volumetric imaging of calcium and glutamate activity at synapses with high spatiotemporal resolution \- PMC, accessed June 30, 2026, https://pmc.ncbi.nlm.nih.gov/articles/PMC8595604/
14. Optical convolution accelerator based on SLM's imaging-casting configuration, accessed June 30, 2026, https://opg.optica.org/abstract.cfm?uri=optcon-4-9-2053
15. Experimental Investigation of Optical Processing With Spatial Light Modulation \- arXiv, accessed June 30, 2026, https://arxiv.org/html/2507.03821v1
16. Experimental Investigation of Optical Processing With Spatial Light Modulation \- arXiv, accessed June 30, 2026, https://arxiv.org/pdf/2507.03821
17. Language Model Behavioral Phases are Consistent Across Architecture, Training Data, and Scale \- arXiv, accessed June 30, 2026, https://arxiv.org/html/2510.24963v1
18. arXiv:2404.00859v2 \[cs.LG\] 1 Aug 2024, accessed June 30, 2026, https://arxiv.org/pdf/2404.00859
19. config.json · honicky/pythia-14m-hdfs-logs at main \- Hugging Face, accessed June 30, 2026, https://huggingface.co/honicky/pythia-14m-hdfs-logs/blob/main/config.json
20. Add files using upload-large-folder tool · chinese-babylm-org/babylm-chinese-pythia-14M-epoch10 at add0793 \- Hugging Face, accessed June 30, 2026, https://huggingface.co/chinese-babylm-org/babylm-chinese-pythia-14M-epoch10/commit/add07937ff505431301ebab5d7203f0c7e7d265c
21. Upload folder using huggingface\_hub · yikang0131/pythia\_chinese-llama\_14m\_1B\_seed30 at 2a48e11, accessed June 30, 2026, https://huggingface.co/yikang0131/pythia\_chinese-llama\_14m\_1B\_seed30/commit/2a48e1183fca57020790b1171f32c4ee2a9ccfad
22. (PDF) Geometry of Decision Making in Language Models \- ResearchGate, accessed June 30, 2026, https://www.researchgate.net/publication/397983459\_Geometry\_of\_Decision\_Making\_in\_Language\_Models
23. Geometry of Decision Making in Language Models \- OpenReview, accessed June 30, 2026, https://openreview.net/pdf/4f7f449178a12ce539315d6e395db6a68538b54d.pdf
24. Hidden Dynamics of Massive Activations in Transformer Training \- arXiv, accessed June 30, 2026, https://arxiv.org/html/2508.03616
25. Pythia: Interpreting Transformers Across Time and Scale \- GitHub, accessed June 30, 2026, https://github.com/eleutherai/pythia
26. pythia-14M \- Model Openness Tool, accessed June 30, 2026, https://mot.isitopen.ai/model/pythia-14M
27. TinyStories: How Small Can Language Models Be and Still Speak Coherent English? \- arXiv, accessed June 30, 2026, https://arxiv.org/abs/2305.07759
28. TinyStories: How Small Can Language Models Be and Still Speak Coherent English? \- arXiv, accessed June 30, 2026, https://arxiv.org/pdf/2305.07759
29. Regional-TinyStories \- Hugging Face, accessed June 30, 2026, https://huggingface.co/TinyStories-Regional
30. config.json · roneneldan/TinyStories-1M at refs/pr/11 \- Hugging Face, accessed June 30, 2026, https://huggingface.co/roneneldan/TinyStories-1M/blame/refs%2Fpr%2F11/config.json
31. config.json · onnx-community/TinyStories-3M-ONNX at, accessed June 30, 2026, https://huggingface.co/onnx-community/TinyStories-3M-ONNX/blame/b8a0cb31c28e16808f6c5aaf0b02e2a046681528/config.json
32. config.json · onnx-community/TinyStories-3M-ONNX at main, accessed June 30, 2026, https://huggingface.co/onnx-community/TinyStories-3M-ONNX/blob/main/config.json
33. roneneldan/TinyStories-1M · Actual number of parameters? \- Hugging Face, accessed June 30, 2026, https://huggingface.co/roneneldan/TinyStories-1M/discussions/5
34. Train Tiny Stories dataset from the paper \- "TinyStories: How Small Can Language Models Be and Still Speak Coherent English?" \- GitHub, accessed June 30, 2026, https://github.com/SauravP97/tiny-stories-hf
35. roneneldan/TinyStories · Datasets at Hugging Face, accessed June 30, 2026, https://huggingface.co/datasets/roneneldan/TinyStories
36. Code for the TinyStories experiments from "Mechanistically analyzing the effects of fine-tuning on procedurally defined tasks". \- GitHub, accessed June 30, 2026, https://github.com/RobertKirk/tinystories-wrappers
37. GitHub \- karpathy/llama2.c: Inference Llama 2 in one file of pure C, accessed June 30, 2026, https://github.com/karpathy/llama2.c
38. llama2.c/README.md at master · karpathy/llama2.c · GitHub, accessed June 30, 2026, https://github.com/karpathy/llama2.c/blob/master/README.md
39. llama2, accessed June 30, 2026, https://www.marble.onl/posts/llama2.html
40. ModelCloud/tinyllama-15M-stories \- Hugging Face, accessed June 30, 2026, https://huggingface.co/ModelCloud/tinyllama-15M-stories
41. Running Llama inference on Intel Itanium, part 1 | by Tomáš Glozar | Medium, accessed June 30, 2026, https://medium.com/@tglozar/running-llama-inference-on-intel-itanium-part-1-be62ff3f5c2f
42. Running Large Language Model in Vanilla Minecraft : r/MinecraftCommands \- Reddit, accessed June 30, 2026, https://www.reddit.com/r/MinecraftCommands/comments/1sbhgyf/running\_large\_language\_model\_in\_vanilla\_minecraft/
43. CASE STUDY Tiny LLMs in C \+ AIfES Intelligent Sensor Fault Detection for Embedded & Space Systems \- ResearchGate, accessed June 30, 2026, https://www.researchgate.net/publication/405222924\_CASE\_STUDY\_Tiny\_LLMs\_in\_C\_AIfES\_Intelligent\_Sensor\_Fault\_Detection\_for\_Embedded\_Space\_Systems
44. StentorLabs/Stentor-12M \- Hugging Face, accessed June 30, 2026, https://huggingface.co/StentorLabs/Stentor-12M
45. README.md · Flexan/StentorLabs-Stentor-12M-Instruct-GGUF at main \- Hugging Face, accessed June 30, 2026, https://huggingface.co/Flexan/StentorLabs-Stentor-12M-Instruct-GGUF/blob/main/README.md?code=true
46. StentorLabs/Stentor3-20M \- Hugging Face, accessed June 30, 2026, https://huggingface.co/StentorLabs/Stentor3-20M
47. epfml/FineWeb-HQ · Datasets at Hugging Face, accessed June 30, 2026, https://huggingface.co/datasets/epfml/FineWeb-HQ
48. allenai/DataDecide-c4-20M \- Hugging Face, accessed June 30, 2026, https://huggingface.co/allenai/DataDecide-c4-20M