Sub-Billion Capacity Allocation, Tokenizer Economics, and Student Blueprint

Full report

On this page

1. Executive Recommendation and Assumptions

The deployment of large language models in edge environments, specifically within standard CPU hardware and browser-based WebAssembly (WASM) runtimes, enforces rigorous constraints on parameter counts, memory footprints, and computational complexity. The following analysis demonstrates that sub-billion parameter models operating in the 0.3B to 1.5B range can execute practical conversational tasks, structured output generation, and instruction following, provided that architectural redundancy is aggressively pruned and tool-delegation boundaries are strictly enforced. The fundamental assumptions governing this research are rooted in the physical and systemic limitations of the target environment. First, the inference environment is constrained by the 32-bit address space limit inherent to standard WASM implementations, which functionally caps memory at 4GB. Practical stability requires the combined footprint of the model weights, runtime state, Key-Value (KV) cache, and host-copy duplication to remain below 2.5GB to avoid out-of-memory browser crashes1. Second, long-context configurations do not inherently guarantee long-context reasoning capabilities. Extending context lengths in sub-billion models primarily serves retrieval-augmented generation and deterministic summarization, rather than complex multi-turn reasoning over extended horizons. Third, a smaller artifact provides value only if the compression preserves the syntactic coherence and semantic reliability of the model. Extreme quantization paradigms or arbitrary layer dropping that destroys representation stability are rejected in favor of targeted architectural distillation and quantization-aware training. The subsequent sections establish a comprehensive framework for designing, training, and deploying student blueprints that maximize the utility of the restricted parameter budget while maintaining cryptographic provenance over tokenizer identity and chat templates.

2. Public Configuration and License Matrix

The current landscape of sub-billion and low-billion parameter models features diverse architectural choices regarding vocabulary scaling, attention mechanisms, hidden dimensions, and depth. Table 1 outlines the exact configurations of leading dense and hybrid models in this regime, providing a comparative baseline for student blueprint design.

All observed Transformer models utilize Rotary Positional Embeddings, SwiGLU activation functions in the feed-forward network, and Root Mean Square Normalization. The Qwen2.5 series introduces QKV attention biases, whereas Llama-3.2 and SmolLM2 omit biases entirely to maximize parameter efficiency15. The Mamba-370M model represents a State Space Model (SSM) architecture, discarding traditional attention mechanisms in favor of selective state spaces to achieve linear time sequence modeling17.

3. Recomputed Parameter Ledger

A precise accounting of model parameters reveals significant discrepancies between advertised, rounded counts and the exact active parameters. The discrepancy between non-embedding parameters and total parameters is frequently obscured in model cards to make architectures appear more computationally efficient than their memory footprint implies. The fundamental formula for calculating total active parameters in a standard causal Transformer is defined as the sum of embedding parameters, layer block parameters, and final normalization scale parameters: [Figure omitted from source export] The individual components are derived as follows: The embedding matrix parameters ([Figure omitted from source export]) equal the vocabulary size ([Figure omitted from source export]) multiplied by the hidden dimension ([Figure omitted from source export]). If embeddings are untied, an identical number of parameters is required for the output head ([Figure omitted from source export]). The attention parameters ([Figure omitted from source export]) encompass the query, key, value, and output projections. For Grouped-Query Attention, this is calculated as [Figure omitted from source export], alongside any applicable biases. The feed-forward network parameters ([Figure omitted from source export]) utilizing a SwiGLU activation function require three weight matrices (Gate, Up, Down), calculated as [Figure omitted from source export]. Normalization parameters ([Figure omitted from source export]) for RMSNorm require [Figure omitted from source export] parameters per layer, plus [Figure omitted from source export] for the final normalization block. Recurrent blocks, such as those found in Mamba-2 architectures, follow a different derivation based on State Space Duality. The parameters are allocated to the input projection, 1D convolution, and SSM matrices. The total SSM parameter footprint per layer is driven by the state dimension ([Figure omitted from source export]), the head dimension, and the expansion factor, substituting the quadratic [Figure omitted from source export] overhead with linear state projections19. Low-Rank Adaptation (LoRA) modules modify the active parameter count during fine-tuning. An adapter applied to a weight matrix of dimension [Figure omitted from source export] with a rank [Figure omitted from source export] adds [Figure omitted from source export] parameters21. Applying these derivations reveals the true parameter footprints of contemporary models. For the Llama-3.2-1B model, the embedding matrix requires [Figure omitted from source export] parameters. The 16 transformer blocks require [Figure omitted from source export] parameters each, totaling [Figure omitted from source export]. Combining these with the final norm yields an exact active parameter count of [Figure omitted from source export]9. The advertised count is rounded down to 1B, ignoring the fact that the actual parameter count is approximately 23.5% larger. Similarly, the Qwen2.5-0.5B model advertises 0.5B parameters, but the exact calculation yields [Figure omitted from source export] parameters, of which the embedding matrix alone consumes [Figure omitted from source export] parameters, or 27.5% of the total budget7. The vocabulary size dictates the proportion of the parameter budget available for the reasoning layers. Table 2 illustrates the maximum number of standard transformer blocks that can fit within a strict parameter budget when accommodating different vocabulary sizes, assuming a constant hidden dimension of [Figure omitted from source export], [Figure omitted from source export], [Figure omitted from source export], and [Figure omitted from source export].

In the sub-billion regime, massive vocabularies heavily cannibalize the useful transformer-block budget. A 0.3B model adopting a 100K vocabulary loses 25% of its potential depth compared to a model with a pruned 32K vocabulary.

4. Tokenizer and Embedding Economics

The decision to reuse a large teacher tokenizer versus training a specialized, smaller student tokenizer represents a critical tension in sub-billion capacity allocation. Tokenization algorithms—whether Byte Pair Encoding, WordPiece, or Unigram—determine how raw text is mapped into discrete integer IDs23. This mapping dictates downstream performance, cross-tokenizer distillation compatibility, and WASM memory limits. Frontier models such as Qwen2.5 and Llama-3.2 employ massive vocabularies of 151K and 128K, respectively25. Large vocabularies increase the information density per token, thereby accelerating autoregressive generation speed and reducing the length of the required KV cache context. However, as demonstrated in the parameter ledger, these vocabularies impose a severe memory penalty on the embedding matrix. Reusing a 150K teacher tokenizer in a 0.3B student model forces a catastrophic reduction in the network's reasoning depth. Vocabulary pruning presents a mathematically superior alternative for edge deployments. Pruning an inherited teacher vocabulary down to 32,000 tokens by excising rare multilingual characters, esoteric symbols, and underutilized code chunks saves substantial memory while preserving core English and syntactic logic26. To prevent catastrophic failures when encountering pruned characters, a robust byte fallback mechanism must be implemented. Byte fallback ensures that out-of-vocabulary terms and complex Unicode characters are not destructively collapsed into a generic unknown token. Instead, they are gracefully decomposed into their constituent UTF-8 byte sequences, allowing the model to process arbitrary strings at the cost of temporary sequence elongation23. However, adopting a pruned or completely novel tokenizer severs the direct probabilistic link between the teacher and the student, creating a significant hurdle for knowledge distillation. Standard Forward Kullback-Leibler divergence matching requires the teacher and student to share an identical output distribution space27. When vocabularies mismatch, direct logit distillation fails because the indices no longer align, forcing practitioners to rely on suboptimal sequence-level distillation or hard-label fine-tuning29. Recent advancements in cross-tokenizer distillation resolve this economic friction. The X-Token framework introduces a sparse projection matrix constructed via canonicalized string matching, enabling deterministic alignment across divergent tokenizers by mathematically bridging the vocabularies during the distillation loss calculation27. Alternatively, Byte-Level Distillation provides a universal interface by mapping the teacher's token-level output distribution into byte-level probabilities. The student attaches a temporary, lightweight byte-level decoder head during training to match these probabilities. Once distillation concludes, the byte-level head is discarded, leaving a highly capable student that absorbed the teacher's dense knowledge despite possessing a radically smaller, memory-efficient vocabulary31.

5. Depth, Width, Attention, and State Tradeoffs

The architectural geometry of a sub-billion student must balance sequential execution costs against representational expressivity. The configuration of the embedding matrices, the aspect ratio of the layers, and the nature of the attention mechanism directly dictate the model's latency in a browser environment. The architectural decision to tie or untie the input embedding matrix and the final output unembedding matrix involves a strict trade-off between parameter efficiency and representation quality. In models under 1.5B parameters, weight tying is predominantly utilized to halve the vocabulary parameter footprint33. However, tying forces a single matrix to optimize for two conflicting objectives: encoding the semantic similarity of input tokens and decoding the contextual probability of output tokens35. Empirical analysis demonstrates that tied embeddings suffer from "unembedding bias," where the strong cross-entropy gradients from the output layer dominate the matrix33. This causes the shared embedding space to become highly anisotropic, clustering token vectors into a narrow geometric cone and degrading the model's internal representations35. Untied models achieve near-perfect isotropy in the input embeddings, separating semantic encoding from syntactic prediction. While untying requires dedicating an additional 50M to 150M parameters to the output head, the resulting representational freedom stabilizes residual gradients and dramatically improves conversational quality34. For ultra-minimal 0.3B browser tiers, tying is an unavoidable necessity; for 1.5B models, untying is highly recommended. The balance between network depth (number of layers) and width (hidden dimension) dictates the latency profile on consumer hardware. Deeper networks increase the number of sequential operations, directly inflating the time-to-first-token and slowing the tokens-per-second metric in non-batched inference environments. Wide models process faster by leveraging the parallelization capabilities of hardware ALUs36. For sub-billion models, capping depth at [Figure omitted from source export] or [Figure omitted from source export] while expanding the hidden dimension ensures that matrix multiplications remain large enough to overcome the bandwidth-bound overhead of launching numerous small operations, optimizing for the latency constraints of WASM execution. Multi-Head Attention (MHA) allocates a dedicated KV cache for every query head, leading to exponential memory growth during long-context generation. Grouped-Query Attention mitigates this by sharing a single KV head across multiple query heads, reducing the KV cache footprint by a factor proportional to the group size37. Sliding-window attention further restricts memory growth by constraining the attention horizon to the most recent [Figure omitted from source export] tokens, ensuring the KV cache never exceeds a fixed, predefined boundary regardless of the total sequence length37. State Space Models, specifically Mamba-2 utilizing State Space Duality, eliminate the KV cache entirely in favor of a fixed-size recurrent state39. The memory footprint remains strictly [Figure omitted from source export] regarding sequence length during the decode phase, rendering Mamba architectures fundamentally immune to dynamic memory out-of-bounds errors at extended context lengths17. Hybrid architectures that interleave sliding-window attention with recurrent SSM blocks provide the optimal balance, leveraging attention for precise, short-range factual retrieval and recurrent blocks for long-range, memory-efficient context routing17.

6. Artifact-Size Formulas by Precision

Model weights must be compressed prior to browser transmission to minimize download latency and load-time duplication. The GGUF format standardizes block-wise quantization for local execution, ensuring high-fidelity compression without requiring specialized neural processing units42. Artifact size estimation depends on the precision format, block metadata, and structural alignment. For an arbitrary model, the total artifact size in bytes is mathematically derived as: [Figure omitted from source export] Where [Figure omitted from source export] represents the number of elements in tensor [Figure omitted from source export], [Figure omitted from source export] is the average bits per element accounting for block metadata, and [Figure omitted from source export] covers the global GGUF header, tokenizer byte dictionaries, chat templates, and tensor alignment padding43. In a common block-wise asymmetric quantization scheme like Q4\_K\_M or Q4\_1, weights are partitioned into groups of size [Figure omitted from source export] (typically 32). Each block stores the 4-bit quantized weights alongside a higher-precision scaling factor ([Figure omitted from source export]) and a zero-point ([Figure omitted from source export])21. If the scale and zero-point are stored as 16-bit floats (2 bytes each), the metadata overhead per block is 4 bytes. The effective bits per weight ([Figure omitted from source export]) is calculated as:[Figure omitted from source export] For a 4-bit format with [Figure omitted from source export] and 4 bytes of metadata:[Figure omitted from source export] bits. For a 0.8B parameter model, the weight footprint evaluates to [Figure omitted from source export] MB. The [Figure omitted from source export] overhead typically adds 10 to 20 MB for standard vocabularies and serialization padding.

7. Inference-Memory Formulas by Context

Browser memory management must meticulously account for weights, the dynamically expanding KV cache, WASM linear memory layout, and the Javascript heap overhead. Failure to model peak memory allocation during the prefill phase guarantees browser eviction. The KV cache memory consumption for standard Grouped-Query Attention scales linearly with sequence length and batch size46. The required memory in bytes is formulated as: [Figure omitted from source export] Where [Figure omitted from source export] accounts for the distinct Key and Value matrices, [Figure omitted from source export] is the number of layers, [Figure omitted from source export] is the number of KV heads, [Figure omitted from source export] is the head dimension, [Figure omitted from source export] is the sequence length in tokens, [Figure omitted from source export] is the batch size, and [Figure omitted from source export] is the byte width of the precision format (e.g., 2 for FP16, 1 for FP8)47. Applying this formula to a 0.8B parameter model ([Figure omitted from source export], [Figure omitted from source export], [Figure omitted from source export]) at FP16 precision ([Figure omitted from source export]) for a single user ([Figure omitted from source export]):[Figure omitted from source export] bytes. Table 3 models the KV cache footprint for this configuration across various prompt lengths.

Inference memory modeling must explicitly separate the steady-state decode phase from the volatile prefill phase. During the initial prefill, the model computes the massive [Figure omitted from source export] self-attention matrix for the entire input sequence simultaneously49. For an 8K sequence, this transient dense matrix operation generates a massive memory spike inside the WASM allocator. Furthermore, JavaScript-to-WASM memory duplication occurs during weight initialization and array buffer host-copies51. Therefore, an artifact requiring 500 MB for weights and 200 MB for the KV cache demands a minimum safety ceiling of 1.2GB to 1.5GB of actual browser memory to withstand prefill allocator volatility without triggering browser OOM crash-handlers.

8. Capability Allocation and Tool-Delegation Boundaries

Sub-billion parameter models possess strict structural limitations. A 0.5B model lacks the parameter capacity to function as an omniscient knowledge graph or a deep logical reasoning engine. The integration blueprint must explicitly decouple syntactic reasoning from factual storage, leveraging the model purely as a linguistic router and instruction follower. At the 0.3B scale, the model is restricted to basic syntactic routing, summarization of explicitly provided text, and short-form instruction following. Arithmetic logic, complex entity tracking, and exact historical fact retrieval are entirely absent53. The model must act as an interface that calls deterministic tools—such as passing string equations to a Javascript math runtime—and relies entirely on external retrieval-augmented generation to answer queries. At the 0.8B scale, ambiguity handling and contextual precedence improve significantly. The model can accurately rewrite user queries for semantic search and generate consistently formatted structured JSON objects. Conflict resolution between the system prompt and user input reliably favors the system prompt, preventing basic prompt injection bypasses. At the 1.5B scale, practical conversational continuity emerges. The model can explain simple code snippets, handle multi-turn context referencing, and provide feasible fallback advice when a tool fails or a source conflict occurs40. Claims that a 1.0B parameter model inherently guarantees intelligent long-context behavior across 32K tokens are demonstrably false. The model will suffer from severe attention dilution and "lost in the middle" phenomena. Extensive context windows in sub-billion models should be utilized for strict document summarization or structured data parsing, not complex multi-step reasoning.

9. Training and Initialization Options

Training a small model from a random initialization requires immense datasets exceeding two trillion tokens to achieve optimal Chinchilla-compliant capacity55. Instead, initializing the student from an existing pre-trained teacher accelerates convergence and establishes strong prior representations. Layer dropping, formalized through Boomerang Distillation, offers a computationally efficient initialization vector. Rather than training a randomly initialized small model, a 0.8B student is initialized by structurally pruning a 1.5B teacher. Boomerang distillation replaces student layers with contiguous blocks of teacher layers to maintain representation stability57. While random layer dropping drastically increases perplexity by disrupting the residual flow, dropping contiguous, deeper blocks preserves up to 95% of the original model's performance, as residual stream variations plateau in middle-to-deep layers59. Knowledge distillation must occur at the token level. Sequence-level distillation—training the student on the hard text generated by the teacher—suffers from exposure bias and discards the rich, fine-grained information encoded in the teacher's probability distributions62. Token-level Knowledge Distillation utilizing Kullback-Leibler (KL) divergence matches the student's probability distributions with the teacher's soft labels62. However, standard KL distillation is constrained by the softmax function, which compresses large differences in logit values into nearly identical, near-zero probabilities, blinding the student to the "dark knowledge" contained within the negative logits64. Concrete Score Distillation resolves this by discarding softmax and matching the relative logit differences across all vocabulary pairs. Crucially, CSD handles logit shift invariance, allowing the student to perfectly map the teacher's relative probability structure without being forced to replicate absolute logit magnitudes, dramatically accelerating convergence in autoregressive generation64. For edge deployment, Quantization-Aware Training is strictly recommended over Post-Training Quantization. PTQ applies quantization mathematically after training concludes, degrading the delicate representations of sub-billion models and causing catastrophic accuracy loss67. QAT simulates the 4-bit clipping and rounding logic during the fine-tuning phase, allowing the gradient descent updates to adjust the weights to cluster safely around representable integer values67. QAT guarantees that the model shipped in the WASM artifact maintains parity with its uncompressed validation metrics.

10. Three Complete Student Blueprints

The following blueprints provide exact specifications for sub-billion models tailored for edge browser deployment, derived from the mathematical constraints and distillation frameworks analyzed above.

Blueprint Alpha: Minimal Browser Tier (~0.3B)

• Objective: Immediate load times and ultra-low memory usage for basic command parsing, RAG summarization, and deterministic tool routing.
• Logical Configuration: [Figure omitted from source export], [Figure omitted from source export], [Figure omitted from source export], [Figure omitted from source export] (GQA), Sliding Window Attention ([Figure omitted from source export]).
• Tokenizer: Pruned to 32,000 tokens with byte fallback.
• Embeddings: Tied. The pruned vocabulary reduces the embedding parameters to 32.7M, allocating the remaining budget entirely to the 16 transformer blocks.
• Total Parameters: \~230 Million.
• Memory Formula (2K Context): \~130 MB Artifact Size (Q4\_K\_M) \+ 8 MB (KV Cache) \+ 120 MB (WASM allocator overhead) [Figure omitted from source export] 258 MB Peak Browser Memory.
• Training Method: Initialized via Boomerang Distillation from a 1.0B teacher. Trained using Token-level Concrete Score Distillation over 500B tokens.
• Expected Risks: Severe hallucination if prompted without external retrieval context. Absolute inability to perform zero-shot math or logical deduction.

Blueprint Beta: Balanced General Tier (~0.8B)

• Objective: Robust local conversational agent capable of semantic query rewriting, structured JSON output generation, and source-conflict resolution.
• Logical Configuration: [Figure omitted from source export], [Figure omitted from source export], [Figure omitted from source export], [Figure omitted from source export] (GQA), Full Context Attention.
• Tokenizer: 64,000 mixed-domain vocabulary for balanced code and language representation.
• Embeddings: Untied. The 64K vocabulary requires \~98M parameters for the input matrix and \~98M for the output matrix, freeing the internal representations from unembedding bias and maximizing isotropy35.
• Total Parameters: \~780 Million.
• Memory Formula (8K Context): \~450 MB Artifact Size (Q4\_K\_M) \+ 144 MB (KV Cache) \+ 300 MB (WASM allocator overhead) [Figure omitted from source export] 894 MB Peak Browser Memory.
• Training Method: Quantization-Aware Training with sequence-level preference optimization (DPO) to enforce rigid structured output compliance and tone matching69.
• Expected Risks: Slower time-to-first-token on low-end mobile CPUs due to the larger artifact size and untied embedding matrix calculations.

Blueprint Gamma: Higher-Quality Local Tier (~1.5B)

• Objective: Advanced coding explanation, multi-turn conversational continuity, and feasible fallback advice handling.
• Logical Configuration: Hybrid Mamba-2 (State Space Duality) interleaved with Attention17. [Figure omitted from source export] (28 Mamba-2 layers, 4 Attention layers), [Figure omitted from source export], [Figure omitted from source export].
• Tokenizer: 100,000 universal vocabulary.
• Embeddings: Tied (to save 204M parameters, routing them instead into the recurrent state pathways).
• Total Parameters: \~1.4 Billion.
• Memory Formula (32K Context): \~790 MB Artifact Size (Q4\_0) \+ 32 MB (Recurrent State) \+ 64 MB (Sparse KV Cache for Attention) \+ 400 MB (WASM allocator overhead) [Figure omitted from source export] 1.28 GB Peak Browser Memory.
• Training Method: Token-level KL distillation from an 8B teacher using Cross-Tokenizer X-Token alignment27.
• Expected Risks: Hybrid architectures require custom WebGL/WebGPU operator implementations within Javascript frameworks, which may lack optimal hardware support on legacy targets, risking a fallback to slow scalar CPU execution52.

11. Pilot, Rejection, and Promotion Criteria

Before committing thousands of GPU hours to full pre-training or distillation, candidate blueprints must pass a rigorous, computationally cheap source-screen pilot experiment. This ensures structural viability and establishes strict helpfulness regression budgets. The pilot protocol initiates by constructing the student architecture using contiguous layer patching from the teacher. The student undergoes a short calibration run, training for exactly 500 million tokens on a high-quality corpus (e.g., FineWeb-Edu) utilizing a constant learning rate with a sharp 20% linear cooldown to immediately reveal the model's converged loss floor56. Following the calibration run, the blueprint is subjected to three explicit rejection criteria:

1. Validation Perplexity Threshold: If the perplexity of the student on a held-out validation set is more than 35% higher than the teacher's perplexity evaluated at the same token budget, the blueprint is rejected due to insurmountable representation collapse.
2. Untouched Outputs and Paraphrase Consistency: The model must process unseen paraphrases of training prompts. The exact-match structured output compliance rate must not deviate by more than 10% across semantic paraphrases. If the model shatters on novel syntactic structures, the distillation mapping failed.
3. Ordinary-Helpfulness Regression Budget: The student must undergo an automated evaluation against a known dense baseline of identical parameter count. If the blueprint demonstrates a \>15% degradation in ordinary helpfulness (e.g., failing simple greeting continuity, refusal handling, or basic instruction formatting), the architectural configuration is rejected.

A blueprint is promoted to full-scale training only if it passes all three thresholds, proving its structural integrity prior to massive compute expenditure.

12. Meaningful Portfolio Implications

Operating a unified portfolio of varying student sizes (0.3B, 0.8B, 1.5B) forces a strategic decision regarding tokenizer diversity. Maintaining a single universal tokenizer (e.g., a 128K vocabulary) across the entire portfolio drastically simplifies deployment mechanics. It eliminates the footprint of fetching different multi-megabyte tokenizer.json dictionaries and streamlines WASM runtime compilation and caching protocols across browser instances52. However, as demonstrated in the tokenizer economics analysis, an oversized vocabulary on a 0.3B model mathematically paralyzes its reasoning budget. The structural efficiency gained by mapping the 0.3B model to a bespoke 32K pruned vocabulary far outweighs the minor runtime cost of loading an additional 2MB JSON dictionary into the browser. The friction previously associated with training diverse tokenizers is entirely neutralized by modern cross-tokenizer distillation protocols like Byte-Level Distillation, which allow an 8B teacher with a 128K vocabulary to seamlessly supervise a 0.3B student with a 32K vocabulary via a shared byte-level probability space31. Therefore, tokenizer diversity is highly recommended; each tier must adopt the vocabulary size that mathematically optimizes its specific parameter budget.

13. Unknowns Requiring Local Training or Execution

Several systemic variables remain theoretically sound but require rigorous local execution testing prior to production deployment. The Javascript garbage collector operates unpredictably; host memory spikes occurring during array buffer transfers from the JS heap into the contiguous WASM linear memory can trigger silent browser OOM tab-crashes. These transient prefill spikes are notoriously difficult to profile offline and vary wildly between Chrome, Safari, and Firefox engines. Additionally, the exact performance of INT4 dequantization kernels within the browser remains a volatile unknown. WebGPU execution relies on hardware-specific shading language support. Testing is required to determine whether the fallback to scalar CPU dequantization destroys the latency advantages of quantization, particularly for the Mamba-2 selective scan operators and associative scan algorithms, which rely heavily on specialized matrix decompositions that may not map efficiently to WebGL fallbacks39.

14. Annotated primary-source bibliography with immutable links and dates

The parameters, mathematical formulas, and architectural paradigms referenced in this report derive from the explicit analysis of the following immutable model artifacts, technical reports, and source repositories:

Qwen2.5 Family Technical Reports and Repositories (Alibaba, Dec 2024\). Analyzed for dense GQA configurations, massive vocabulary scaling (151K), and the implementation of QKV attention biases. Distributed under Apache 2.0. Verified via config.json revisions eaa56b5 and 24e7668.

SmolLM2 Family (HuggingFaceTB, Feb 2025\). "SmolLM2: When Smol Goes Big". Analyzed for sub-billion depth vs. width tradeoffs, parameter allocations, and dataset optimization in the 135M and 360M parameter ranges. Distributed under Apache 2.0. Verified via config.json revisions 28e66ca and 057be69b.

Llama 3.2 Series (Meta, Sep 2024\). Analyzed for exact parameter computations (proving the 1B model is 1.23B active parameters), GQA caching formulas, and tied embedding dynamics. Distributed under the custom Meta Llama 3.2 Community License.

Mamba-2 and State Space Models (Dao & Gu, May 2024\). "Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality". Analyzed for recurrent state derivations, SSD block matrix decompositions, and [Figure omitted from source export] memory footprints during inference. Distributed under Apache 2.0.

Cross-Tokenizer Distillation and Byte Fallbacks (Various, 2024-2026). "Cross-Tokenizer LLM Distillation through a Byte-Level Interface" and "X-Token". Analyzed for solving vocabulary mismatches via sparse projection matrices and byte-level probability matching to enable disparate tokenizer usage across model portfolios.

Weight Tying in Language Models (2024-2026). Analyzed for unembedding bias, vector anisotropy, and the qualitative differences between tied and untied embedding matrices in low-parameter regimes.

KV Cache Optimization Guides (2025-2026). Analyzed for precise memory footprint formulas regarding GQA and Sliding Window Attention across 512 to 32K context lengths.

GGUF and Quantization Mechanisms (2024-2026). Analyzed for artifact size derivations, block-wise uniform quantization (Q4\_0, Q4\_K\_M), scale/zero-point metadata overhead, and effective bit-per-weight calculations.

Quantization-Aware Training vs. Post-Training Quantization (2024-2025). Analyzed for training implications on sub-billion models where PTQ destroys representation stability, necessitating QAT for edge deployment.

Boomerang Distillation and Layer Dropping (2025-2026). Analyzed for efficient model initialization strategies using contiguous teacher layer patching and redundancy reduction.

Transformers.js and WASM Execution (2025-2026). Analyzed for browser memory limits (4GB), allocator volatility, cross-origin caching, and WebGL/WebGPU fallback mechanics.

Concrete Score Distillation and Token-level KL (2025-2026). "Distillation of Large Language Models via Concrete Score Matching". Analyzed for resolving softmax smoothing issues and logit shift invariance in direct logit distillation.

• 7
• 3
• 9
• 13
• 23
• 33
• 37
• 21
• 67
• 57
• 1
• 62

Works cited

1. sudarshantanwer/Brainbox-AI: Offline AI chat application ... \- GitHub, https://github.com/sudarshantanwer/Brainbox-AI
2. Support for transformers · Issue \#375 · webmachinelearning/webnn \- GitHub, https://github.com/webmachinelearning/webnn/issues/375
3. config.json · mtheiken/SmolLM2-FT-MyDataset at main \- Hugging Face, https://huggingface.co/mtheiken/SmolLM2-FT-MyDataset/blame/main/config.json
4. config.json · HuggingFaceTB/SmolLM2-135M at main, https://huggingface.co/HuggingFaceTB/SmolLM2-135M/blob/main/config.json
5. config.json · jjzha/SmolLM2-360M-Instruct-fs1-2708 at 057be69b0d3d264ddd5d27a3d0e30152e8275878 \- Hugging Face, https://huggingface.co/jjzha/SmolLM2-360M-Instruct-fs1-2708/blame/057be69b0d3d264ddd5d27a3d0e30152e8275878/config.json
6. step\_000000000000/config.json · aklein4/iTTT-Cluster\_horizons-diff-muon at 4a88b30994a852480a2e99d4872b75ae7c02e08e \- Hugging Face, https://huggingface.co/aklein4/iTTT-Cluster\_horizons-diff-muon/blob/4a88b30994a852480a2e99d4872b75ae7c02e08e/step\_000000000000/config.json
7. config.json · Qwen/Qwen2.5-0.5B-Instruct at main \- Hugging Face, https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct/blame/main/config.json
8. config.json · zeycan/Qwen-Qwen2.5-0.5B-Instruct at 0cfd92b9c70b37b6b87c566331e2c8e61aef9a88 \- Hugging Face, https://huggingface.co/zeycan/Qwen-Qwen2.5-0.5B-Instruct/blob/0cfd92b9c70b37b6b87c566331e2c8e61aef9a88/config.json
9. config.json · onnx-community/Llama-3.2-1B-Instruct-ONNX at c5b77c42b9a13105f54d55be507e1a65b5b8149b \- Hugging Face, https://huggingface.co/onnx-community/Llama-3.2-1B-Instruct-ONNX/blame/c5b77c42b9a13105f54d55be507e1a65b5b8149b/config.json
10. config.json · evdev3/Llama-3.2-1B-Instruct-4bit at main \- Hugging Face, https://huggingface.co/evdev3/Llama-3.2-1B-Instruct-4bit/blob/main/config.json
11. config.json · onnx-community/Qwen2.5-1.5B at 5db89022df9032256e08756fa1c6b731b8b5e064 \- Hugging Face, https://huggingface.co/onnx-community/Qwen2.5-1.5B/blame/5db89022df9032256e08756fa1c6b731b8b5e064/config.json
12. config.json · zlyngkhoi/qwen-qwen2.5-1.5b-instruct-truthfulqa-sensitivity-hqq at main, https://huggingface.co/zlyngkhoi/qwen-qwen2.5-1.5b-instruct-truthfulqa-sensitivity-hqq/blob/main/config.json
13. state-spaces/mamba-370m at main \- Hugging Face, https://huggingface.co/state-spaces/mamba-370m/tree/main
14. config.json · state-spaces/mamba-370m-hf at main \- Hugging Face, https://huggingface.co/state-spaces/mamba-370m-hf/blob/main/config.json
15. Upload folder using huggingface\_hub · unsloth/NVIDIA-Nemotron-Nano-9B-v2 at dc0c6fb, https://huggingface.co/unsloth/NVIDIA-Nemotron-Nano-9B-v2/commit/dc0c6fb9a0be982a6959105d5cc9273ccd2862f6
16. vllm.model\_executor.models.qwen3, https://docs.vllm.ai/en/v0.9.1/api/vllm/model\_executor/models/qwen3.html
17. Chapter 32: Post-Transformer Architectures \- Mamba and Hybrid Models \- Wayland Zhang, https://waylandz.com/llm-transformer-book-en/chapter-32-post-transformer-architectures/
18. Mamba Linear-Time Sequence Modeling With Selective State Spaces \- Scribd, https://www.scribd.com/document/1002949306/Mamba-Linear-Time-Sequence-Modeling-With-Selective-State-Spaces
19. How to compute total FLOPs and Parameters for Mamba layer? · Issue \#817 \- GitHub, https://github.com/state-spaces/mamba/issues/817
20. How to compute MACs or FLOPs of mamba · Issue \#110 \- GitHub, https://github.com/state-spaces/mamba/issues/110
21. The Complete Guide to LLM Quantization with vLLM: Benchmarks & Best Practices, https://jarvislabs.ai/blog/vllm-quantization-complete-guide-benchmarks
22. unknown\_url
23. Tokens with Meaning: A Hybrid Tokenization Approach for Turkish \- arXiv, https://arxiv.org/html/2508.14292v3
24. Let's Build the GPT Tokenizer: A Complete Guide to Tokenization in LLMs \- Fast.ai, https://www.fast.ai/posts/2025-10-16-karpathy-tokenizers
25. A survey of LLMs architecture \- Kaggle, https://www.kaggle.com/code/misterfour/a-survey-of-llms-architecture
26. LightOnOCR-1B: Making Knowledge Machine-Readable \- LightOn, https://lighton.ai/fr-blog-posts/lightonocr-1b-making-knowledge-machine-readable
27. X-Token: Projection-Guided Cross-Tokenizer Knowledge Distillation \- arXiv, https://arxiv.org/html/2605.21699v1
28. Breaking the Tokenizer Barrier: On-Policy Distillation across Model Families \- arXiv, https://arxiv.org/html/2606.09456v1
29. Overcoming Vocabulary Mismatch: Vocabulary-agnostic Teacher Guided Language Modeling \- arXiv, https://arxiv.org/html/2503.19123v1
30. (PDF) X-Token: Projection-Guided Cross-Tokenizer Knowledge Distillation \- ResearchGate, https://www.researchgate.net/publication/405132677\_X-Token\_Projection-Guided\_Cross-Tokenizer\_Knowledge\_Distillation
31. Cross-Tokenizer LLM Distillation through a Byte-Level Interface \- arXiv, https://arxiv.org/html/2604.07466v1
32. Cross-Tokenizer LLM Distillation through a Byte-Level Interface \- ACL Anthology, https://aclanthology.org/2026.customnlp4u-1.9.pdf
33. Weight Tying Biases Token Embeddings Towards the Output Space \- arXiv, https://arxiv.org/html/2603.26663v1
34. The pragmatic tradeoff of tied embeddings | Silen \- Blog, https://blog.silennai.com/tied-embeddings
35. Weight tying in language models: when and why LLMs share embeddings \- Medium, https://medium.com/@vishal09vns/weight-tying-in-language-models-when-and-why-llms-share-embeddings-7f5f5376c625
36. INITIALIZATION AND REGULARIZATION OF FACTORIZED NEURAL LAYERS \- Microsoft, https://www.microsoft.com/en-us/research/wp-content/uploads/2021/03/main\_Initialization-Regularization-Factorized-Neural-Layers\_final.pdf
37. Techniques for KV Cache Optimization in Large Language Models \- omrimallis, https://www.omrimallis.com/posts/techniques-for-kv-cache-optimization/
38. Sliding Window Attention (SWA) \- rasbt/LLMs-from-scratch \- GitHub, https://github.com/rasbt/LLMs-from-scratch/blob/main/ch04/06\_swa/README.md
39. Mamba-2: Algorithms and Systems | Princeton Language and Intelligence, https://pli.princeton.edu/blog/2024/mamba-2-algorithms-and-systems
40. Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality \- OpenReview, https://openreview.net/pdf/54bf495d93336f1f195f264c1b6c2805169b3492.pdf
41. Characterizing State Space Model (SSM) and SSM-Transformer Hybrid Language Model Performance with Long Context Length \- arXiv, https://arxiv.org/html/2507.12442v2
42. Which Quantization Should I Use? A Unified Evaluation of llama.cpp Quantization on Llama-3.1-8B-Instruct \- arXiv, https://arxiv.org/html/2601.14277
43. LLM Quantization: All You Need to Know\! \- Cloudthrill, https://cloudthrill.ca/llm-quantization-all-you-need-to-know
44. Model Quantization: Concepts, Methods, and Why It Matters | NVIDIA Technical Blog, https://developer.nvidia.com/blog/model-quantization-concepts-methods-and-why-it-matters/
45. Aman's AI Journal • Primers • Quantization, https://aman.ai/primers/ai/quantization/
46. KV Cache: The Secret to Fast LLM Inference \- Abhik Sarkar, https://www.abhik.ai/concepts/transformers/kv-cache
47. KV Cache Optimization: Serve 10x More Users on the Same GPU (2026) | Spheron Blog, https://www.spheron.network/blog/kv-cache-optimization-guide/
48. KV Cache / Token (bf16) | Sebastian Raschka, PhD, https://sebastianraschka.com/llm-architecture-gallery/kv-cache-calculations/
49. How LLM Inference Works \- Arpit Bhayani, https://arpitbhayani.me/blogs/how-llm-inference-works/
50. Mamba2: The Hardware-Algorithm Co-Design That Unified Attention and State Space Models | by Daniel Stallworth | Medium, https://medium.com/@danieljsmit/mamba2-the-hardware-algorithm-co-design-that-unified-attention-and-state-space-models-77856d2ac4f4
51. GSoC 2026: Opportunities for the AI projects \- Joplin Forum, https://discourse.joplinapp.org/t/gsoc-2026-opportunities-for-the-ai-projects/49228
52. Transformers.js vs ONNX Runtime Web: Browser ML 2026 \- PkgPulse, https://www.pkgpulse.com/guides/transformersjs-vs-onnx-runtime-web-2026
53. SmolLM2: When Smol Goes Big — Data-Centric Training of a Small Language Model \- arXiv, https://arxiv.org/html/2502.02737v1
54. \[2412.15115\] Qwen2.5 Technical Report \- arXiv, https://arxiv.org/abs/2412.15115
55. MZN — LLM Company Anatomy · Public Reference Atlas \+ Provisional Position Map, https://mzncompany.com/llmframework/mzn-llm-anatomy.html
56. Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations \- arXiv, https://arxiv.org/html/2405.18392v3
57. Boomerang Distillation Enables Zero-Shot Model Size Interpolation \- arXiv, https://arxiv.org/html/2510.05064v1
58. Understanding Layer Patching in Model Size Interpolation \- arXiv, https://arxiv.org/html/2607.08170v1
59. Pluggable Pruning with Contiguous Layer Distillation for Diffusion Transformers \- arXiv, https://arxiv.org/html/2511.16156v2
60. The Unreasonable Ineffectiveness of the Deeper Layers \- arXiv, https://arxiv.org/html/2403.17887v2
61. Iterative Layer-wise Distillation for Efficient Compression of Large Language Models \- arXiv, https://arxiv.org/html/2511.05085v1
62. Token-Based Knowledge Distillation \- Emergent Mind, https://www.emergentmind.com/topics/token-based-knowledge-distillation
63. Delta Knowledge Distillation for Large Language Models \- arXiv, https://arxiv.org/html/2509.14526v1
64. Distillation of Large Language Models via Concrete Score Matching \- arXiv, https://arxiv.org/pdf/2509.25837
65. Motivation for logit-level distillation and limitations of prior work.... \- ResearchGate, https://www.researchgate.net/figure/Motivation-for-logit-level-distillation-and-limitations-of-prior-work-a-Statistics-of\_fig1\_396049488
66. Distillation of Large Language Models via Concrete Score Matching \- arXiv, https://arxiv.org/html/2509.25837v1
67. What is Quantization Aware Training? QAT vs. PTQ | SabrePC Blog, https://www.sabrepc.com/blog/deep-learning-and-ai/what-is-quantization-aware-training-qat-vs-ptq
68. Quantization in Machine Learning: QAT vs. PTQ | by Ali Borji | Medium, https://medium.com/@aliborji/quantization-in-machine-learning-qat-vs-ptq-064bfd4f9b58
69. How Quantization Aware Training Enables Low-Precision Accuracy Recovery, https://developer.nvidia.com/blog/how-quantization-aware-training-enables-low-precision-accuracy-recovery/
70. On LLM Knowledge Distillation \- A Comparison between Forward KL and Reverse KL, https://d2jud02ci9yv69.cloudfront.net/2025-04-28-llm-knowledge-distil-157/blog/llm-knowledge-distil/
71. Chrome Brings Cross-Origin Model Caching to Transformers.js | News, https://getaibook.com/news/chrome-brings-cross-origin-model-caching-to-transformersjs/
72. Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations \- arXiv, https://arxiv.org/html/2405.18392v2
73. Qwen2.5 Technical Report \- BibBase, https://bibbase.org/network/publication/qwen-yang-yang-zhang-hui-zheng-yu-li-etal-qwen25technicalreport-2025
74. SmolLM2: When Smol Goes Big \-- Data-Centric Training of a Small Language Model \- arXiv, https://arxiv.org/abs/2502.02737
75. meta / llama-3.2-11b-vision-instruct \- NVIDIA API Documentation, https://docs.api.nvidia.com/nim/reference/meta-llama-3\_2-11b-vision-instruct
76. meta-llama/Llama-3.2-1B \- Hugging Face, https://huggingface.co/meta-llama/Llama-3.2-1B
77. \[2405.21060\] Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality \- arXiv, https://arxiv.org/abs/2405.21060