Distillation and Deployment Strategies for Browser-Native Tiny Language Models (.slm)

Full report

On this page

The Paradigm Shift Toward Edge-Native Small Language Models

The operational landscape of large language models (LLMs) has historically been dominated by cloud-based infrastructure, an approach that inherently trades user privacy, offline availability, and latency for computational power. However, the maturation of WebAssembly (WASM) and WebGPU technologies has catalyzed a paradigm shift, enabling the direct execution of neural networks within the highly constrained environments of consumer web browsers1. This architectural migration necessitates a specialized class of highly compressed, ultra-efficient models. This report details a comprehensive, end-to-end knowledge distillation and deployment strategy for producing TinyRustLM-compatible .slm (Small Language Model) artifacts. The target parameter regimes span 50M, 100M, 300M, and 500M parameters. The primary objective is to distill the expansive knowledge, nuanced instruction-following capabilities, and logical reasoning patterns of frontier teacher models (e.g., the 32B to 72B parameter class) into sub-billion parameter students. These students must demonstrate robust proficiency in core web-native UI tasks: general conversational chat, text summarization, localized rewriting, contextual code explanation, strict JSON generation, and local-agent instruction following. Achieving this level of proficiency in heavily constrained models requires a departure from traditional scaling laws. It necessitates sub-billion architectural optimization, structured synthetic dataset generation with rigorous decontamination protocols, sophisticated cross-tokenizer and symmetric distillation methodologies, preference alignment tailored for low-capacity targets, and advanced mixed-precision quantization algorithms that preserve semantic coherence at the edge.

Architecting Sub-Billion Student Models

Designing functional models in the 50M to 500M parameter range requires fundamental architectural realignments. At these scales, the parameter budget must be allocated with extreme precision to maximize representational depth while adhering to strict memory footprints and time-to-first-token (TTFT) latency limits3.

The Vocabulary-Embedding Bottleneck

A critical architectural bottleneck at the 50M and 100M scale is the allocation of parameters to the vocabulary embedding matrix. State-of-the-art teacher models, such as those in the Qwen2.5 series, utilize expansive vocabularies exceeding 151,000 tokens to ensure superior multilingual and coding compression rates6. However, if a 50M parameter student model with a modest hidden dimension of 512 attempts to utilize a 151,000-token vocabulary, the input embedding layer alone consumes approximately 77.7 million parameters, immediately exceeding the entire model budget before a single attention block is instantiated8. To construct viable 50M and 100M .slm models, the vocabulary size must be restricted to approximately 32,000 tokens. While this slightly reduces tokenization efficiency for rare linguistic constructs, it prevents the embedding matrix from cannibalizing the parameter budget necessary for the model's logical depth. Furthermore, weight tying—the practice of sharing the exact same parameter weights between the input embedding layer and the final pre-softmax output projection head—is absolutely mandatory8. This reclamation of parameters enables the construction of deeper transformer architectures, which empirical evidence demonstrates are vastly superior to shallow-and-wide configurations at sub-billion scales4.

Recommended .slm Target Architectures

The architectures optimized for the .slm family prioritize a "deep and thin" structural configuration. These models integrate Grouped-Query Attention (GQA) across all scales to minimize the Key-Value (KV) cache memory footprint during autoregressive generation, a critical optimization for browser environments where RAM allocations per tab are strictly limited4.

These target configurations utilize Rotary Position Embeddings (RoPE) for context expansion, SwiGLU activation functions in the feed-forward networks (FFN) for enhanced non-linear representation, and RMSNorm for stable training dynamics7. For instance, the 50M model, explicitly targeted at 12 layers with a 512 hidden dimension, successfully allocates over 67% of its parameters directly to the active transformer blocks, enabling sufficiently complex internal feature representations despite its extreme smallness8.

Dataset Design and Synthetic Generation Protocols

The intrinsic capabilities of an .slm model are strictly bounded by the structural quality and cleanliness of its distillation dataset. Because the resulting .slm models are intended for unencumbered commercial and public deployment, the training corpus must be constructed entirely from permissive, open-access datasets (e.g., those bearing MIT, Apache 2.0, or CC-BY licenses) combined with synthetically generated data derived exclusively from open-weights teachers10.

Structured Task Curriculum for Local Browser Assistants

The training curriculum for .slm models must eschew generalized, sprawling world knowledge in favor of hyper-specialized local assistant behaviors. The curriculum is partitioned into highly focused task domains:

1. JSON Generation and Local-Agent Following (JsonTuning): Standard text-to-text instruction tuning frequently results in brittle, ill-formatted outputs when applied to small models. The JsonTuning paradigm mitigates this by mapping all task inputs, instructions, and expected outputs into explicit JSON schemas during the training phase12. By structurally coercing the data, the model learns the underlying syntax of JSON as a primary modality, significantly improving its robustness when generating structured data required for browser UI rendering, DOM manipulation routing, or local API tool calling12.
2. Code Explanation and UI Rewriting: Models operating in the browser must frequently interact with web components and user text. Datasets modeled after OpenCodeInstruct are utilized to provide diverse programming questions and step-by-step reasoning, filtered strictly for front-end and web-assembly languages (JavaScript, HTML, CSS, Rust)14. Synthetic prompts are generated commanding the teacher model to provide highly concise, low-latency explanations suitable for narrow side-panel UI displays, stripping away verbose conversational filler.
3. Summarization and General Chat Behavior: General conversational capabilities are distilled using the Magpie methodology or the Cosmopedia dataset construction approach, which synthetically generates billions of high-quality tokens from a mixture of open-weight LLMs10. The focus remains on instructional compliance and summarization of provided context (e.g., reading a loaded webpage) rather than relying on the model's parameterized memory for factual trivia.

Synthetic Data Generation and Contamination Prevention

The reliance on synthetic data introduces a catastrophic risk: evaluation contamination. If the LLM teacher generates synthetic training data that inadvertently mirrors standard evaluation benchmarks (e.g., MMLU, GSM8K, HumanEval), the student model will appear artificially capable through sheer memorization of the test set rather than generalized learning16. To prevent model collapse and guarantee generalization, the synthetic data pipeline must enforce rigorous decontamination and quality gating protocols:

• MinHash Deduplication and Test-Set Decontamination: All synthetically generated responses undergo exhaustive 13-gram MinHash overlap detection against the targeted evaluation test splits16. Any generated training example exhibiting a Jaccard similarity coefficient greater than 0.1 with an evaluation set is immediately purged from the training corpus16.
• Model-Based Quality Filtering (ResoFilter): Simple heuristic filters (e.g., word count thresholds, n-gram repetition penalties) remove degenerate generations17. However, advanced gradient-based filtering techniques like ResoFilter are required for high-density distillation. ResoFilter evaluates the induced change in a reference model's parameter weights during a simulated forward-backward pass. Data points that cause excessive, destabilizing weight shifts are flagged as noisy and discarded, isolating only the synthetic data that contributes meaningful, convergent semantic improvements18.
• The Additive Accumulation Rule: To circumvent model collapse—a mathematical phenomenon where models trained exclusively on synthetic data lose generalized capabilities and suffer from narrowing variance—the training corpus must adhere to an accumulation strategy rather than a replacement strategy. The final dataset retains at least 10% high-quality, human-curated real data (e.g., filtered permissive web text) alongside the synthetic generations to act as an anchor for linguistic diversity15.

Response-Style Distillation vs. Next-Token Distillation

Transferring knowledge from a 32B parameter teacher to a 50M parameter student requires selecting the appropriate distillation modality. The core dichotomy lies between response-style (sequence-level) distillation and next-token (token-level) distillation.

Sequence-Level (Response-Style) Distillation

Sequence-level distillation treats the teacher model as an advanced data annotator. The teacher generates complete sequences (responses) to a given set of prompts, and the student is trained on these generated sequences using standard Supervised Fine-Tuning (SFT) cross-entropy loss19. While computationally inexpensive because it only requires forward passes from the teacher to generate the dataset once, sequence-level distillation suffers from exposure bias and fails to transfer the nuanced "dark knowledge" embedded in the teacher's alternate probability distributions19. It is highly effective for teaching the student what style to adopt (e.g., JSON formatting, concise chat), but it struggles to impart deep reasoning pathways.

Token-Level (Next-Token) Distillation

Token-level distillation aligns the intermediate outputs of the student directly with the teacher at every single generation step. Instead of optimizing against a hard, one-hot encoded label, the student minimizes the Kullback-Leibler (KL) divergence between its output probability distribution ([Figure omitted from source export]) and the teacher's full probability distribution ([Figure omitted from source export]) across the entire vocabulary19. The mathematical formulation relies on either Forward KL or Reverse KL divergence. Forward KL ($D\_{KL}(p | | q\_\\theta)$) forces the student to cover all modes of the teacher's distribution, often leading to overestimation of low-probability tokens. Reverse KL ([Figure omitted from source export]) encourages the student to seek the primary mode, preventing the generation of tokens the teacher deems unlikely, but potentially reducing generation diversity19. To balance these effects, adaptive methodologies such as Speculative Knowledge Distillation (SpecKD) are deployed. SpecKD introduces a token-level gating mechanism that verifies the student's token proposals against the teacher's distribution. Distillation loss is selectively applied only to "accepted" tokens that align with the teacher's high-confidence predictions, masking out rejected tokens and creating an implicit, stable curriculum that prevents the student from indiscriminately mimicking noise22. For the .slm family, a hybrid approach is adopted: sequence-level distillation establishes the initial instructional formatting, followed by intensive token-level distillation to impart reasoning capabilities.

Deep Knowledge Transfer: Logit and Hidden-State Distillation

While token-level distillation is powerful, it faces a severe structural roadblock when the teacher and student utilize different vocabularies, and it fails to leverage the rich, intermediate layers of the neural network.

Cross-Tokenizer Logit Distillation (ULD and ALM)

Standard token-level logit distillation necessitates that both models share an identical tokenizer, as the KL divergence is computed index-by-index across the vocabulary arrays23. Because the .slm models are engineered with a constrained 32,000-token vocabulary while the teacher models utilize 151,000-token vocabularies, standard logit distillation mathematically fails7. This discrepancy is resolved through the application of Universal Logit Distillation (ULD) and Approximate Likelihood Matching (ALM).

• Universal Logit Distillation (ULD): ULD circumvents the tokenizer mismatch by framing the logit alignment as an Optimal Transport problem. Rather than matching indices, ULD computes the Wasserstein distance between the unaligned probability distributions23. It generates a closed-form mapping matrix that dynamically shifts the semantic mass of the teacher's highly granular tokens to the most appropriate tokens in the student's compressed vocabulary25.
• Approximate Likelihood Matching (ALM): ALM addresses cross-tokenizer distillation by operating on variable-length text chunks rather than individual tokens. It identifies aligned chunks of text generated by different tokenizers and approximates the chunk likelihood difference using a binarized f-divergence27. This creates a probabilistic framework that enables sequence likelihood evaluation across arbitrary vocabulary disparities29.

By integrating ULD and ALM, a 32B parameter teacher with a vast multilingual vocabulary can directly supervise the continuous logit outputs of a 50M parameter student, drastically elevating the student's perplexity baseline.

Hidden-State and Relation Distillation (DistillLens and MiniLM)

Treating the teacher's internal reasoning architecture as a black box limits the efficacy of distillation. To produce competent sub-billion models, the structural thought processes within the intermediate transformer layers must be explicitly transferred30. 1\. Symmetric Hidden-State Matching (DistillLens): Traditional hidden-state distillation relies on Mean Squared Error (MSE) projections to force the student's hidden states to match the teacher's31. This approach is fundamentally flawed because it ignores the uncertainty profiles and distributional nature of the representations. The DistillLens framework solves this via the "Logit Lens" methodology. It mathematically projects the intermediate hidden states of both the teacher ([Figure omitted from source export]) and the student ([Figure omitted from source export]) into the shared vocabulary space using their respective unembedding matrices30. Once these hidden states are projected into probability distributions, a symmetric divergence objective—specifically Jensen-Shannon Divergence (JSD)—is applied: $$ \\mathcal{L}{JSD} \= \\frac{1}{2} D{KL}(P | | M) \+ \\frac{1}{2} D\_{KL}(Q | | M) $$ where [Figure omitted from source export]. This symmetric constraint imposes a dual-sided penalty. It simultaneously prevents the student from exhibiting underconfident diffusion and overconfident hallucination, meticulously aligning the evolving thought processes layer-by-layer and preserving the high-entropy information conduits essential for deductive reasoning30. 2\. Multi-Head Self-Attention Relation Distillation (MiniLM): Beyond hidden state magnitudes, the relational interactions between tokens within the self-attention mechanism house critical syntactic and semantic knowledge. Utilizing the MiniLMv2 methodology, the student is forced to mimic the multi-head self-attention relations of the teacher. This is accomplished by computing the scaled dot-products of Query-Query (Q-Q), Key-Key (K-K), and Value-Value (V-V) interaction matrices within the attention modules, and transferring these relational structures using KL divergence32. Because this mechanism transfers mathematical relations rather than absolute vector dimensions, it is entirely agnostic to the number of attention heads, allowing a 32-head teacher to seamlessly impart attention patterns into an 8-head 50M student33.

Preference Distillation for Local Alignment

Standard alignment methodologies, such as Direct Preference Optimization (DPO), update the model by maximizing the probabilistic margin between a chosen response and a rejected response relative to a frozen reference model36. While highly effective for massive frontier models, DPO's binary win-or-loss supervision is severely inadequate for 50M-500M .slm models38. Sub-billion parameter models lack the internal capacity to deduce complex, continuous reward landscapes from simple discrete binary labels, leading to underfitting, mode collapse, and alignment failure38.

Teacher Value-Based Knowledge Distillation (TVKD)

To successfully align .slm models for safe and helpful browser interactions, the deployment strategy utilizes Teacher Value-based Knowledge Distillation (TVKD). TVKD circumvents the limitations of binary feedback by distilling the teacher model's internal reward modeling directly into the student, transitioning from behavior cloning to true preference distillation38. TVKD operates by extracting the state-value function, [Figure omitted from source export], from an already DPO-aligned teacher model38. This value function serves as a dense, token-by-token estimation of expected future rewards, acting as a proxy for the teacher's internal judgment38. TVKD injects this value function into the student's optimization objective as a continuous auxiliary soft reward38. Crucially, this injection must not corrupt the primary alignment objective. It is formulated to strictly satisfy Potential-Based Reward Shaping (PBRS). By adding a potential function [Figure omitted from source export], TVKD provides mathematical guarantees that the global reward structure and the optimal policy of the original DPO objective are preserved perfectly38. This mechanism provides the low-capacity student model with nuanced, magnitude-aware supervision—teaching it precisely how much better one response is than another—without necessitating expensive online rollouts or the deployment of external reward models38.

Post-Training Compression: Mixed-Precision q4/q3 Quantization

To execute entirely within the localized VRAM and RAM constraints of consumer hardware—often necessitating a total footprint of less than 500MB to fit comfortably within a single browser tab's WebAssembly memory limit—the distilled FP32/BF16 models must undergo extreme quantization2. However, aggressive uniform quantization (e.g., blanket conversion to 4-bit INT4 or 3-bit INT3) induces catastrophic quality degradation in sub-billion parameter models. Unlike 70B parameter models, which possess massive representational redundancy that absorbs rounding and clipping errors, small models are exquisitely sensitive to precision loss42. Uniformly applying a 3-bit or 4-bit constraint to a 100M parameter model effectively destroys its capacity for multi-step reasoning and mathematical logic44.

The ResQ Mixed-Precision Paradigm (q4/q3 Allocation)

To achieve the requisite compression ratios without sacrificing semantic coherence, the strategy mandates a mixed-precision quantization protocol. Precision is not uniformly truncated; it is dynamically allocated layer-by-layer and tensor-by-tensor based on architectural sensitivity. This protocol utilizes a Principal Component Analysis (PCA) based outlier suppression methodology inspired by the ResQ algorithm47. In Transformer architectures, a distinct, highly localized subset of activation channels exhibits massive numerical outliers. Uniformly quantizing these specific channels to low-bit integers causes irreparable activation overflows and accuracy collapse47.

1. Low-Rank Subspace Preservation: The pipeline performs PCA to identify a low-rank subspace (typically consisting of 1/8th of the hidden dimension) that contains the highest activation variances. The parameter coefficients within this critical subspace are shielded from aggressive compression and retained in higher precision (INT8)47.
2. q4/q3 Dynamic Base Quantization: The remaining 7/8ths of the hidden dimension, which exhibit predictable and stable distributions, are aggressively quantized to 4-bit (q4) and 3-bit (q3) formats. Because empirical analysis demonstrates that "depth self-heals quantization error"—meaning errors injected in early layers are progressively mitigated by deeper layers—the middle-to-late transformer blocks are compressed to the highly aggressive q3 format, while the initial layers remain at q447.
3. Embedding and Output Head Fidelity: The tied vocabulary embedding matrix and the final output projection head remain entirely in FP16 or high-fidelity INT8. Quantizing input embeddings to q4 or q3 causes severe, unrecoverable degradation in the model's ability to discriminate between phonetically or semantically similar tokens44.

This precise mixed-precision recipe enables a "bigger-but-smaller" operational regime. A 300M parameter model heavily compressed via q4/q3/INT8 mixed-precision routinely outperforms an uncompressed 100M FP16 model on reasoning benchmarks, while occupying an identical memory footprint and delivering superior inference speeds on hardware accelerators49.

Training and Evaluation Split Rules: Proving Capability

Tiny models are highly susceptible to dataset memorization, acting as stochastic parrots that regurgitate patterns rather than capable systems that generalize rules. To definitively prove that the .slm models possess genuine capability for browser UI integration rather than merely memorizing canned UI examples, uncompromising training and evaluation split rules must be enforced.

Decontamination and Split Rules

The training/evaluation split is not random; it is cryptographically isolated. A specific suite of human-curated evaluation sets (e.g., IFEval for instruction following, LiveCodeBench for algorithmic reasoning) is designated as the gold-standard evaluation tier14. The training corpus is passed through the 13-gram MinHash filter against these exact evaluation sets16. Furthermore, temporal isolation is utilized: the student models are evaluated on newly crafted, proprietary "UI state" datasets containing DOM structures, JSON schemas, and browser commands that were created after the cutoff date of the teacher model's pretraining data. If the student model performs well on this temporally isolated data, it proves zero-shot generalization to novel web environments.

Structured Quality Gates

Before an .slm model is certified for deployment, it must pass a gauntlet of automated quality gates designed specifically for web-native tasks:

• JSON Schema Adherence Gate: The model is evaluated against a held-out suite of 500 complex JSON schemas. The quality gate strictly demands a \>98% syntactic validity rate (parseable JSON) and a \>92% schema adherence rate (correct keys, correct value types, no hallucinated keys) for local agent routing tasks12.
• Code Explanation Pass@1 Gate: Using a decontaminated subset of front-end specific code, the model must successfully trace and explain basic DOM manipulation, CSS specificity, and asynchronous JavaScript patterns. This is evaluated via an LLM-as-a-judge mechanism utilizing a larger teacher model to verify semantic accuracy14.
• Rewrite and Summarization Gate: Performance is tracked using semantic similarity metrics (ROUGE-L, BLEURT) against reference texts rather than exact token match, ensuring the model generalizes semantic meaning and intent rather than parroting specific vocabularies51.

To further defeat memorization, evaluation prompts are systematically perturbed during the testing phase. Contextual variables, named entities, and syntactic structures within the prompts are algorithmically altered. If the .slm model's accuracy degrades precipitously upon these minor perturbations, it triggers a flag for data contamination or severe overfitting, halting deployment.

Rust Runtime Implications and the .slm Metadata Schema

The culmination of the distillation and quantization pipeline is the optimized .slm file, an artifact engineered explicitly for execution via Rust-based WebAssembly (WASM) and WebGPU runtimes within the browser.

WebGPU and Candle Execution Mechanics

The runtime environment leverages frameworks akin to Hugging Face's Candle, a minimalist machine learning framework written entirely in Rust. Candle circumvents the massive binary overhead of traditional PyTorch or ONNX bindings, compiling directly to ultra-lightweight WASM payloads. Minimalist implementations, such as wasmicro, can reduce the required WASM bundle size to a mere 199KB for full transformer inference2. In the browser, WebGPU provides low-level, high-throughput access to the client's local GPU architecture1. Because web browsers operate on a strict, single-threaded JavaScript event loop, inference operations must be meticulously offloaded to Web Workers to prevent catastrophic UI blocking and page unresponsiveness1. Memory management in this WASM environment is severely constrained by browser architecture (often capped at 4GB for 32-bit WASM implementations). To prevent memory leaks and garbage collection pauses, the Rust runtime enforces strict tensor ownership. Models utilize raw Vec\<f32\> structures, actively eschewing reference-counted pointers (Rc, Arc) and mutual exclusion locks (Mutex) during the forward pass43. The .slm format leverages the safetensors backing standard, enabling zero-copy deserialization where the quantized model weights are memory-mapped directly into the WebGPU buffer, completely bypassing the computationally expensive CPU-to-GPU memory duplication overhead2.

The .slm Metadata Specification

To ensure robust versioning, hardware compatibility profiling, and runtime configuration, every .slm file embeds a strictly defined, standardized JSON metadata header. This metadata allows the browser runtime to dynamically allocate buffer resources and verify model lineage before initiating the inference engine.

Summary

The production of highly capable, sub-billion parameter .slm models for browser-native execution demands a radical departure from the established pipelines of massive foundation models. By deliberately restricting vocabulary sizes to reclaim parameters for deep transformer layers, leveraging structured JsonTuning alongside stringently filtered synthetic data, and employing sophisticated cross-tokenizer logit distillation (ULD) with symmetric hidden-state matching (DistillLens), the architectural capacity of 50M to 500M models is systematically maximized. Furthermore, integrating Teacher Value-based Knowledge Distillation (TVKD) mathematically guarantees nuanced preference alignment without succumbing to the instability of standard DPO in low-capacity models. Culminating this pipeline with mixed-precision PCA-based quantization (q4/q3/FP16) preserves critical activation outliers and self-heals depth errors. The resulting .slm artifact—tightly integrated with Rust/WASM runtimes and executed via WebGPU—delivers an unprecedented, secure, low-latency AI inference engine operating entirely at the edge.

Works cited

1. AIRI: Complete Guide to Building Your Own AI VTuber Like Neuro-sama, https://explainx.ai/blog/airi-ai-vtuber-neuro-sama-guide-2026
2. Running BERT in web browsers \- Grokipedia, https://grokipedia.com/page/Running\_BERT\_in\_web\_browsers
3. MobileLLM-Flash: Latency-Guided On-Device LLM Design for Industry Scale Deployment, https://arxiv.org/html/2603.15954v2
4. \[2402.14905\] MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases \- arXiv, https://arxiv.org/abs/2402.14905
5. MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases \- arXiv, https://arxiv.org/pdf/2402.14905
6. Qwen2 \- Hugging Face, https://huggingface.co/docs/transformers/model\_doc/qwen2
7. Qwen2.5-0.5B: Specifications and GPU VRAM Requirements \- ApX Machine Learning, https://apxml.com/models/qwen2-5-0-5b
8. unknown\_url
9. Qwen2-0.5B: Specifications and GPU VRAM Requirements \- ApX Machine Learning, https://apxml.com/models/qwen2-0-5b
10. Building Instruction-Tuning Datasets from Human-Written Instructions with Open-Weight Large Language Models \- arXiv, https://arxiv.org/pdf/2503.23714
11. Building Instruction-Tuning Datasets from Human-Written Instructions with Open-Weight Large Language Models | OpenReview, https://openreview.net/forum?id=6vTv9M9ZAA¬eId=hsC5cj7bkO
12. JsonTuning: Towards Generalizable, Robust, and Controllable Instruction Tuning \- ACL Anthology, https://aclanthology.org/2025.findings-acl.1232.pdf
13. Small Language Models Eat the Edge: The 32× Cost Disruption Reshaping Enterprise AI, https://www.practicallogix.com/small-language-models-eat-the-edge-the-32x-cost-disruption-reshaping-enterprise-ai/
14. \[2504.04030\] OpenCodeInstruct: A Large-scale Instruction Tuning Dataset for Code LLMs, https://arxiv.org/abs/2504.04030
15. Synthetic Data for LLM Training: Decision Guide 2026 \- Digital Applied, https://www.digitalapplied.com/blog/synthetic-data-generation-llm-training-decision-guide-2026
16. Synthetic Data Generation on GPU Cloud: Distilabel, Augmentoolkit, and Nemotron-4 for LLM Fine-Tuning (2026 Guide) | Spheron Blog, https://www.spheron.network/blog/synthetic-data-generation-pipelines-gpu-cloud-distilabel-augmentoolkit-nemotron/
17. Mastering LLM Techniques: Text Data Processing | NVIDIA Technical Blog, https://developer.nvidia.com/blog/mastering-llm-techniques-data-preprocessing/
18. ResoFilter: Fine-grained Synthetic Data Filtering for Large Language Models through Data-Parameter Resonance Analysis \- arXiv, https://arxiv.org/html/2412.14809v3
19. Token-Based Knowledge Distillation \- Emergent Mind, https://www.emergentmind.com/topics/token-based-knowledge-distillation
20. On LLM Knowledge Distillation \- A Comparison between Forward KL and Reverse KL, https://d2jud02ci9yv69.cloudfront.net/2025-04-28-llm-knowledge-distil-157/blog/llm-knowledge-distil/
21. Daily Papers \- Hugging Face, https://huggingface.co/papers?q=KL-divergence%20distillation
22. SpecKD: Speculative Decoding for Effective Knowledge Distillation of LLMs \- arXiv, https://arxiv.org/html/2510.24021v1
23. Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs \- arXiv, https://arxiv.org/html/2402.12030v3
24. NeurIPS Poster Universal Cross-Tokenizer Distillation via Approximate Likelihood Matching, https://neurips.cc/virtual/2025/poster/119176
25. Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs \- arXiv, https://arxiv.org/html/2402.12030v2
26. Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs \- arXiv, https://arxiv.org/pdf/2402.12030
27. Universal Cross-Tokenizer Distillation via Approximate Likelihood Matching \- arXiv, https://arxiv.org/pdf/2503.20083
28. \[2503.20083\] Universal Cross-Tokenizer Distillation via Approximate Likelihood Matching, https://arxiv.org/abs/2503.20083
29. Cross-Tokenizer Likelihood Scoring Algorithms for Language Model Distillation \- arXiv, https://arxiv.org/html/2512.14954v2
30. DistillLens: Symmetric Knowledge Distillation Through Logit Lens \- arXiv, https://arxiv.org/html/2602.13567v1
31. DistillLens: Symmetric Knowledge Distillation Through Logit Lens \- arXiv, https://arxiv.org/pdf/2602.13567
32. MiniLM: Compact Transformer Model \- Emergent Mind, https://www.emergentmind.com/topics/minilm
33. MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers \- ACL Anthology, https://aclanthology.org/2021.findings-acl.188.pdf
34. (PDF) MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers (2021) | Wenhui Wang | 147 Citations \- SciSpace, https://scispace.com/papers/minilmv2-multi-head-self-attention-relation-distillation-for-4l0z3vfobt
35. unilm/minilm/README.md at master \- GitHub, https://github.com/microsoft/unilm/blob/master/minilm/README.md
36. Direct Preference Optimization in NeMo RL \- NVIDIA Documentation, https://docs.nvidia.com/nemo/rl/latest/guides/dpo.html
37. Direct Preference Optimization (DPO): An In-depth Analysis | by Deepak kumar sahoo | The Synaptic Stack | Medium, https://medium.com/the-synaptic-stack/direct-preference-optimization-dpo-an-in-depth-analysis-62a3d1f343a4
38. Preference Distillation via Value based Reinforcement Learning \- arXiv, https://arxiv.org/pdf/2509.16965
39. Preference Distillation via Value based Reinforcement Learning \- OpenReview, https://openreview.net/forum?id=f4GBN307sm
40. Preference Distillation via Value based Reinforcement Learning \- arXiv, https://arxiv.org/html/2509.16965v1
41. Preference Distillation via Value based Reinforcement Learning, https://neurips.cc/media/neurips-2025/Slides/116869.pdf
42. SLMs vs Quantized LLMs: What Actually Works for Browser AI | by Vishal Mysore \- Medium, https://medium.com/@visrow/slms-vs-quantized-llms-what-actually-works-for-browser-ai-c4bf3d1485b8
43. Xzdes/wasmicro \- GitHub, https://github.com/Xzdes/wasmicro
44. Day 18: 21 Days of Building a Small Language Model: Quantization : r/LocalLLaMA \- Reddit, https://www.reddit.com/r/LocalLLaMA/comments/1pvw9jo/day\_18\_21\_days\_of\_building\_a\_small\_language\_model/
45. Optimizing Large Language Models through Quantization: A Comparative Analysis of PTQ and QAT Techniques \- arXiv, https://arxiv.org/html/2411.06084v1
46. The Complete Guide to LLM Quantization \- LocalLLM.in, https://localllm.in/blog/quantization-explained
47. ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residuals, https://icml.cc/virtual/2025/poster/46466
48. Mixed-Precision Quantization for Language Models: Techniques and Prospects \- arXiv, https://arxiv.org/html/2510.16805v1
49. Variable Bit-width Quantization: Learning Per-Group Precision for “Bigger-but-Smaller” Language Models \- arXiv, https://arxiv.org/html/2607.02893v1
50. We Tested Quantized LLMs: Cost and Performance Results \- Latitude.so, https://latitude.so/blog/quantized-llms-cost-performance-results
51. Delta Knowledge Distillation for Large Language Models \- arXiv, https://arxiv.org/html/2509.14526v1
52. GitHub \- AutomataNexus/AxonML: PyTorch-equivalent deep learning framework in pure Rust — 22 crates, CUDA GPU, vision, LLMs, ONNX, autograd, distributed training, https://github.com/AutomataNexus/AxonML