Quantization-Aware Distillation and Architecture Design for Tiny Local Inference

Full report

On this page

Introduction

The deployment of Large Language Models (LLMs) in browser-local WebAssembly (WASM) environments introduces a paradigm shift in artificial intelligence engineering. By pushing inference to the edge, systems secure strict data privacy, eliminate network latency overhead, and achieve zero-cost cloud execution. However, browser-based Rust/WASM environments—specifically those governed by single-threaded scalar execution paradigms like the TinyRustLM ecosystem—impose draconian constraints on model size, memory allocation, and operational topology1. Designing an end-to-end model distillation pipeline to compress highly capable instruction-following teacher models into sub-billion parameter student models requires navigating a complex multidimensional optimization space. The resulting artifacts must be deterministic, highly quantizable, and strictly compliant with custom binary container specifications, all while preserving the reasoning, summarization, JSON structuring, and coding capabilities of models ten to fifty times their scale. This report establishes an exhaustive technical blueprint for constructing a Quantization-Aware Distillation (QAD) pipeline targeting the SLM1 (.slm) format. The analysis provides rigorous formulations for cross-tokenizer alignment, advanced loss functions for hidden-state and logit matching, synthetic data curriculum design, and evaluation gating for localized execution.

The Browser-Local Execution Environment and SLM1 Contract

The boundary conditions of the target environment dictate the entire distillation and architecture strategy. The TinyRustLM environment operates a scalar CPU execution thread within the main browser thread, bounded by a strict 128 MiB single-transfer allocation ceiling1.

Memory and Allocation Boundaries

The 128 MiB limit is a hard physical boundary for the artifact transfer within the WebAssembly linear memory layout1. Consequently, the total footprint of the model weights, alongside the dynamically allocated Key-Value (KV) cache, forward scratch buffers, and logit arrays, must remain strictly within defined tolerances. Memory allocations in this runtime are explicitly derived from the architecture parameters and calculated precisely before parsing admission1.

The sum of these allocations must securely pass the 128 MiB runtime threshold, leaving minimal overhead for the JavaScript response buffers and token vectors1. This strict budgeting dictates that parameter counts, sequence lengths, and precision formats are inextricably linked during the distillation design phase.

The SLM1 Binary Specification

The output of the distillation pipeline must be serialized into the custom SLM1 container format. The .slm version 1 format intentionally bypasses standard GGUF complexities in favor of a rigid, 108-byte fixed header and 64-byte aligned tensor directories2. The binary contract demands precise byte offsets for critical metadata, ensuring deterministic memory mapping during instantiation.

The runtime currently supports three data types: f32 (float32, type ID 1), q8\_0 (8-bit quantization, type ID 2), and q4\_0 (4-bit block-wise quantization, type ID 3\)2. The tensor directories follow the fixed header, requiring exactly 64 bytes per entry to define the FNV-1a hash of the tensor name, data type, rank, and payload offset2.

Sub-Billion Architecture Selection and Scaling Topology

Conventional scaling laws, which predict performance gains by widening embedding dimensions in multi-billion parameter regimes, fail systematically in the sub-billion parameter domain. To maximize the reasoning capacity of student models fitting within the 128 MiB boundary, the architecture must depart from scaled-down approximations of massive models3.

Deep and Thin Network Topologies

Empirical evaluations of sub-billion parameter models indicate that depth (the sheer number of sequential Transformer layers) is significantly more critical than width (the size of the hidden dimension) for abstract reasoning and semantic retention3. A deep and thin architecture allows the network to process hierarchical feature representations over more successive non-linear transformations5. This design choice effectively offsets the loss of expressive capacity caused by narrow embedding dimensions, enabling small models to maintain logical coherence over conversational contexts.

Parameter Reclamation via Embedding Sharing

In standard large language models, the vocabulary embedding layer constitutes a minimal fraction of the total parameter count. However, in a highly constrained 125M parameter model, a standard 32,000-token vocabulary combined with a 768-dimensional embedding matrix consumes approximately 24.5 million parameters—representing roughly 20% of the network's entire capacity4. Tying the input embeddings to the final pre-softmax output projection layer reclaims these parameters entirely3. The SLM1 format is purpose-built to support tied embeddings, managed through Bit 0 of the flags integer at Offset 162. The parameters liberated through embedding tying are systematically reallocated to instantiate additional Transformer blocks, directly extending the reasoning depth of the model without violating the overall memory ceiling.

Latency Optimization and Grouped-Query Attention

To govern the expanding Key-Value cache—a volatile requirement for the browser's dynamic memory budget—Grouped-Query Attention (GQA) is enforced3. By reducing the ratio of KV heads to Query heads (e.g., 16 Query heads to 4 KV heads), the runtime memory consumed during autoregressive decoding is drastically reduced. This permits the scalar WASM environment to support longer context windows, accommodating up to 2,048 tokens without exhausting the 128 MiB hard limit1. Furthermore, immediate block-wise weight sharing is utilized to optimize execution latency. In this paradigm, adjacent Transformer blocks (e.g., Block [Figure omitted from source export] and Block [Figure omitted from source export]) share identical weight matrices but maintain independent residual streams7. This structural reuse effectively doubles the logical depth of the network with only a marginal increase in serialized file size. In CPU-bound WASM environments, memory bandwidth and weight transfer often bottleneck execution more than raw floating-point operations; thus, computing the same block twice sequentially while keeping the weights hot in the L1/L2 cache significantly reduces latency5.

Cross-Tokenizer Alignment via Byte-Level Interfaces

A principal point of failure in modern distillation pipelines is tokenizer mismatch. Highly capable teacher models (e.g., Llama-3, Qwen) utilize massive vocabularies often exceeding 100,000 tokens, whereas a TinyRustLM-compatible student model must employ a minimal vocabulary (e.g., 32,000 tokens) to minimize embedding parameter overhead10. Direct logit distillation fundamentally requires identical probability spaces. If the teacher tokenizes a string into four tokens and the student parses the exact same string into seven tokens, standard sequence alignment and Kullback-Leibler (KL) divergence fail catastrophically10.

Approximate Likelihood Matching (ALM)

To circumvent the necessity of complex combinatorial mapping heuristics, the pipeline implements Approximate Likelihood Matching (ALM) through a shared Byte-Level Distillation (BLD) interface13. Every tokenizer can ultimately be decomposed into a sequence of raw UTF-8 bytes. The ALM algorithm bridges disparate vocabularies by computing chunk-level probabilities mapped to the underlying invariant byte sequence14. The alignment process functions sequentially: The training text is tokenized independently by the teacher and the student. The teacher's token probabilities are projected down into byte-level transition probabilities using an algorithmic decomposition strategy13. Boundaries where the student and teacher tokenizations align at the exact same byte-offset in the string are identified as synchronous "chunks"14. Because specific bytes never appear at the start of a token (a phenomenon known as tokenization bias), a debiasing masking function is applied14. This function prevents the student model from penalizing valid string continuations that misalign purely because of its specific Byte-Pair Encoding (BPE) merge rules14. The resulting cross-tokenizer KL divergence is computed exclusively over these aligned byte-chunks, enabling the highly constrained student model to receive exact probability gradients from a teacher utilizing a completely alien tokenizer architecture14.

The Quantization-Aware Distillation (QAD) Mathematical Framework

Distilling a multi-billion parameter instruction-tuned model into a highly constrained student requires multiple mathematically aligned objectives. Standard Post-Training Quantization (PTQ) severely degrades sub-billion models due to rounding noise, activation outliers, and representational drift18. Alternatively, Quantization-Aware Training (QAT) relies heavily on standard supervised task loss, which introduces severe training instability and often fails to recover emergent capabilities18. Quantization-Aware Distillation (QAD) mitigates this by integrating low-precision arithmetic simulation directly into the continuous distillation process, matching the high-precision teacher's soft targets rather than hard labels18. The total distillation loss is a composite of feature-level, attention-level, and logit-level alignments, formulated as: [Figure omitted from source export]

Feature-Level Alignment: Centered Kernel Alignment (CKA)

Forcing the student's hidden states to mimic the teacher's is computationally prohibitive due to dimensionality mismatch (e.g., Teacher [Figure omitted from source export], Student [Figure omitted from source export]). Linear Centered Kernel Alignment (CKA) circumvents the need for parameterized projection matrices by measuring the statistical dependence between the feature spaces using layer-wise Gram matrices24. Given centered activation matrices for [Figure omitted from source export] tokens, [Figure omitted from source export] and [Figure omitted from source export], CKA is computed as: [Figure omitted from source export] The loss is defined to maximize this alignment: [Figure omitted from source export]. CKA is invariant to isotropic scaling and orthogonal transformations, making it exceptionally resilient to the magnitude shifts and rotations inherently caused by fake-quantization operators during training25. To focus the distillation process on semantically dense tokens—such as entities, mathematical operators, or reasoning steps—rather than generic stop words, Attention-weighted CKA (AwCKA) is employed27. This variant applies a temporal weighting vector [Figure omitted from source export] derived directly from the teacher's final layer self-attention scores, guiding the CKA matrix to prioritize critical temporal junctures27.

Attention Map Alignment Distillation (AMAD)

Vision and language research demonstrates that transferring attention distributions helps student models replicate the reasoning trajectories and cross-modal references of their teachers29. However, the teacher and student possess vastly different configurations of attention heads (e.g., 32 heads versus 12 heads). AMAD utilizes a cross-model soft-alignment mechanism29. Each teacher head [Figure omitted from source export] contributes to student head [Figure omitted from source export] based on a dynamically learned similarity-based weighting matrix [Figure omitted from source export]. The divergence between the softmax probability distributions of the attention maps is minimized without enforcing a strict one-to-one mapping29: [Figure omitted from source export]

Logit-Level Distillation: Decoupled BiLD

Traditional Kullback-Leibler divergence calculated over the entire vocabulary space is dominated by the teacher's highest-probability mode, which often masks the rich relational "dark knowledge" found in the long-tail logits30. Conversely, when the teacher expresses extremely high confidence in a single token, low-confidence long-tail logits represent unstructured noise that harms the highly limited capacity of sub-billion student models31. The Bi-directional Logits Difference (BiLD) loss addresses this by explicitly decoupling the top-K logits from the tail31. Given temperature-scaled teacher probabilities [Figure omitted from source export] and student probabilities [Figure omitted from source export]: [Figure omitted from source export] Here, [Figure omitted from source export] isolates the KL divergence restricted to the indices of the teacher's top-K predictions, pulling the student toward the teacher's primary semantic manifold31. Simultaneously, [Figure omitted from source export] penalizes the student if its highest confidence top-K predictions diverge significantly from the teacher's corresponding indices at those specific positions31. This decoupled approach prevents the student from expending valuable parameter capacity on modeling sub-optimal token probabilities (e.g., assigning a 0.0001 probability to the word "airplane" in a context strictly about "dogs") while rigorously enforcing adherence to the correct outputs.

Quantization-Aware Optimization: STE and FOGZO

Because the final artifact must be quantized to q4\_0 or q8\_0 for the WASM environment, fake quantization nodes are injected into the student model's computational graph during training18. The q4\_0 format relies on a block size of 32 weights sharing a single scale factor33. This is represented algebraically as: [Figure omitted from source export], where [Figure omitted from source export] is the discrete 4-bit integer34. The rounding operation to discrete integers is fundamentally non-differentiable, severing the gradient flow during backpropagation. To allow gradients to propagate backward, the Straight-Through Estimator (STE) is employed, approximating the derivative of the round function as an identity matrix36: [Figure omitted from source export] Where [Figure omitted from source export] represents the full-precision latent weights and [Figure omitted from source export] represents the simulated quantized weights37. While standard STE is sufficient for general 8-bit precision, the highly aggressive block-wise q4\_0 format introduces significant quantization bias. First-Order-Guided Zeroth-Order (FOGZO) gradient estimation mitigates this STE bias by incorporating randomized gradient smoothing, allowing the network parameters to settle into optimal quantization-friendly local minima without gradient vanishing37.

Sequence-Level Distillation and Synthetic Curriculum Design

Off-policy distillation—where the student is trained purely on static text datasets—frequently leads to exposure bias39. When the student deviates even slightly from the gold-standard trajectory during autoregressive generation, it enters unseen distributional spaces and begins to hallucinate, as it has never been trained on how to recover from its own errors.

On-Policy Self-Distillation (OPSD)

To correct exposure bias, the pipeline relies on On-Policy Self-Distillation (OPSD)39. In this regimen, the student model generates full sequence rollouts based on instruction prompts. The teacher model is then queried to score the student's generated trajectory, providing dense, token-level logit feedback on the student's chosen path39. This guarantees that the student learns how to recover from sub-optimal token selections. The objective minimizes the divergence over the student's empirical rollout distribution rather than a static dataset39: [Figure omitted from source export]

Mitigating Canned-Prompt Overfitting

If the prompt dataset consists exclusively of generic requests (e.g., "Write a poem about a tree," "Summarize this article"), a sub-100M model will rapidly overfit, memorizing stylistic tics rather than acquiring generalized reasoning and instruction-following capabilities. Synthetic instruction generation must enforce extreme entropy across the curriculum. Using a diverse unlabelled seed dataset (e.g., FineWeb-Edu, Wikipedia), an orchestration teacher (e.g., Llama-3-70B) is prompted to generate highly constrained, domain-specific tasks based on the unlabelled text. Prompts must require strict structural outputs (e.g., "Return ONLY valid JSON with keys X and Y"). Furthermore, injecting adversarial complexity—such as disjointed syntax, multi-constraint negative prompts, and spelling errors—into the synthetic prompts ensures the student learns robust attention distributions rather than superficial text-pattern matching.

Chat Template Preservation

Tiny models are notoriously fragile regarding prompt formatting. The precise chat template (e.g., ChatML, or the specific Llama-3 instruction format utilizing explicit \<|begin\_of\_text|\>, \<|start\_header\_id|\>, and \<|end\_of\_text|\> tokens) must be rigorously preserved throughout the distillation process. During logit matching, loss masking must not be applied to control tokens. The distillation loss must heavily penalize the student if its probability of emitting a stop token diverges from the teacher's, preventing catastrophic run-on generation in the local WASM runtime where generation cancellation controls may be limited1.

Dataset Mixture Design

A 135M or 85M model cannot function as a universal polymath. To perform effectively in a constrained browser environment, its training mixture must ruthlessly prioritize specific narrow competencies over broad factual world knowledge.

Concrete Algorithms and Training Configurations

Training sub-billion models via distillation requires carefully orchestrated phases. Attempting to apply QAD, OPSD, and structural pruning simultaneously typically results in complete mode collapse. A decoupled architectural framework—where the student is housed on a training engine (e.g., PyTorch FSDP2) and the teacher operates on a high-throughput inference engine (e.g., vLLM or SGLang)—is strictly recommended to avoid memory bottlenecks and maximize GPU utilization41.

The Two-Stage Training Algorithm

Stage 1: Feature Alignment and FP32 Distillation (The "Warmup") The objective of this stage is to initialize the student weights to mirror the teacher's latent manifold before introducing quantization noise.

1. Initialize the deep-and-thin student model with tied embeddings.
2. Disable all quantization nodes. Train entirely in bfloat16.
3. Execute forward passes on the synthetic dataset mixture. Apply CKA hidden-state matching ([Figure omitted from source export]) and AMAD attention matching ([Figure omitted from source export]).
4. Hyperparameters: Utilize a high peak learning rate ([Figure omitted from source export]) with a cosine decay schedule over a massive corpus (e.g., 20 billion tokens). Batch size should be maximized to GPU memory limits (e.g., 1024 to 2048).

Stage 2: Quantization-Aware On-Policy Distillation (The "Compression") The objective of this stage is to adapt the student to q4\_0 or q8\_0 precision while mastering strict instruction following via active rollout generation.

1. Inject STE/FOGZO fake-quantization nodes matching the SLM1 q4\_0 block size of 32 weights per scale factor33.
2. Shift the loss function to the Decoupled BiLD logit loss and initiate OPSD rollouts31.
3. Hyperparameters: Utilize a significantly lower peak learning rate ([Figure omitted from source export]) to allow weights to settle into quantization-friendly local minima without catastrophic forgetting. Apply a high temperature scaling factor ([Figure omitted from source export]) to soften the teacher's logit distributions.

Evaluation Gates, Memorization Detection, and Failure Modes

Before an artifact is promoted to a release candidate for browser execution, it must pass strict quality and structural gates to ensure it does not compromise the scalar WASM environment.

Detecting Memorization and Prompt-Specific Behavior

Distilled models with highly constrained capacities are prone to memorizing the exact phrasing of the teacher's training set rather than learning the underlying algorithmic task. Evaluation must explicitly test out-of-distribution (OOD) generalization.

• Per-Token Entropy Analysis: High confidence (low entropy) on training prompts combined with catastrophic perplexity on slight semantic variations indicates severe memorization.
• Reverse String and Algorithmic Checks: Testing the model on pure algorithmic tasks (e.g., reversing the spelling of novel, randomly generated strings) proves algorithmic acquisition. If a 135M model perfectly recites historical facts but fails basic reasoning tasks, it has memorized static data at the expense of necessary reasoning capabilities.
• Needle-in-a-Haystack (NIAH): Evaluates context window utilization and retrieval capability under GQA up to the maximum 2,048 tokens.

Recognized Failure Modes

• Representational Drift: Aggressive output logit alignment can mask internal feature reorganization, leading to brittle reasoning25. Maintaining AwCKA regularization throughout Stage 2 prevents the internal layers from collapsing into superficial mappings25.
• Quantization Amplification: Deep networks accumulate rounding errors sequentially. A q4\_0 block scale outlier in an early layer will catastrophically impact final output logits. Utilizing FOGZO gradient estimation smooths the loss landscape, preventing outlier saturation during QAD37.
• Repetition Collapse: The student fixates on specific tokens, looping endlessly during autoregressive decoding. Ensuring high entropy in OPSD student rollouts through dynamic temperature sampling mitigates this loop behavior40.

Recommended Experimental Targets

To identify the optimal balance of reasoning capability versus runtime viability, three distinct model configurations should be trained and benchmarked against the TinyRustLM ecosystem bounds.

Note: The 360M parameter model explicitly violates the 128 MiB transfer ceiling of the TinyRustLM runtime1. Deployment of this artifact would mandate an architectural refactor of the runtime to support chunked, verified persistent local storage (Gate C in the MiRust specification roadmap)1.

Provenance and SLM Packaging

Upon successful passing of the evaluation gates, the final bfloat16 PyTorch/JAX checkpoint is subjected to definitive structural quantization according to the strict SLM1 binary contract2. The packaging algorithm executes as follows: First, the tensors are materialized and grouped into quantization blocks of 32 elements. For q4\_0, the maximum absolute value per block determines the scaling scalar34. Second, each tensor is assigned a precise 64-byte directory entry containing its FNV-1a name hash, data type identifier (3 for q4\_0), rank, structural dimensions, and absolute byte offset2. Next, the 108-byte header is constructed. Bit 0 of the flags is toggled to 1 to confirm tied embeddings. The vocabulary size, layer count, and context length are injected into their precise byte offsets2. Finally, the entire payload is hashed (with bytes 100–107 temporarily zeroed), and the generated non-cryptographic checksum is written to the header2. The completed .slm binary is then subjected to a rigorous admission test against the tinyrustlm-runtime parser2. If the serialized memory footprint of the artifact, combined with the calculated KV cache and forward scratch buffers, exceeds the 128 MiB global limit, the build is rejected by the Continuous Integration (CI) pipeline1. Evaluative evidence, including NIAH scores and structured JSON pass rates, is serialized into a separate cryptographic manifest to maintain provenance without violating the SLM1 binary specification, ensuring a deterministic, transparent release cycle for edge-deployed intelligence.

Works cited

1. Implementation operations \- MiRust, https://mirust.com/implementation-operations/
2. Implementation \- MiRust, https://mirust.com/implementation/
3. MobileLLM Optimizing Sub-billion Parameter Language Models for On-Device Use Cases. In ICML 2024\. \- GitHub, https://github.com/facebookresearch/mobilellm
4. Meta MobileLLM Advances LLM Design for On-Device Use Cases \- InfoQ, https://www.infoq.com/news/2024/11/meta-mobilellm/
5. Meta Researchers Present MobileLLM | ml-news – Weights & Biases \- Wandb, https://wandb.ai/byyoung3/ml-news/reports/Meta-Researchers-Present-MobileLLM--Vmlldzo4NjA5MTc3
6. Meta AI Releases MobileLLM 125M, 350M, 600M and 1B Model Checkpoints, https://www.marktechpost.com/2024/10/31/mete-ai-releases-mobilellm-125m-350m-600m-and-1b-model-checkpoints/
7. \[2402.14905\] MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases \- arXiv, https://arxiv.org/abs/2402.14905
8. MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases \- arXiv, https://arxiv.org/pdf/2402.14905
9. The Complete Guide to LLM Quantization with vLLM: Benchmarks & Best Practices, https://jarvislabs.ai/blog/vllm-quantization-complete-guide-benchmarks
10. Dual-Space Knowledge Distillation with Key-Query Matching for Large Language Models with Vocabulary Mismatch \- arXiv, https://arxiv.org/html/2603.22056v1
11. Dual-Space Knowledge Distillation for Large Language Models \- ACL Anthology, https://aclanthology.org/2024.emnlp-main.1010.pdf
12. Overcoming Vocabulary Mismatch: Vocabulary-agnostic Teacher Guided Language Modeling \- arXiv, https://arxiv.org/html/2503.19123v1
13. \[2604.07466\] Cross-Tokenizer LLM Distillation through a Byte-Level Interface \- arXiv, https://arxiv.org/abs/2604.07466
14. NeurIPS Poster Universal Cross-Tokenizer Distillation via Approximate Likelihood Matching, https://neurips.cc/virtual/2025/poster/119176
15. Cross-Tokenizer LLM Distillation through a Byte-Level Interface \- ResearchGate, https://www.researchgate.net/publication/403682275\_Cross-Tokenizer\_LLM\_Distillation\_through\_a\_Byte-Level\_Interface
16. CROSS-TOKENIZER LIKELIHOOD SCORING ALGORITHMS FOR LANGUAGE MODEL DISTILLATION \- OpenReview, https://openreview.net/pdf?id=hD69qj15Os
17. Cross-Tokenizer LLM Distillation through a Byte-Level Interface \- ACL Anthology, https://aclanthology.org/2026.customnlp4u-1.9.pdf
18. Quantization-Aware Distillation for NVFP4 Inference Accuracy Recovery \- arXiv, https://arxiv.org/html/2601.20088v1
19. Quantization-Aware Distillation for NVFP4 Inference Accuracy Recovery \- Research at NVIDIA, https://research.nvidia.com/labs/nemotron/files/NVFP4-QAD-Report.pdf
20. Quantization-Aware Training: Empowering efficient AI on edge devices | qat \- Wandb, https://wandb.ai/onlineinference/qat/reports/Quantization-Aware-Training-Empowering-efficient-AI-on-edge-devices--VmlldzoxMTcyOTEwMA
21. Enable NVFP4 Inference for Nemotron with Quantization-Aware Distillation, https://research.nvidia.com/labs/nemotron/nemotron-qad/
22. LLM Quantization and Knowledge Distillation: How It Works \- ScriptsHub Technologies, https://scriptshub.net/resources/blogs/llm-quantization-and-knowledge-distillation/
23. (PDF) Quantization-Aware Distillation for NVFP4 Inference Accuracy Recovery, https://www.researchgate.net/publication/400178491\_Quantization-Aware\_Distillation\_for\_NVFP4\_Inference\_Accuracy\_Recovery
24. ICLR Poster Improving Language Model Distillation through Hidden State Matching, https://iclr.cc/virtual/2025/poster/30163
25. Beyond Output Matching: Preserving Internal Geometry in NVFP4 LLM Distillation \- arXiv, https://arxiv.org/html/2606.05682v2
26. (PDF) Beyond Output Matching: Preserving Internal Geometry in NVFP4 LLM Distillation, https://www.researchgate.net/publication/406039642\_Beyond\_Output\_Matching\_Preserving\_Internal\_Geometry\_in\_NVFP4\_LLM\_Distillatio
27. Attention-weighted Centered Kernel Alignment for Knowledge Distillation in Large Audio-Language Models Applied to Speech Emotion \- arXiv, https://arxiv.org/pdf/2602.01547
28. Attention-weighted Centered Kernel Alignment for Knowledge Distillation in Large Audio-Language Models Applied to Speech Emotion Recognition \- arXiv, https://arxiv.org/html/2602.01547
29. No Head Left Behind \- Multi-Head Alignment Distillation for Transformers \- Amazon Science, https://cdn.amazon.science/6d/59/a65b4252470cb893eaad167f7804/no-head-left-behind-multi-head-alignment-distillation-for-transformers.pdf
30. Knowledge Distillation: How to Compress Large Models into Small Ones That Actually Work, https://miraflow.ai/blog/knowledge-distillation-compress-large-models-into-small-ones
31. BiLD: Bi-directional Logits Difference Loss for Large Language Model Distillation \- ACL Anthology, https://aclanthology.org/2025.coling-main.78.pdf
32. DeepKD: A Deeply Decoupled and Denoised Knowledge Distillation Trainer \- OpenReview, https://openreview.net/forum?id=wTxh9bKSXh\&referrer=%5Bthe%20profile%20of%20Haiduo%20Huang%5D(%2Fprofile%3Fid%3D\~Haiduo\_Huang1)
33. Unsloth Dynamic 2.0 GGUFs, https://unsloth.ai/docs/basics/unsloth-dynamic-2.0-ggufs
34. GGUF · Hugging Face, https://huggingface.co/docs/hub/gguf
35. The Complete Guide to LLM Quantization: Demystifying q4\_K\_M \- Zenn, https://zenn.dev/taku\_sid/articles/20250415\_llm\_quantization?locale=en
36. Straight-Through Estimator (STE) \- Emergent Mind, https://www.emergentmind.com/topics/straight-through-estimator-ste
37. Improving the Straight-Through Estimator with Zeroth-Order Information \- arXiv, https://arxiv.org/html/2510.23926v1
38. Improving the Straight-Through Estimator with Zeroth-Order Information \- NIPS, https://proceedings.neurips.cc/paper\_files/paper/2025/file/f49d76cf84df83a611883c621c96d2d9-Paper-Conference.pdf
39. On-Policy Self-Distillation for Large Language Models \- Siyan Zhao, https://siyan-zhao.github.io/assets/img/opsd/opsd\_v3.pdf
40. Primers • Knowledge Distillation \- aman.ai, https://aman.ai/primers/ai/knowledge-distillation/
41. KDFlow: A User-Friendly and Efficient Knowledge Distillation Framework for Large Language Models \- arXiv, https://arxiv.org/html/2603.01875v2