Advanced Paradigms in Tiny Language Model Compression: Distillation, Pruning, and Edge Deployment

Full report

On this page

The rapid scaling of Large Language Models (LLMs) has catalyzed breakthroughs in natural language understanding and generation, but the accompanying explosion in parameter counts has fundamentally limited deployment on resource-constrained edge devices. To bridge the gap between cloud-scale intelligence and edge-native constraints, the industry is shifting toward highly optimized Small Language Models (SLMs) and bespoke .slm or .gguf artifacts. While post-training quantization (PTQ) has historically served as the primary mechanism for reducing memory footprints, quantization alone is insufficient. It merely reduces the precision of a dense computational graph, preserving architectural redundancies and incurring severe accuracy degradation at extreme low-bit regimes. The creation of state-of-the-art tiny models necessitates a paradigm shift: moving beyond simple numerical precision reduction to structural graph compression, targeted knowledge transfer, and hardware-aware sparsity. This analysis explores the theoretical foundations and practical applications of vocabulary reduction, structured and unstructured pruning, low-rank factorization, and advanced teacher-student distillation. Furthermore, the analysis investigates the intersection of these compressed architectures with modern sparse runtimes deployed via Rust and WebAssembly (WASM), culminating in a definitive pipeline for engineering high-quality .slm models resilient to quality collapse.

Phase 1: Lexical Restructuring and Vocabulary Reduction

A substantial portion of an SLM's memory footprint is consumed by vocabulary-related components, specifically the input embedding matrix and the language modeling (LM) head. As models scale down, the proportion of parameters dedicated to the vocabulary scales inversely. For instance, in a 70B parameter model, the embedding layer may account for a mere 3% of the total architecture, whereas in a 1.8B parameter model, the embedding layer can consume up to 34% of the parameter budget1. This architectural imbalance highlights vocabulary reduction as a critical, yet historically underutilized, vector for model compression.

Static Vocabulary Trimming and Embedding Shrinkage

Static vocabulary trimming operates on the principle of lexical locality, observing that only a narrow subset of the total Byte-Pair Encoding (BPE) vocabulary is required for specific downstream tasks or target languages2. The traditional BPE algorithm prioritizes high-frequency byte-level merges across a massive multilingual pre-training corpus, resulting in vocabularies that often exceed 100,000 tokens1. For domain-specific or single-language .slm deployments, retaining this global vocabulary is computationally wasteful. The trimming process begins by analyzing a target corpus to identify the maximally utilized subset of tokens. The tokenizer is then reconstructed with this reduced vocabulary. Crucially, because the embedding layer and the LM head are structurally tied to the tokenizer's dictionary, shrinking the vocabulary allows for the physical deletion of corresponding rows and columns in the embedding and projection matrices1. To mitigate the introduction of out-of-vocabulary (OOV) errors for tokens that were pruned, the algorithm dynamically generates new embeddings by averaging the latent representations of surviving constituent subword tokens1. This ensures that the continuous concept space remains intact, allowing the network to parse unseen or pruned words through a composition of smaller, retained subwords. This approach exploits a fundamental computation asymmetry within transformer architectures2. The input embedding layer relies on memory-bandwidth-bound lookup operations, while the LM head relies on compute-intensive matrix multiplications. By shrinking the vocabulary—often adjusting the final size to a hardware-friendly multiple of 8 or 64—the SLM achieves exponential reductions in LM head FLOPs while simultaneously relieving the memory bus during the prefill stage5. Empirical studies on models like GPT-2 Small demonstrate that trimming a 50,257-token vocabulary down to 32,768 tokens reduces the embedding layer parameters by approximately 35%, yielding a total model parameter reduction of nearly 11% without requiring any architectural modifications to the transformer backbone5. The sequencing of this operation is paramount; trimming must precede task-specific fine-tuning, as reversing the order destroys learned representations that rely on the pre-pruned vocabulary space5.

Advanced Tokenizer Constraints and Character-Level Projections

While standard BPE trimming is highly effective, more extreme compression demands novel output projection strategies. Frameworks such as SpeLLM address the linear scaling cost of the output projection layer by decoupling the input vocabulary from the generation vocabulary7. In this architecture, the standard LM head is replaced by multiple parallel linear heads, each projecting the final hidden state into a minimal, character-level vocabulary7. By predicting text at the character level across parallel heads while maintaining a standard BPE input, the model drastically reduces the LM head size. This combinatorial expansion of expressive capacity allows the model to maintain rapid decoding speeds while minimizing the memory constraints typically associated with massive output embedding tables7.

Dynamic Lossless Vocabulary Reduction (LVR)

While static trimming is highly effective for isolated deployments, it inherently sacrifices the model's generalist capabilities and creates a rigid artifact that struggles to interact with models utilizing disparate tokenizers. To resolve this, advanced inference techniques now leverage dynamic Lossless Vocabulary Reduction (LVR)8. LVR functions without altering the underlying pre-trained weights. Instead, it establishes a theoretical framework that consistently handles token probabilities across original and sub-vocabularies. During the autoregressive generation phase, the model dynamically transforms its next-token prediction distribution in real-time, recalculating softmax weights so they can be expressed over a restricted, specified subset of tokens8. Because the probabilities over a sub-vocabulary can be expressed as a mathematical superposition of probabilities over the original vocabulary, the network behaves as if it is operating at full capacity9. This technique prevents the phrasing degradation that typically occurs when tokens are aggressively masked, enabling heterogeneous AI models with incompatible foundational tokenizers to communicate via a "maximum common vocabulary"8. Theoretical proofs validate that this bijective mapping between reduced and original vocabularies is mathematically reversible, making it highly suitable for production ensembles where distinct models (e.g., a 150,000-token LLM and a 130,000-token LLM) can seamlessly exchange next-token predictions through a shared 60,000-token subspace10.

Phase 2: Structural Pruning Across Depth and Width

Once the embedding bottleneck is resolved, compression efforts must address the dense computation occurring within the transformer blocks. Pruning removes redundant parameters, but the distinction between structured and unstructured pruning dictates the resulting model's compatibility with standard hardware accelerators. Structured pruning physically alters the dimensions of the matrices, while unstructured pruning introduces zeroes into matrices that retain their original shape.

Layer Dropping and Depth Pruning

Layer dropping, or depth pruning, operates on the observation that modern over-parameterized LLMs exhibit severe redundancy along their depth axis. Entire transformer blocks—comprising both the multi-head self-attention and the feed-forward networks (FFNs)—can often be removed with minimal degradation to downstream perplexity12. The identification of expendable layers relies on intrinsic metrics rather than computationally expensive gradient-based searches. The Block Influence (BI) metric, introduced by the ShortGPT framework, quantifies the transformation a specific layer applies to the hidden states passing through it12. The formulation calculates the expected cosine similarity between the input representation [Figure omitted from source export] and the output representation [Figure omitted from source export] of layer [Figure omitted from source export]: [Figure omitted from source export] A low BI score indicates that the input and output vectors are nearly identical in the high-dimensional latent space12. Consequently, the layer is acting as an approximate identity function and contributing negligibly to the network's reasoning capabilities12. Research indicates that deeper layers in LLMs exhibit the highest redundancy; dropping a significant percentage of these terminal layers yields inference speedups of 2x to 5x while maintaining upwards of 95% of the original zero-shot performance13. For instance, deleting the final 10 layers of a 40-layer LLM architecture can reduce parameter counts by 25% while only degrading benchmark scores by negligible margins12. However, relying solely on sequential layer importance assumes error monotonicity—the flawed assumption that a lower sum of individual per-layer errors guarantees a lower total compression error14. To counteract this, advanced layer pruning frameworks evaluate layers in overlapping triplets, recognizing that interlace pruning (removing the most redundant of two consecutive layers while freezing the third to serve as a stable anchor) prevents the representation collapse that occurs when large blocks of continuous trainable layers are removed simultaneously16.

Computational Invariance and Width Pruning

While layer dropping removes entire blocks, width pruning seeks to compress the intermediate dimensions of all remaining blocks. The SliceGPT framework pioneers a post-training sparsification scheme that bypasses the need for immediate recovery fine-tuning by exploiting computational invariance within the transformer architecture17. SliceGPT identifies that applying an orthogonal transformation matrix [Figure omitted from source export] to the output of one component is computationally neutral if the inverse transformation is applied to the subsequent component. For networks connected via RMSNorm, this invariance is mathematically guaranteed, as [Figure omitted from source export]18. The algorithm projects the signal matrix between blocks onto its principal components using Principal Component Analysis (PCA) and physically slices off the rows and columns corresponding to the lowest variance18. This structured reduction of the embedding dimension shrinks the query, key, value, and output projection matrices uniformly, allowing dense matrix multiplication to continue seamlessly on edge hardware17.

MLP Neuron Pruning and Attention Head Reduction

Beyond uniform width reduction, structural pruning must target the two primary sub-components of the transformer block independently: the Multi-Layer Perceptron (MLP) and the Multi-Head Attention (MHA) modules. The SwiGLU FFNs typically account for two-thirds of a model's parameter count. Model Minification frameworks target these massive blocks by employing a Taylor-First-Order sensitivity analysis to rank the importance of intermediate neurons21. The sensitivity score [Figure omitted from source export] for a neuron is approximated by the inner product of the weights and their gradients. Following the removal of low-scoring neurons, the architecture applies a closed-form Ridge Regression reconstruction to the remaining matrices21. This second-order healing process minimizes the L2 distance between the pruned layer's output and the original dense output, allowing highly parameterized models to tolerate the physical deletion of up to 30% of their intermediate width21. Smaller, dense models (e.g., 135M parameters) are far more fragile and typically require conservative minification, maintaining 80% to 90% retention to prevent the destruction of polysemantic features21. Attention head pruning operates on a similar principle of redundancy elimination. Rather than aggregating individual element importance, advanced methodologies utilize Pearson similarity-based head pruning coupled with greedy search algorithms22. Other frameworks, such as Object-DINO, cluster attention heads across all layers based on the similarities of their spatial-temporal patches, automatically identifying and preserving object-centric clusters while excising redundant semantic heads23.

Phase 3: Low-Rank Factorization and Activation Whitening

Singular Value Decomposition (SVD) offers a parallel route to structural compression by decomposing large weight matrices into two smaller, low-rank matrices24. Traditionally, applying SVD directly to LLM weights and truncating the smallest singular values results in catastrophic performance loss, as standard SVD assumes a uniform distribution of input activations26. In reality, LLM activations contain extreme outliers that are critical for preserving language generation capabilities. To resolve this, advanced factorization techniques like SVD-LLM introduce Truncation-Aware Data Whitening24. Using a calibration dataset, the algorithm extracts the activation matrix [Figure omitted from source export] and derives a whitening matrix [Figure omitted from source export] via Cholesky decomposition, such that [Figure omitted from source export]27. The algorithm then performs SVD on the product of the weight matrix and the whitening matrix. Because the whitened input space is orthonormal, the Eckart-Young-Mirsky theorem applies perfectly, guaranteeing that truncating the smallest singular values directly minimizes the reconstruction error of the activations27.

Following truncation, a layer-wise closed-form update tunes the newly compressed weight matrices against fresh activations to heal the localized damage, significantly elevating the network's resilience at high compression ratios29. However, engineers must be wary of the Cross-Modal Alignment Gap when compressing vision-language models. The mismatch matrix [Figure omitted from source export] quantifies the geometric deviation between text-whitened and image-whitened subspaces; using a text-optimized basis on multimodal data amplifies the truncation error, necessitating Joint-Whitening SVD to dynamically reallocate redundancy across disparate modalities27.

Phase 4: Next-Generation Knowledge Distillation

Structural pruning and factorization inherently degrade the network's latent reasoning capabilities. To recover and even exceed the baseline performance within the new .slm topology, the pipeline must employ advanced Knowledge Distillation (KD). Standard supervised fine-tuning (SFT) over hard labels is insufficient for language modeling; the student model requires the rich, probabilistic guidance of a massive teacher model.

Logit Distillation: The Shift to Reverse Kullback-Leibler Divergence

Historically, sequence-level logit distillation relied on minimizing the Forward Kullback-Leibler Divergence (Forward KLD) between the teacher's distribution [Figure omitted from source export] and the student's distribution [Figure omitted from source export]. Forward KLD forces the student to cover all modes of the teacher's distribution31. In the context of generative language modeling, where the output space is vast and multimodal, an under-parameterized student lacks the capacity to map the entirety of the teacher's knowledge. Attempting to do so causes the student to assign unreasonably high probabilities to void regions of the teacher's distribution, manifesting as hallucinations, severe exposure bias, and over-smoothed predictions during free-form generation31. Modern distillation frameworks, such as MiniLLM and Generalized Knowledge Distillation (GKD), abandon Forward KLD in favor of Reverse KLD31. The objective is formulated as: [Figure omitted from source export] Reverse KLD exhibits mode-seeking behavior31. Instead of attempting to memorize the entire distribution, the student concentrates its limited probability mass strictly around the dominant, most confident peaks of the teacher's output31. Because the expectation is calculated over trajectories generated by the student's own policy ([Figure omitted from source export]), the process operates essentially as on-policy reinforcement learning33. The teacher acts as a dense reward model, providing token-level log-probability feedback on the student's actual rollouts35. To stabilize this policy gradient optimization and prevent vanishing gradients in zero-advantage regions, algorithms implement teacher-mixed sampling—interpolating the sampling distribution between the teacher and the student ([Figure omitted from source export]) to prevent the student from exploiting degenerate reward paths—and length normalization to counter the reverse KL penalty's bias toward shorter sequences31. The resulting student model generates highly calibrated, precise responses that drastically outscore models trained on Forward KLD31.

Functional Geometry Transfer and Hidden-State Distillation

While logit-based distillation shapes the final output probabilities, hidden-state distillation forces the student's internal representations to mimic the teacher's reasoning pathways. Traditionally, matching intermediate activations (e.g., via Mean Squared Error) between models with differing hidden dimensions required the introduction of learnable linear projectors41. These projectors introduce noise, consume parameter budget, and implicitly assume that aligning representations geometrically equates to transferring semantic meaning44. The Flex-KD framework revolutionizes this process by shifting from representation matching to functional geometry transfer44. The relevance of a hidden dimension is dictated solely by its influence on the model's final output. Flex-KD computes the local sensitivity of the teacher's output with respect to perturbations in its hidden states, relying on a gradient-based estimation to identify a task-tangent subspace comprised of the most critical functional directions44. Instead of projecting the entire high-dimensional vector, Flex-KD extracts this low-dimensional, functionally dominant subspace from the teacher and distills it directly into the capacity-matched student44. This parameter-free approach allocates the student's constrained embedding bandwidth exclusively to task-relevant information. By eschewing learnable projectors, Flex-KD consistently outperforms linear projection baselines under severe dimension mismatch scenarios, particularly in generative tasks where rigid alignment often corrupts fluid language generation43.

Dataset Distillation and Tiny Instruction-Following Datasets

Distillation algorithms are only as effective as the calibration data utilized during the training phase. Dataset Distillation (DD) synthesizes massive, redundant pre-training corpora into highly concentrated, high-impact instruction-following datasets48. These datasets isolate the precise chain-of-thought (CoT) reasoning and stylistic alignment required by the edge application48. For instance, datasets like R1-Distill-SFT and OpenThoughts heavily condense the sprawling outputs of massive LLMs into minimal, high-utility reasoning traces48. When applied to LLMs, DD acts as a critical enabler for Knowledge Distillation; it identifies high-impact training examples that reflect the teacher's reasoning processes, guiding the student to learn efficiently without overfitting to redundant, low-quality data48. Furthermore, frameworks like PromptKD introduce soft prompt tuning into the generative distillation loop. By appending learnable abstract concept tokens to the teacher's input, the teacher is coerced into generating student-friendly knowledge49. This adaptive teaching mechanism lowers the perplexity of the target distribution, preventing the massive teacher from emitting esoteric or overly complex lexical structures that the SLM fundamentally cannot replicate49.

Phase 5: Sub-Byte Quantization and Initialization Dynamics

With the network pruned, factored, and distilled, the final architectural compression step involves quantization. However, standard Post-Training Quantization (PTQ) applied naively to a heavily sparsified model often leads to catastrophic accuracy degradation, as the global scaling factors become wildly skewed by the few remaining outliers in the network51. To safely deploy 2-bit or 4-bit precision formats, engineers must utilize quantization-aware initialization frameworks like LoftQ (LoRA-Fine-Tuning-aware Quantization)53. If standard QLoRA is applied to a heavily compressed model, the zero-initialized adapters are forced to waste their representational capacity correcting the massive quantization errors introduced in the base weights53. LoftQ mitigates this by alternating between quantizing the base weights and fitting the low-rank adapters, ensuring that the combination of the quantized base and the adapters starts as close as possible to the original full-precision weights53. This intelligent co-initialization shrinks the initial quantization gap dramatically, which is exactly what makes extreme 2-bit base models trainable in practice53. Furthermore, implementing Rank-Stabilized LoRA (scaling the adapter update by [Figure omitted from source export] instead of the traditional [Figure omitted from source export]) ensures that higher rank dimensions do not destabilize the training process, maximizing the efficiency of the parameter-efficient fine-tuning phase prior to final artifact export53.

Phase 6: Unstructured Sparsity and Edge Runtimes (Rust/WASM)

While structured pruning physically shrinks the computational graph, unstructured pruning zeros out individual weights randomly throughout the network56. Historically, unstructured sparsity was dismissed in production environments because AI accelerators like GPUs and TPUs rely on dense block processing; the irregular memory access patterns of unstructured zeros resulted in latency regressions despite theoretical FLOP reductions58. However, edge deployments governed by Central Processing Units (CPUs) and WebAssembly (WASM) dictate entirely different hardware-software economics.

The Resurgence of Sparse CPU Kernels

Modern consumer CPUs, ARM processors, and edge architectures execute scalar and narrow SIMD (Single Instruction, Multiple Data) instructions (e.g., AVX-512, AMX)58. Advanced software implementations have demonstrated that unstructured sparsity translates directly into massive real-world speedups on these architectures. Algorithms like the Sparsity-Aware Vector Engine (SAVE) and the SparseProp backpropagation framework dynamically detect zeros in dense representations and skip ineffectual computations at the vector lane level57. Libraries such as XNNPACK provide highly optimized Sparse Matrix-Vector Multiplication (SpMV) kernels targeted for ARM and WASM64. By keeping non-zero values pinned in the L1 cache, vectorizing the dense activation matrices, and prefetching input channels, models exhibiting 70% to 95% unstructured sparsity achieve 1.3x to 2.4x inference speedups on edge devices64. This proves that unstructured sparsity, when paired with cache-aware architectural modifications, is a highly viable mechanism for pushing SLM performance boundaries in resource-constrained environments64.

Rust and WASM AI Ecosystems

The deployment of .slm and .gguf artifacts requires a runtime that balances raw performance with aggressive sandboxing and minimal cold-start latency. Python-based runtimes inherently suffer from the Global Interpreter Lock (GIL), heavy standard libraries, and severe garbage collection overhead, disqualifying them from edge sensor or browser-based execution. Rust has emerged as the definitive systems language for machine learning at the edge, offering zero-cost abstractions, memory safety without garbage collection, and seamless compilation to bare-metal WebAssembly70. Two premier frameworks dominate this ecosystem, each catering to different operational paradigms:

At the virtualization layer, WasmEdge acts as a high-performance runtime tailored for cloud-native and edge AI76. Utilizing the WASI-NN (WebAssembly System Interface for Neural Networks) standard, WasmEdge executes quantized .gguf models directly from Rust-compiled WASM binaries77. The combination of memory-safe Rust logic and WasmEdge's optimized backend allows edge architectures to run hybrid pipelines—where a tiny routing model dynamically parses unstructured data and hands it off to a distilled SLM for structured generation—all within an isolated, low-power sandbox78.

Phase 7: The Practical .slm Compression Pipeline

Integrating these advanced theoretical concepts requires a rigorously ordered compilation pipeline. Executing compression steps out of sequence will result in catastrophic error accumulation and the irreversible destruction of the model's latent representation. The optimal methodology for generating a resilient .slm model dictates the following progression:

1. Profiling and Importance Scoring: Before any modifications are made, the model undergoes profiling against a calibration dataset to compute Block Influence (BI) for depth redundancy and Taylor-First-Order sensitivity for width redundancy13.
2. Vocabulary Trimming and Embedding Shrinkage: The target domain tokens are identified, and the tokenizer is reconstructed. The corresponding rows in the embedding and LM heads are deleted, and out-of-vocabulary mappings are established via subword averaging1. This must occur before any structural pruning to prevent downstream alterations from adapting to embeddings that will ultimately be deleted5.
3. Structured Pruning (Depth and Width): High-BI transformer blocks are dropped entirely13. For the remaining blocks, RMSNorm orthogonal transformations are applied, and the lowest-variance components are sliced off via the SliceGPT methodology18. Object-DINO clustering excises redundant attention heads23.
4. Low-Rank Factorization: The model computes a Cholesky whitening matrix to create an orthonormal activation space27. SVD truncates the lowest singular values, followed by layer-wise Ridge Regression healing to smooth out the remaining internal redundancies21.
5. Teacher-Student Distillation: The massive architectural damage is repaired by transferring task-tangent hidden states via Flex-KD46. Concurrently, Reverse KLD is optimized via on-policy rollouts (MiniLLM) across a tiny, high-yield dataset distilled from a larger teacher model33.
6. Sub-Byte Quantization: The network applies LoftQ to co-initialize the quantized base weights with low-rank adapters, preventing quantization errors from overriding the distilled logic53. The model undergoes a final phase of Rank-Stabilized QLoRA fine-tuning55.
7. Serialization and Export: The final structured graphs, modified tokenizers, and generated unstructured sparsity masks are packed into a .slm or .gguf binary, heavily optimized for execution via WasmEdge, Candle, or Burn runtimes72.

Phase 8: Collapse Diagnostics and Metadata Verification

Aggressive structural reduction introduces severe risks. Over-pruning inevitably leads to representation collapse, characterized by activation variance shrinkage82. As the network's capacity decreases, the model loses the ability to distinguish between nuanced latent concepts, folding distinct semantic representations into a singular, homogeneous state. This damages polysemantic features—neurons that encode multiple distinct concepts—causing the model's zero-shot reasoning to fail catastrophically even if basic language fluency is maintained21. Furthermore, if the distillation phase utilizes standard Forward KL or operates over insufficient calibration data, the student model will suffer from mode-averaging. The model will begin generating repetitive, safe, and highly probabilistic text, entirely stripping away the linguistic diversity and complex chain-of-thought capabilities inherited from the teacher34. To detect and prevent quality collapse prior to deployment, the engineering framework must enforce rigorous diagnostic testing:

For a .slm artifact to be viable in decentralized or highly regulated edge environments, it must carry embedded cryptographic and structural metadata. The binary must store the exact retained sub-vocabulary mapping to guarantee tokenizer symmetry. It must include the specific Block Influence threshold targeted during layer dropping, alongside the Reverse KL divergence decay curves achieved during the MiniLLM distillation phase. For WASM CPU deployments, unstructured sparsity masks and tile sizes (specifically optimized for AVX-512 or ARM SIMD instructions) must be explicitly defined in the file header58. Storing this evaluation proof guarantees that downstream runtimes can initialize the correct sparse matrix multiplication kernels and verify the model's structural integrity before committing inference cycles.

Works cited

1. Efficient Vocabulary Reduction for Small Language Models \- ACL Anthology, https://aclanthology.org/2025.coling-industry.64.pdf
2. VocabTailor: Dynamic Vocabulary Selection for Downstream Tasks in Small Language Models \- arXiv, https://arxiv.org/html/2508.15229v2
3. VocabTailor: Dynamic Vocabulary Selection for Downstream Tasks in Small Language Models \- arXiv, https://arxiv.org/html/2508.15229v1
4. VocabTailor: Dynamic Vocabulary Selection for Downstream Tasks in Small Language Models \- ACL Anthology, https://aclanthology.org/2026.findings-acl.1418.pdf
5. Distillation in 18 days vs. CPU in 9 minutes: The full story of 'Trimming' to reduce vocabulary size by 40% without retraining \- note, https://note.com/snake\_dragon/n/n4685fd43f8b1?hl=en
6. Introduction to Trimming \- Hugging Face, https://huggingface.co/blog/lbourdois/introduction-to-trimming
7. SpeLLM: Input Tokens, Output Chars \- arXiv, https://arxiv.org/html/2507.16323v1
8. Getting LLMs To Talk To Each Other | NTT STORY, https://group.ntt/en/magazine/blog/token-standardization/
9. NTT Establishes World's First Framework for Lossless Vocabulary Reduction Across LLMs \-- Enables seamless cooperation and knowledge transfer across heterogeneous models \- NTT Group, https://group.ntt/en/newsrelease/2026/04/22/260422a.html
10. Lossless Vocabulary Reduction for Auto-Regressive Language Models \- OpenReview, https://openreview.net/forum?id=xAvqHtLVgz
11. NTT Breaks Vocabulary Barrier Between Heterogeneous AI Language Models | IBTimes JP, https://jp.ibtimes.com/ntt-breaks-vocabulary-barrier-between-heterogeneous-ai-language-models-100659
12. SHORTGPT: LAYERS IN LARGE LANGUAGE MODELS ARE MORE REDUNDANT THAN YOU EXPECT \- OpenReview, https://openreview.net/pdf?id=JMNht3SmcG
13. ShortGPT: Layers in Large Language Models are More Redundant Than You Expect \- arXiv, https://arxiv.org/html/2403.03853v1
14. EVOPRESS: ACCURATE DYNAMIC MODEL COMPRES- SION VIA EVOLUTIONARY SEARCH \- OpenReview, https://openreview.net/pdf?id=QhW8k5Ph77
15. How Layer Dropping Speeds Up LLM Inference \- Newline, https://www.newline.co/@zaoyang/how-layer-dropping-speeds-up-llm-inference--5f671d14
16. INTERLACE: Interleaved Layer Pruning and Efficient Adaptation in Large Vision-Language Models, https://openaccess.thecvf.com/content/CVPR2026F/papers/Madinei\_INTERLACE\_Interleaved\_Layer\_Pruning\_and\_Efficient\_Adaptation\_in\_Large\_Vision-Language\_CVPRF\_2026\_paper.pdf
17. slicegpt: compress large language models \- arXiv, https://arxiv.org/pdf/2401.15024
18. Compressing Large Language Models: Introducing SliceGPT \- Origins AI, https://originshq.com/blog/compressing-large-language-models-introducing-slicegpt/
19. SliceViT | CS231n \- Stanford University, https://cs231n.stanford.edu/2024/papers/slicevit.pdf
20. A REVIEW ON SLICEGPT: COMPRESS LARGE LANGUAGE MODELS BY DELETING ROWS AND COLUMNS | by Adeyemooluwatobi | Medium, https://medium.com/@adeyemooluwatobi2/a-review-on-slicegpt-compress-large-language-models-by-deleting-rows-and-columns-7883966035cb
21. (PDF) Model Minification: A Taylor-Ridge Framework for Structured Compression, https://www.researchgate.net/publication/399829361\_Model\_Minification\_A\_Taylor-Ridge\_Framework\_for\_Structured\_Compression
22. liyunqianggyn/Awesome-LLMs-Pruning \- GitHub, https://github.com/liyunqianggyn/Awesome-LLMs-Pruning
23. Structured Pruning of Large Language Models \- ResearchGate, https://www.researchgate.net/publication/347235497\_Structured\_Pruning\_of\_Large\_Language\_Models
24. svd-llm: truncation-aware singular value decomposition for large language model compression, https://par.nsf.gov/servlets/purl/10647136
25. SVD-LLM V2: Optimizing Singular Value Truncation for Large Language Model Compression, https://par.nsf.gov/servlets/purl/10647140
26. svd-llm: truncation-aware singular value decomposition for large language model compression \- ICLR Proceedings, https://proceedings.iclr.cc/paper\_files/paper/2025/file/3104e1ab39875cf54fe1eb4473e7c5a1-Paper-Conference.pdf
27. JW-SVD: Bridging the Cross-Modal Mismatch in Post-Training MLLM Compression \- ACL Anthology, https://aclanthology.org/2026.acl-long.1977.pdf
28. SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression \- arXiv, https://arxiv.org/html/2403.07378v5
29. \[Literature Review\] SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression \- Moonlight, https://www.themoonlight.io/en/review/svd-llm-truncation-aware-singular-value-decomposition-for-large-language-model-compression
30. SVD-LLM V2: Optimizing Singular Value Truncation for Large Language Model Compression \[Quick Review\] \- Liner, https://liner.com/review/svdllm-v2-optimizing-singular-value-truncation-for-large-language-model
31. MiniLLM: Efficient LLM Distillation \- Emergent Mind, https://www.emergentmind.com/topics/minillm
32. On LLM Knowledge Distillation \- A Comparison between Forward KL and Reverse KL, https://d2jud02ci9yv69.cloudfront.net/2025-04-28-llm-knowledge-distil-157/blog/llm-knowledge-distil/
33. MiniLLM: On-Policy Distillation of Large Language Models \- arXiv, https://arxiv.org/html/2306.08543v6
34. Hybrid Policy Distillation for LLMs \- OpenReview, https://openreview.net/pdf?id=eFBO8ECxb2
35. Aikipedia: On-Policy Distillation \- Champaign Magazine, https://champaignmagazine.com/2025/10/29/aikipedia-on-policy-distillation/
36. chrisliu298/awesome-on-policy-distillation \- GitHub, https://github.com/chrisliu298/awesome-on-policy-distillation
37. On-Policy Distillation \- Thinking Machines Lab, https://thinkingmachines.ai/blog/on-policy-distillation/
38. Unmasking On-Policy Distillation: Where It Helps, Where It Hurts, and Why \- arXiv, https://arxiv.org/html/2605.10889v1
39. Asymmetric On-Policy Distillation: Bridging Exploitation and Imitation at the Token Level, https://arxiv.org/html/2605.06387v3
40. MiniLLM: Knowledge Distillation of Large Language Models \- arXiv, https://arxiv.org/html/2306.08543v3
41. DistillKit v0.1 by Arcee Labs: The Technical Paper, https://www.arcee.ai/blog/distillkit-v0-1-by-arcee-ai
42. Knowledge Distillation: Compress 671B Models to 7B (2026) | Local AI Master, https://localaimaster.com/blog/knowledge-distillation-guide
43. Task-Based Flexible Feature Distillation for LLMs \- arXiv, https://arxiv.org/html/2507.10155v1
44. What Should Feature Distillation Transfer in LLMs? A Task-Tangent Geometry View \- arXiv, https://arxiv.org/html/2507.10155v3
45. What Should Feature Distillation Transfer in LLMs? A Task-Tangent Geometry View \- arXiv, https://arxiv.org/abs/2507.10155
46. Flexible Feature Distillation for Large Language Models \- OpenReview, https://openreview.net/forum?id=aiMINHhIiQ
47. FLEXIBLE FEATURE DISTILLATION FOR LARGE LAN- GUAGE MODELS \- OpenReview, https://openreview.net/pdf?id=aiMINHhIiQ
48. Knowledge distillation and dataset distillation of large language models: emerging trends, challenges, and future directions \- PMC, https://pmc.ncbi.nlm.nih.gov/articles/PMC12634706/
49. PromptKD: Distilling Student-Friendly Knowledge for Generative Language Models via Prompt Tuning, https://promptkd.github.io/
50. Daily Papers \- Hugging Face, https://huggingface.co/papers?q=soft%20abstract%20tokens
51. HQP: Sensitivity-Aware Hybrid Quantization and Pruning for Ultra-Low-Latency Edge AI Inference \- ResearchGate, https://www.researchgate.net/publication/400583634\_HQP\_Sensitivity-Aware\_Hybrid\_Quantization\_and\_Pruning\_for\_Ultra-Low-Latency\_Edge\_AI\_Inference
52. CLoQ: Enhancing Fine-Tuning of Quantized LLMs via Cali- brated LoRA Initialization \- NSF Public Access Repository, https://par.nsf.gov/servlets/purl/10630233
53. PEFT Beyond QLoRA: DoRA, GaLore & LoftQ Compared \- AppScale Blog, https://appscale.blog/en/blog/parameter-efficient-fine-tuning-peft-lora-qlora-dora-galore-2026
54. LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Models \- arXiv, https://arxiv.org/html/2310.08659
55. Mastering Low-Rank Adaptation (LoRA): The Ultimate Guide to Efficient Fine-Tuning of Large Language Models \- Prasun Maity, https://prasunmaity.medium.com/mastering-lora-the-ultimate-guide-to-efficient-llm-fine-tuning-5c9de67e7fe2
56. The LLM Inference Optimization: Quantization to Speculative Decoding Part 1 | DigitalOcean, https://www.digitalocean.com/community/tutorials/llm-inference-optimization-stack-part-1
57. SparseProp: Efficient Sparse Backpropagation for Faster Training of Neural Networks at the Edge \- Proceedings of Machine Learning Research, https://proceedings.mlr.press/v202/nikdan23a/nikdan23a.pdf
58. 1 Introduction \- arXiv, https://arxiv.org/html/2502.12444v1
59. Sparser, Faster, Lighter Transformer Language Models \- arXiv, https://arxiv.org/html/2603.23198v2
60. Samoyeds: Accelerating MoE Models with Structured Sparsity Leveraging Sparse Tensor Cores \- arXiv, https://arxiv.org/html/2503.10725v1
61. SAVE: Sparsity-Aware Vector Engine for Accelerating DNN Training and Inference on CPUs, https://www.microarch.org/micro53/papers/738300a796.pdf
62. "A SIMD SPARSE MATRIX-VECTOR MULTIPLICATION ALGORITHM FOR COMPUTATIONAL" by Nirav Harish Kapadia \- Purdue e-Pubs, https://docs.lib.purdue.edu/ecetr/200/
63. Computing the sparse matrix vector product using block-based kernels without zero padding on processors with AVX-512 instructions \- PMC, https://pmc.ncbi.nlm.nih.gov/articles/PMC7924463/
64. Fast Sparse ConvNets \[Quick Review\] \- Liner, https://liner.com/review/fast-sparse-convnets
65. SAVE: Sparsity-Aware Vector Engine for Accelerating DNN Training and Inference on CPUs \- i-acoma, https://iacoma.cs.uiuc.edu/iacoma-papers/micro20\_3.pdf
66. Exploiting and Coping with Sparsity to Accelerate DNNs on CPUs \- i-acoma, https://iacoma.cs.uiuc.edu/iacoma-papers/FINAL-GONG-DISSERTATION-2021.pdf
67. Adaptive Hybrid Storage Format for Sparse Matrix–Vector Multiplication on Multi-Core SIMD CPUs \- MDPI, https://www.mdpi.com/2076-3417/12/19/9812
68. A Systematic Literature Survey of Sparse Matrix-Vector Multiplication \- arXiv, https://arxiv.org/html/2404.06047v1
69. Efficient Sparsity for Deep Learning \- AHA Agile Hardware Project, https://aha.stanford.edu/sites/g/files/sbiybj20066/files/media/file/aha-retreat-2022\_gale\_efficient-sparsity-for-deep-learning\_update\_0.pdf
70. Evaluating and Improving Automated Repository-Level Rust Issue Resolution with LLM-based Agents \- arXiv, https://arxiv.org/html/2602.22764v1
71. LLMs running in the browser | Kevin Scott, https://thekevinscott.com/llms-in-the-browser/
72. candle\_core \- Rust \- Docs.rs, https://docs.rs/candle-core/
73. What makes Rust the best language for Deep Learning \- Reddit, https://www.reddit.com/r/rust/comments/1bixnh4/what\_makes\_rust\_the\_best\_language\_for\_deep/
74. ML Library Comparison: Burn vs Candle : r/rust \- Reddit, https://www.reddit.com/r/rust/comments/1op3ad1/ml\_library\_comparison\_burn\_vs\_candle/
75. From Julia to Rust: a differentiable tensor stack for scientific computing in the agentic AI era \- tensor4all, https://tensor4all.org/blog/introducing-tenferro-rs/
76. qijianpeng/awesome-edge-computing: A curated list of awesome edge computing, including Frameworks, Simulators, Tools, etc. \- GitHub, https://github.com/qijianpeng/awesome-edge-computing
77. LLM inference | WasmEdge Developer Guides, https://wasmedge.org/docs/develop/rust/wasinn/llm\_inference/
78. Effortless JSON Generation with Osmosis‑Structure‑0.6B \- secondstate.io, https://www.secondstate.io/articles/osmosis-structure-0.6b/
79. Getting started with OpenAI's gpt-oss \- secondstate.io, https://www.secondstate.io/articles/openai-gpt-oss/
80. llms-full.txt \- Gaianet.ai, https://docs.gaianet.ai/llms-full.txt
81. LFX Workspace: Rust Coder · Issue \#4038 \- GitHub, https://github.com/WasmEdge/WasmEdge/issues/4038
82. Daily Papers \- Hugging Face, https://huggingface.co/papers?q=activation%20variance%20shrinkage
83. ASKD: Reinforcement Learning-Style Knowledge Distillation with Quality-Adaptive Skewness, https://ojs.aaai.org/index.php/AAAI/article/view/40780/44741