Advanced Structural Compression Methodologies for TinyRustLM and .slm Edge Deployments

Full report

On this page

The deployment of Large Language Models (LLMs) in highly constrained edge environments, particularly browser-based WebAssembly (WASM) and WebGPU targets, imposes strict limitations on computational latency, memory bandwidth, and storage capacity. While conventional post-training quantization to formats such as 8-bit or 4-bit integer representations significantly reduces the memory footprint of weight matrices, it fundamentally fails to decrease the actual number of active parameters or the total volume of sequential mathematical operations required during the forward pass. To push beyond the theoretical limits of pure quantization and achieve true edge viability for the TinyRustLM architecture, structural compression techniques must be aggressively employed. These techniques surgically alter the foundational topology of the transformer network, reducing the active compute overhead before quantization is even applied. This comprehensive analysis systematically evaluates the entire spectrum of structural compression techniques—encompassing layer dropping, advanced attention head pruning, feed-forward network (FFN) neuron reduction, low-rank factorization, grouped-query attention conversion, and vocabulary tying. Furthermore, it establishes a definitive end-to-end compression pipeline tailored for the proprietary .slm (v1) deployment format, defines stringent runtime compatibility and validation protocols, calculates precise hardware memory impacts, and architects a highly resilient, semantically grounded benchmark design to prevent performance misrepresentation.

1. Top-Down Structural Topology Reductions

1.1 Layer Dropping and Dynamic Depth Reduction

The traditional transformer architecture exhibits a high degree of over-parameterization, particularly within its depth. Recent investigations into the redundancy of transformer layers have uncovered a phenomenon characterized as the unreasonable ineffectiveness of deeper layers1. Because the transformer relies heavily on residual connections, the output of the final layer is essentially an accumulation of transformations applied by all preceding layers3. Consequently, deeper layers frequently process highly overlapping representational spaces and exhibit profound qualitative similarity to their immediate neighbors, rendering them highly susceptible to safe excision. Layer dropping, or depth pruning, capitalizes on this redundancy by entirely removing specific transformer blocks—each comprising a self-attention module and a corresponding MLP module. The selection of layers for removal is typically governed by the Block Influence (BI) score, a sophisticated metric that calculates the cosine similarity between the input and output hidden states of each layer over a representative calibration dataset1. Layers exhibiting cosine similarities approaching unity introduce negligible transformations to the feature space and are flagged for deletion. Reverse-order pruning, which sequentially removes layers starting from the penultimate layer and moving backward, has proven remarkably effective due to the compounding representational stability found in the deepest network segments1. However, coarse-grained block dropping is often suboptimal because it enforces the simultaneous deletion of both the attention and MLP submodules. Extensive profiling reveals a stark asymmetry in submodule redundancy: pruning attention layers yields substantial acceleration and alleviates Key-Value (KV) cache memory overhead with minimal accuracy degradation, whereas dropping MLP layers catastrophically impairs the model's ability to distinguish between tokens5. Advanced implementations for TinyRustLM must therefore decouple depth pruning, dropping redundant attention modules while preserving the integrity of the knowledge-dense FFN blocks. To mitigate the inevitable representation mismatch introduced by structural deletion, a parameter-efficient healing phase is absolutely critical. This is achieved by freezing the remaining pruned network and performing a brief fine-tuning pass using Low-Rank Adaptation (LoRA) or QLoRA, which restores the severed sequential data flow at a fraction of the computational cost of full retraining1. For highly advanced deployments, dynamic layer dropping frameworks like SkipGPT train a lightweight, BERT-based router that evaluates the input prompt at runtime and dynamically determines the optimal combination of layers to execute, adapting the computational depth to the complexity of the specific user query1.

1.2 Attention Head Pruning via Activation and Entropy Dynamics

While depth reduction targets the sequential length of the model, width reduction targets parallel capacity. Attention head pruning specifically excises individual heads within the Multi-Head Attention (MHA) modules. This procedure directly reduces the massive tensor contractions required for query-key mapping and value aggregation, yielding proportional reductions in both FLOPs and KV-cache allocation8. The historical approach to head pruning relies on Head Importance Scores (HIS) derived from a first-order Taylor expansion of the loss function. This gradient-based metric ranks heads by their expected contribution to the global loss minimization8. Unfortunately, relying exclusively on single-sided gradient sensitivity ignores the long-tail distribution characteristics inherent to attention mechanisms, where a minute fraction of highly active tokens dominates the probability mass during complex reasoning tasks9. Utilizing standard HIS or purely magnitude-based metrics like Wanda frequently flattens these critical attention distributions, leading to sharp accuracy cliffs when the pruning ratio exceeds a moderate threshold8. To ensure robust performance in .slm targets, two highly sophisticated methodologies must govern the pruning selection: Firstly, the Dual Taylor Expansion framework must be utilized. Unlike conventional metrics that assume a static activation distribution and only model weight perturbations, Dual Taylor simultaneously models the impacts of both weight perturbations and activation variations between the calibration data and real-world deployment data9. By capturing the joint effect of input variability and structural sparsity, the algorithm accurately preserves the high-selectivity, long-tail assignments crucial for factual recall, reducing the Kullback-Leibler (KL) divergence of the pruned attention distribution by over sixty percent compared to single-variable approaches10. Secondly, Head Importance-Entropy Scoring (HIES) provides an indispensable secondary validation. HIES quantifies the concentration of each attention head's focus over input tokens by calculating the Shannon entropy of its attention weight distribution8. Higher entropy denotes a diffuse, generalized focus, whereas lower entropy indicates a highly concentrated attention pattern. Empirical analysis establishes a strong correlation between persistent entropy collapse—where an attention head's entropy remains artificially depressed—and systemic training instability8. Pruning algorithms integrating HIES prioritize the removal of these unstable, collapsed heads, alongside highly redundant diffuse heads, preserving the critical middle-variance heads that drive dynamic reasoning.

1.3 FFN Neuron Pruning and MLP Rank Reduction

The Multi-Layer Perceptron (MLP) modules within a transformer are generally understood to function as distributed key-value memories, storing the vast majority of the network's factual knowledge. Consequently, MLP modules are heavily over-parameterized, often accounting for more than two-thirds of the total parameters in a standard dense architecture12. Evaluation of Taylor-based importance scores across modern LLMs reveals that average parameter importance within MLP modules is substantially lower than within attention modules, indicating that FFN neuron pruning represents the most resource-efficient vector for massive parameter reduction12. Executing FFN pruning effectively requires addressing the granular limitations of standard cross-entropy loss criteria. Conventional pruning evaluates neuron importance based on a one-hot label encoding, which severely penalizes structural changes that alter the primary prediction but completely ignores the preservation of the secondary, potential vocabulary distribution12. To maintain the generative fidelity and semantic depth of the pruned model, an Information Entropy criterion must be deployed. This label-free metric calculates importance by measuring how the removal of a specific FFN neuron alters the total information entropy of the model's global prediction distribution, thus ensuring the preservation of the model's comprehensive reasoning capabilities across the entire vocabulary space12. Furthermore, FFN pruning relies on the Hybrid-grained Weight Importance Assessment (HyWIA), which continuously modulates between fine-grained, individual-weight assessments and coarse-grained, neuron-level assessments to identify optimal structural removal patterns14. Because the removal of knowledge-bearing neurons introduces massive distribution shifts in the forward activations, FFN layers must be pruned strictly sequentially rather than globally. Sequential pruning algorithms, such as Putri, finalize the sparsity mask for a single layer, execute a localized weight update to compensate for the induced error, and only then proceed to calculate importance scores for the subsequent layer15. This iterative calibration ensures that subsequent FFN modules dynamically adjust to the altered representations of their predecessors, enabling successful function even at extreme sparsity ratios approaching ninety percent.

2. Advanced Matrix Optimizations and Hardware Alignment

2.1 Low-Rank Factorization of Projection Matrices

When entire neurons or attention heads cannot be safely removed, the internal dimensionality of the surviving projection matrices can be compressed utilizing low-rank factorization. This technique decomposes a massive, computationally expensive weight matrix into the sequential product of two vastly smaller matrices, trading a marginal degree of approximation error for massive savings in storage and arithmetic complexity16. Singular Value Decomposition (SVD) provides the mathematically optimal low-rank approximation by truncating the smallest singular values. However, naive application of SVD to LLM weights yields catastrophic degradation. This occurs because modern transformer activations exhibit extreme outliers—specific features that are magnitude orders larger than the mean—which dominate the arithmetic operations but are inadvertently discarded by standard spectral truncation18. To successfully apply low-rank factorization for TinyRustLM, specialized pipelines are mandated:

1. Activation-Aware SVD (ASVD): The algorithm mitigates activation outliers by deeply integrating activation distribution patterns directly into the weight decomposition process. The foundational weight matrix is multiplied by a highly specific diagonal scaling matrix derived from the absolute mean or Fisher information of the input channels, forcing the SVD computation to allocate enhanced mathematical focus to weights associated with massive activation spikes18.
2. SVD-LLM and Truncation-Aware Data Whitening: Building upon activation awareness, the SVD-LLM protocol utilizes a Cholesky decomposition of the activation covariance matrix to perform truncation-aware data whitening. This mathematically guarantees a direct, linear mapping between the singular values selected for truncation and the resulting compression loss21. Recent advancements, denoted as SVD-LLM V2, utilize this theoretical truncation loss to dynamically assign highly heterogeneous compression ratios across the network. Instead of applying a flat global rank reduction, the algorithm aggressively compresses redundant intermediate layers while preserving the rank of highly sensitive initial and terminal blocks22.
3. Trainable Rank Decomposition Matrices (EDoRA): An alternative to purely post-hoc SVD is the integration of trainable low-rank factors. Frameworks like Efficient Weight-Decomposed Low-Rank Adaptation (EDoRA) perform a truncated SVD, freeze the principal subspace matrices, and inject a small, highly trainable core matrix into the center of the decomposition24. This allows the model to dynamically tune its compressed representation during the final healing phase, drastically shrinking the adaptation footprint while matching the accuracy of dense fine-tuning17.

2.2 Multi-Head to Grouped-Query Attention Conversion

The primary bottleneck for high-throughput, long-context inference in autoregressive language models is not arithmetic logic unit (ALU) saturation, but rather memory bandwidth exhaustion caused by the Key-Value (KV) cache25. Standard Multi-Head Attention (MHA) mandates the storage of distinct [Figure omitted from source export] and [Figure omitted from source export] vectors for every single query head. The memory complexity for the KV cache scales as [Figure omitted from source export], where [Figure omitted from source export] is batch size, [Figure omitted from source export] is the number of heads, [Figure omitted from source export] is sequence length, and [Figure omitted from source export] is the embedding dimension per head27. Grouped-Query Attention (GQA) dramatically reduces this overhead by partitioning the [Figure omitted from source export] query heads into [Figure omitted from source export] discrete groups. Within each group, all query heads attend to a single, shared [Figure omitted from source export] and [Figure omitted from source export] projection, reducing the KV cache footprint by a massive factor of [Figure omitted from source export]26. To avoid the astronomical costs of pre-training a GQA model from scratch, pre-trained MHA models are structurally converted post-training. The baseline conversion technique involves naively mean-pooling the existing [Figure omitted from source export] and [Figure omitted from source export] projection matrices within a designated group28. However, this uniformly destructive averaging severely degrades performance because distinct attention heads frequently learn entirely orthogonal feature representations; combining them blindly destroys the mathematical signal30. For precision deployment, the Align-GQA protocol must govern the conversion. The algorithm executes a Generalized Procrustes Analysis on the attention heads, mapping the covariance between the key and value caches across the calibration dataset30. An optimal orthogonal transformation matrix is computed via Singular Value Decomposition, which maximally aligns the internal feature spaces of the targeted heads without altering the fundamental computational invariance of the network6. Only after the feature spaces are mathematically rotated into perfect alignment are the projection matrices merged. Following the merger, [Figure omitted from source export] regularization is utilized to smoothly prune out any residual parametric redundancy6. Advanced variations, such as Decoupled-Head Attention (DHA) and AsymGQA, further refine this process by utilizing adaptive budgets to determine dynamically which heads exhibit sufficient semantic overlap to warrant grouping, creating asymmetric group sizes that maximize hardware efficiency while preserving critical reasoning pathways26.

2.3 Embedding/Output-Head Tying and Vocabulary Trimming

In sub-billion parameter models characteristic of the TinyRustLM architecture, the vocabulary-related components—specifically the input token embedding matrix and the final pre-softmax language modeling (LM) projection head—consume an immensely disproportionate percentage of total memory capacity, often exceeding thirty percent of the entire parameter budget35. To drastically curtail this overhead, vocabulary trimming (also conceptualized as lexical shortlisting) algorithmically reduces the breadth of the token vocabulary. Multilingual foundational models harbor vast vocabularies—frequently exceeding 128,000 or 250,000 tokens—to process obscure languages and diverse scripts36. For highly targeted .slm deployments, this represents dead weight. Vocabulary trimming employs sophisticated language heuristics, ranging from strict Unicode-based script filtering to empirical token frequency tracking over a massive calibration corpus, to identify and isolate irrelevant tokens38. Excising these dormant dimensions from both the embedding and the output projection matrices reduces their memory footprint by up to fifty percent37. Furthermore, memory efficiency is compounded by enforcing strict weight-tying between the embedding and the LM output head. Instead of maintaining two massive, independent matrices, the architecture leverages the inherent symmetry of the transformations, forcing the output linear layer to share the exact memory pointers of the input embedding matrix35. To overcome the slight loss of representational capacity induced by this symmetry, the Error Correcting Output Codes (ECOC) framework can be deployed within the decision layer. ECOC converts the standard high-dimensional logit output into a hyper-compressed codeword, calculating probability distributions based on distance metrics to predefined vocabulary codewords, thereby compressing the required output parameters by an astounding ninety-nine percent while retaining acceptable accuracy40.

2.4 Sparse Tensor Storage Formats and Hardware Constraints

The theoretical optimization of deep neural networks frequently champions fine-grained, unstructured sparsity—where parameters are independently zeroed based on magnitude or gradient sensitivity. While this preserves maximum model accuracy during high-ratio pruning, it is catastrophically incompatible with the dense hardware execution models utilized by browser-based WebAssembly (WASM), WebGPU, and CPU SIMD instructions25. Modern execution environments rely heavily on vectorization, processing contiguous blocks of data simultaneously. In WASM Relaxed SIMD and WebGPU architectures, threads execute synchronously in warps or subgroups. Unstructured sparsity creates a completely random distribution of non-zero elements, resulting in highly irregular memory access patterns25. When the hardware encounters unstructured sparsity, it suffers from severe barrier divergence, memory bank conflicts, and pipeline starvation. The CPU wastes vast quantities of memory bus bandwidth streaming dense blocks of zeros from DRAM to the cache, ultimately causing the sparse model to execute significantly slower than its dense, unpruned counterpart25. Consequently, TinyRustLM inference mandates the exclusive use of block structured sparsity. Hardware-aware algorithms enforce sparsity at strictly defined intervals that align perfectly with the boundaries of 128-bit or 256-bit SIMD registers41. The 8:16 semi-structured sparsity pattern has emerged as the definitive standard for this architecture. By mandating that exactly eight parameters within every contiguous block of sixteen be pruned, the format offers superior representational flexibility compared to legacy 2:4 sparsity, while guaranteeing the highly predictable, coalesced memory access required by SIMD compute units41. To harness this at runtime without relying on bloated C++ backends, the TinyRustLM ecosystem relies on specialized Rust crates analogous to numr. These libraries provide highly optimized, zero-dependency implementations of Block Sparse Row (BSR) and Compressed Sparse Row (CSR) tensor layouts46. By leveraging fused matrix-multiplication-activation kernels (GEMM epilogues) written in bare-metal Rust and WebAssembly, the runtime maps the 8:16 sparse layouts directly into FP8 (E4M3/E5M2) or Int8 quantized sub-byte instructions, completely sidestepping the massive processing overhead historically associated with browser-based inference42.

3. The .slm Container Format and Security Obfuscation

3.1 Custom Container Architecture

The proprietary .slm (v1) format represents a massive departure from standard serialization frameworks like Safetensors or GGUF. Designed explicitly for extreme low-latency initialization within Javascript and WASM wrappers, it completely eliminates heavy parsing dependencies. The container utilizes a highly rigid, custom little-endian binary layout beginning with a strictly enforced 108-byte header, followed immediately by an embedded BTOK or BPE tokenizer section49. This is followed by a contiguous 64-byte hashed tensor directory mapping the byte offsets, precise tensor shapes, and quantization datatypes (e.g., f32, q8\_0, q4\_0) of every structurally compressed layer in the network49.

3.2 Encoding Structural Changes via Header Obfuscation

Deploying a meticulously compressed and mathematically aligned language model to an untrusted client browser introduces significant intellectual property risks. Competitors can effortlessly download the .slm asset and reverse-engineer the precise proprietary structural optimizations, such as the exact ASVD scaling matrices, the asymmetric GQA grouping layouts, or the specific sequence of pruned FFN neurons51. Because WebAssembly and WebGPU targets demand nearly instantaneous initialization upon page load, wrapping the model in heavy cryptographic protocols (such as AES-256) is computationally prohibitive. Instead, the framework secures proprietary conversion internals utilizing a sophisticated, multi-technique obfuscation framework executed directly within the file header51.

1. Padding and Shifting: The 108-byte header and the 64-byte tensor directory contain reserved, zero-initialized padding fields. During compilation, the tinyrustlm-slm-pack utility injects arbitrary, constant-size byte padding into these regions and executes a systemic byte-shift across all directory offsets. This immediately breaks standard static analysis tools and hex-editors attempting to map the file structure heuristically51.
2. Deterministic XOR Obfuscation: The critical metadata defining the structural sparsity masks, dropped layer indices, and GQA routing configurations is isolated and subjected to an XOR obfuscation protocol. A highly robust pseudo-random sequence is generated from a proprietary seed embedded discreetly within the .slm magic bytes. The structural metadata is bitwise XORed against this sequence51.

During execution, the Rust WASM runtime extracts the seed from the magic sequence, reconstructs the pseudo-random stream, and performs a single-cycle XOR reversal to materialize the correct Directed Acyclic Graph (DAG) for the compute path. This provides a formidable defense against unauthorized traffic analysis and model extraction while adding virtually zero latency to the critical initialization path51.

4. Engineering Deliverables

4.1 End-to-End Compression Pipeline for TinyRustLM

The transformation of a massive, dense Hugging Face artifact into a highly optimized, WASM-ready .slm binary requires a rigorous, multi-stage pipeline:

1. Ingestion and Data Profiling: The baseline model is instantiated alongside a domain-specific calibration dataset. Activation distributions are meticulously recorded to capture extreme outlier events across all hidden dimensions.
2. Vocabulary Optimization: Top\-[Figure omitted from source export] frequency analysis or Unicode script filtering is executed against the calibration corpus. Irrelevant tokens are discarded, the embedding matrix is truncated, and the output pre-softmax projection is completely excised and tied to the input embedding memory pointers35.
3. Align-GQA Structural Conversion: The Multi-Head Attention modules are analyzed using Generalized Procrustes Analysis. Orthogonal transformations align the highly variant feature spaces of distinct attention heads before they are mean-pooled into tightly clustered groups, massively slashing future KV cache requirements6.
4. Information Entropy Pruning: Dual Taylor Expansion and HIES metrics identify and eradicate attention heads suffering from entropy collapse8. Simultaneously, FFN layers undergo sequential, localized pruning guided by Information Entropy scoring to remove redundant factual neurons13.
5. Activation-Aware Rank Reduction: SVD-LLM (V2) performs Cholesky-based data whitening on the remaining dense matrices. Truncated SVD decomposes the matrices, allocating heterogeneous ranks dynamically based on layer-specific theoretical truncation loss22.
6. Depth Reduction and QLoRA Healing: Block Influence (BI) scores dictate the excision of redundant contiguous deep attention layers1. The extensively modified, disjointed architecture undergoes a rapid parameter-efficient fine-tuning pass using QLoRA over the calibration data to heal severed representational pathways and correct accumulated arithmetic drift3.
7. Serialization and Obfuscation: The completely compressed topology is quantized using 8:16 block sparsity into q8\_0 or q4\_0 formats. The metadata is obfuscated via XOR masking, and the entire payload is compiled into the custom 108-byte header .slm container format via the tinyrustlm-slm-pack utility49.

4.2 Compatibility Rules for Runtime Loading

For the Javascript application shell to safely load and execute the model within the tinyrustlm-runtime WebAssembly environment, severe constraints are enforced:

• 128 MiB Transfer Ceiling: The total sum of the .slm file size, the allocated KV cache arrays, and the forward-pass scratch buffers must never exceed a global 128 MiB boundary. Violating this triggers browser-level Out-Of-Memory (OOM) interventions and hard crashes49.
• Dimensionality Contracts: The tensor directory must explicitly validate that the sequence of surviving layers maintains perfectly contiguous dimensional pathways. Furthermore, GQA group routing variables must perfectly divide the total query dimension without remainders29.
• Sparsity Validation: Any tensor declaring sparse block structure must strictly satisfy the 8:16 byte alignment. The runtime matrix-multiplication kernels will categorically reject and abort the load if a sparse matrix fails SIMD boundary alignment checks41.
• Tied Head Enforcement: Models executing with tied vocabularies must declare is\_tied=1 within the 108-byte header, replacing the final LM head payload with a direct zero-copy reference pointer back to the input embedding matrix54.

4.3 Accuracy-Risk Ranking by Technique

Implementing profound structural alterations carries varying degrees of risk regarding permanent degradation of reasoning capabilities. The following table delineates the comparative danger of each methodology.

4.4 File-Size and Memory-Impact Estimates

Applying the full pipeline fundamentally alters the operational footprint of a standard 1.5-billion parameter baseline model (e.g., Qwen or Llama architecture).

4.5 Required Validation Checks to Prevent Corrupted files

Before the WASM environment allocates memory, a rigorous sequence of integrity evaluations must occur to prevent executing a corrupted or maliciously manipulated computational graph.

1. Magic and Version Auditing: The parser instantly verifies the exact sequence of magic bytes indicating a legitimate SLM1 file format and checks the declared version against the compiled runtime compatibility matrix.
2. Directory Integrity: A high-speed cyclic redundancy check (CRC) algorithm validates the 64-byte hashed tensor directory. While non-cryptographic, this instantly catches bit-rot or HTTP stream corruption introduced during the file download without delaying the user experience49.
3. Topology Reassembly: The loader must successfully walk the entire Directed Acyclic Graph (DAG) before execution. If Layer 6 was removed during depth pruning, the system verifies that the output tensor dimensions of Layer 5 perfectly conform to the required input shapes of Layer 7\.
4. Datatype Boundary Alignment: Tensors flagged as compressed formats must satisfy exact arithmetic lengths. q8\_0 layouts must conform perfectly to designated byte-blocks, and any out-of-bounds pointer within a sparse index array must trigger an immediate safety abort to prevent out-of-bounds memory reading within the WebAssembly linear memory heap49.

4.6 A "No-Cheating" Benchmark Design

Traditional evaluation paradigms rely heavily on static, open-source datasets (e.g., MMLU, GSM8K). The massive proliferation of these datasets in the pre-training corpora of foundational models has rendered standard zero-shot metrics entirely useless for evaluating true architectural generalization; models simply regurgitate memorized strings55. For structurally compressed TinyRustLM deployments, a rigorous, cheat-proof benchmarking framework must be instituted. 1\. The Open-Closed Bifurcation: The benchmark must employ a strictly segregated two-stage evaluation modeled on the NeurIPS LLM Fine-Tuning Competition55.

• Open Set: Standard public datasets are utilized exclusively to verify basic runtime mechanics and API functionality.
• Closed Set: The actual scoring is performed against a completely isolated, proprietary dataset consisting of entirely novel prompts, logic puzzles mathematically isomorphic to known problems but syntactically distinct, and recent temporal events. This data is rigorously firewalled from the compression algorithms, forcing the model to rely exclusively on systemic reasoning55.

2\. Semantic Quality Validation via LLM-as-a-Judge: Exact string matching is profoundly brittle when applied to heavily pruned, highly quantized models. Instead, the framework relies on a Validity mechanism (V) and a History-feedback mechanism (H) governed by a massive, uncompressed "Builder Model" (e.g., a state-of-the-art 70B parameter frontier model)56. This judge model evaluates the semantic validity of the TinyRustLM outputs. In complex Theory-of-Mind (ToM) scenarios, the judge assesses whether the compressed model accurately tracked perspective shifts, hidden information, and nested beliefs, awarding points for logical coherence regardless of the specific phrasing utilized57. 3\. The Teleodynamic Efficiency Penalty: To prevent developers from submitting massive, unpruned dense networks that masquerade as optimized edge models, the final benchmark score must fuse accuracy with a draconian structural penalty. The final ranking is determined by calculating the geometric mean of the semantic win rates across all scenarios, which is then strictly divided by the product of the active memory footprint (in megabytes) and the inverse of the token generation latency55. Consequently, a model that relies on an unpruned MHA architecture will secure a high semantic score but will be mathematically obliterated by the efficiency denominator, guaranteeing that only models exhibiting profound, flawlessly executed structural compression top the leaderboard.

Works cited

1. liyunqianggyn/Awesome-LLMs-Pruning \- GitHub, https://github.com/liyunqianggyn/Awesome-LLMs-Pruning
2. Why Lift so Heavy? Slimming Large Language Models by Cutting Off the Layers \- arXiv, https://arxiv.org/html/2402.11700v2
3. THE UNREASONABLE INEFFECTIVENESS OF THE DEEPER LAYERS \- ICLR Proceedings, https://proceedings.iclr.cc/paper\_files/paper/2025/file/cbabc2f70de2dd09f491a8715ec3e80f-Paper-Conference.pdf
4. The Unreasonable Ineffectiveness of the Deeper Layers \- arXiv, https://arxiv.org/html/2403.17887v2
5. Uncovering the Redundancy in Transformers via a Unified Study of Layer Dropping, https://openreview.net/forum?id=1I7PCbOPfe
6. Align Attention Heads Before Merging Them: An Effective Way for Converting MHA to GQA \- arXiv, https://arxiv.org/html/2412.20677v1
7. How Layer Dropping Speeds Up LLM Inference \- Newline, https://www.newline.co/@zaoyang/how-layer-dropping-speeds-up-llm-inference--5f671d14
8. A Unified Head Importance–Entropy Score for Stable and Efficient Transformer Pruning \- arXiv, https://arxiv.org/html/2510.13832v1
9. Dual Taylor Expansion in Model Pruning \- Emergent Mind, https://www.emergentmind.com/topics/dual-taylor-expansion
10. D2 Prune: Sparsifying Large Language Models via Dual Taylor Expansion and Attention Distribution Awareness, https://ojs.aaai.org/index.php/AAAI/article/view/39932/43893
11. On the Limitations of Language-targeted Pruning: Investigating the Calibration Language Impact in Multilingual LLM Pruning | Transactions of the Association for Computational Linguistics \- MIT Press Direct, https://direct.mit.edu/tacl/article/doi/10.1162/TACL.a.599/135000/On-the-Limitations-of-Language-targeted-Pruning
12. SDMPrune: Self-Distillation MLP Pruning for Efficient Large Language Models \- arXiv, https://arxiv.org/html/2506.11120v1
13. High-Fidelity Pruning for Large Language Models \- arXiv, https://arxiv.org/html/2603.08083v1
14. Toward Adaptive Large Language Models Structured Pruning via Hybrid-grained Weight Importance Assessment, https://ojs.aaai.org/index.php/AAAI/article/view/34078/36233
15. (PDF) Prune, Update and Trim: Robust Structured Pruning for Large Language Models, https://www.researchgate.net/publication/404996615\_Prune\_Update\_and\_Trim\_Robust\_Structured\_Pruning\_for\_Large\_Language\_Models
16. Low-rank Factorization: A Comprehensive Guide for 2025 \- Shadecoder, https://www.shadecoder.com/topics/low-rank-factorization-a-comprehensive-guide-for-2025
17. Low-Rank Matrix Factorization in Large Language Models (LLMs) \- Medium, https://medium.com/@vasanthveec/low-rank-matrix-factorization-in-large-language-models-llms-08c2427c9a5e
18. ASVD: ACTIVATION-AWARE SINGULAR VALUE DE- COMPOSITION FOR COMPRESSING LARGE LANGUAGE MODELS \- OpenReview, https://openreview.net/pdf?id=HyPofygOCT
19. ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Models \- arXiv, https://arxiv.org/html/2312.05821v1
20. GitHub \- hahnyuan/ASVD4LLM: Activation-aware Singular Value Decomposition for Compressing Large Language Models, https://github.com/hahnyuan/ASVD4LLM
21. svd-llm: truncation-aware singular value decomposition for large language model compression, https://par.nsf.gov/servlets/purl/10647136
22. SVD-LLM V2: Optimizing Singular Value Truncation for Large Language Model Compression \- arXiv, https://arxiv.org/html/2503.12340v1
23. SVD-LLM V2: Optimizing Singular Value Truncation for Large Language Model Compression \- ACL Anthology, https://aclanthology.org/2025.naacl-long.217.pdf
24. Trainable Rank Decomposition Matrices \- Emergent Mind, https://www.emergentmind.com/topics/trainable-rank-decomposition-matrices
25. To Sparsify or To Quantize: A Hardware Architecture View \- SIGARCH, https://www.sigarch.org/to-sparsify-or-to-quantize-a-hardware-architecture-view/
26. Grouped-Query Attention Overview \- Emergent Mind, https://www.emergentmind.com/topics/group-query-attention-gqa
27. Attention Mechanisms in Transformers: Comparing MHA, MQA, and GQA | Yue Shui Blog, https://syhya.github.io/posts/2025-01-16-group-query-attention/
28. E16 : Grouped Query Attention \- by Praveen Thenraj \- Medium, https://medium.com/papers-i-found/e16-grouped-query-attention-818d201fe78f
29. Grouped-Query Attention (GQA): shrinking the KV cache \- ZeroEntropy, https://zeroentropy.dev/concepts/grouped-query-attention/
30. \[Literature Review\] Align Attention Heads Before Merging Them: An Effective Way for Converting MHA to GQA \- Moonlight, https://www.themoonlight.io/en/review/align-attention-heads-before-merging-them-an-effective-way-for-converting-mha-to-gqa
31. Align Attention Heads Before Merging Them: An Effective Way for Converting MHA to GQA, https://aclanthology.org/2025.findings-emnlp.467/
32. Align Attention Heads Before Merging Them: An Effective Way for Converting MHA to GQA, https://arxiv.org/html/2412.20677v2
33. Align Attention Heads Before Merging Them: An Effective Way for Converting MHA to GQA, https://www.researchgate.net/publication/387541021\_Align\_Attention\_Heads\_Before\_Merging\_Them\_An\_Effective\_Way\_for\_Converting\_MHA\_to\_GQA
34. Align Attention Heads Before Merging Them: An Effective Way for Converting MHA to GQA, https://www.semanticscholar.org/paper/Align-Attention-Heads-Before-Merging-Them%3A-An-Way-Jin-Song/9fd991b3f371833194256b08ad915593188dd942
35. VocabTailor: Dynamic Vocabulary Selection for Downstream Tasks in Small Language Models \- ACL Anthology, https://aclanthology.org/2026.findings-acl.1418.pdf
36. VOCABTRIM: Vocabulary Pruning for Efficient Speculative Decoding in LLMs \- arXiv, https://arxiv.org/pdf/2506.22694
37. GitHub \- asahi417/lm-vocab-trimmer: Vocabulary Trimming (VT) is a model compression technique, which reduces a multilingual LM vocabulary to a target language by deleting irrelevant tokens from its vocabulary. This repository contains a python-library vocabtrimmer, that remove irrelevant tokens from a multilingual LM vocabulary for the target language., https://github.com/asahi417/lm-vocab-trimmer
38. The Ups and Downs of Large Language Model Inference with Vocabulary Trimming by Language Heuristics \- ACL Anthology, https://aclanthology.org/2024.insights-1.17/
39. The Ups and Downs of Large Language Model Inference with Vocabulary Trimming by Language Heuristics \- arXiv, https://arxiv.org/html/2311.09709v2
40. Encoding the LLM Vocabulary Bottleneck \- OpenReview, https://openreview.net/forum?id=NNkDZsBu82
41. From 2:4 to 8:16 sparsity patterns in LLMs for Outliers and Weights with Variance Correction, https://arxiv.org/html/2507.03052v2
42. I built a custom 2-Bit Ternary Inference Engine from scratch in Rust \+ native PyTorch QAT. I'm running GPT-2 XL (1.5B) entirely offline on a Surface Pro 7 at 115 tokens/sec. : r/learnmachinelearning \- Reddit, https://www.reddit.com/r/learnmachinelearning/comments/1tmrv8r/i\_built\_a\_custom\_2bit\_ternary\_inference\_engine/
43. SIMT-Step Execution: A Flexible Operational Semantics for GPU Subgroup Behavior, https://www.researchgate.net/publication/406462036\_SIMT-Step\_Execution\_A\_Flexible\_Operational\_Semantics\_for\_GPU\_Subgroup\_Behavior
44. From 2:4 to 8:16 sparsity patterns in LLMs for Outliers and Weights with Variance Correction \- ACL Anthology, https://aclanthology.org/2026.acl-industry.66.pdf
45. Demystifying the Cost Versus Benefits of Sparse Large Language Model Acceleration, https://www.computer.org/csdl/magazine/mi/2026/02/11410568/2eoNkKYRKCY
46. Understanding ATen: PyTorch's tensor library \- Red Hat Developer, https://developers.redhat.com/articles/2026/02/19/understanding-aten-pytorchs-tensor-library
47. GitHub \- ml-rust/numr: A high-performance numerical computing library for Rust with GPU acceleration, inspired by Numpy, https://github.com/ml-rust/numr
48. numr 0.5.0: The Rust numerical computing library that doesn't make you choose, https://dev.to/farhansyah/numr-050-the-rust-numerical-computing-library-that-doesnt-make-you-choose-cpp
49. Implementation \- MiRust, https://mirust.com/implementation/
50. https://mirust.com/inside-the-slm1-model-container/
51. Hiding in Plain Sight: An IoT Traffic Camouflage Framework for Enhanced Privacy \- arXiv, https://arxiv.org/html/2501.15395v1
52. Adversarial Networks and Machine Learning for File Classification \- arXiv, https://arxiv.org/pdf/2301.11964
53. Align Attention Heads Before Merging Them: An Effective Way for Converting MHA to GQA \- ACL Anthology, https://aclanthology.org/2025.findings-emnlp.467.pdf
54. Documentation \- MiRust, https://mirust.com/docs/
55. NeurIPS 2023 LLM Efficiency Fine-tuning Competition \- arXiv, https://arxiv.org/html/2503.13507v1
56. Mostly Automatic Translation of Language Interpreters from C to Safe Rust \- arXiv, https://arxiv.org/html/2606.27122v1
57. Test-Time Harness for Strong-to-Weak Capability Transfer \- OpenReview, https://openreview.net/pdf?id=igwmpjj3gb