Structural Compression Strategy for TinyRustLM .slm Models

Full report

On this page

Executive assessment

For TinyRustLM, structural compression is more important than further quantization work if the goal is to reduce real parameters and real browser-side compute. The public TinyRustLM docs describe a strict .slm artifact contract with a 33,554,432-byte model budget, fixed f32, q8_0, and q4_0 storage modes, checksum-bound loading, and a runtime that executes a decoder-only transformer through handwritten scalar Rust loops compiled to WebAssembly. The same docs explicitly say there is no batched prefill, no SIMD-specific kernel, no WebGPU dispatch, and that quantization mainly reduces memory bandwidth rather than changing the runtime’s scalar execution topology. That makes architecture-changing methods such as depth reduction, FFN-width pruning, head/KV reduction, vocabulary reduction, and selected low-rank replacement the highest-value levers for TinyRustLM deployment.

The best practical target for TinyRustLM is not “maximum sparsity.” It is a canonical smaller transformer that preserves the existing execution model as much as possible: fewer layers, fewer KV heads, narrower FFNs, possibly fewer attention heads, a trimmed vocabulary when the deployment domain permits it, and tied embeddings where mathematically valid. In contrast, unstructured sparsity is usually a poor default for browser inference unless the runtime has dedicated sparse kernels, because structured pruning is the form that reliably converts into hardware-agnostic speedups and simpler execution graphs. ZipLM makes exactly this point for language models, and TinyRustLM’s current scalar/WASM runtime makes that argument even stronger.

A second major conclusion is format-related: many useful structural changes do not require exposing proprietary conversion internals in the .slm file. If the exporter rewrites the Hugging Face checkpoint into a new canonical dense model with smaller dimensions, the runtime only needs the final executable architecture, not the pruning scores, head-importance traces, teacher logits, or calibration prompts that produced it. New header or tensor-kind semantics are only necessary when the runtime must execute something genuinely new, such as factorized tensors or block-sparse tensors.

Accuracy-risk ranking by technique

The safest technique in this stack is embedding/output-head tying, but only when the model architecture supports shared input and output dimensions and the deployment does not depend on separate learned LM-head behavior. Press and Wolf showed that tying the input embedding and output embedding can improve or preserve language-model quality while substantially reducing parameter count; Hugging Face also documents that models can maintain tied embeddings through resize_token_embeddings() when the model class supports tie_weights(). In TinyRustLM terms, tying is a real parameter reduction, but not a logits-compute reduction.

Next safest is modest grouped-query attention conversion, especially when done with recovery training. Ainslie et al. showed that existing multi-head checkpoints can be uptrained into MQA/GQA variants using about 5% of original pretraining compute and that GQA reaches quality close to multi-head attention while preserving much of MQA’s inference advantage. Hugging Face’s Nemotron docs state the operational rule clearly: when converting an MHA checkpoint to GQA, each grouped key/value head should be constructed by mean-pooling the original heads in that group. For TinyRustLM this matters twice: GQA reduces K/V projection parameters and, more importantly, shrinks the KV cache linearly with the ratio num_key_value_heads / num_attention_heads.

After that come FFN channel pruning and MLP width reduction, which are usually the best structural pruning target in modern decoder blocks because TinyRustLM’s execution path is W1/W3 → SwiGLU → W2, so pruning one intermediate channel removes weights from three matrices at once. Recent LLM pruning work strongly supports structured channel pruning: LLM-Pruner removes non-critical coupled structures using gradient information, MINI-LLM combines magnitude, activation, and forward-estimated gradient sensitivity, SlimLLM uses holistic channel/head scoring plus layer-wise sparsity allocation, and NVIDIA’s Minitron explicitly prunes MLP intermediate dimension, attention heads, and embedding size before continuing training with distillation. For TinyRustLM this is the highest-confidence source of large parameter reduction without inventing a new runtime.

Attention-head pruning is also attractive, but slightly riskier than FFN pruning because some layers remain head-sensitive. Michel et al. found that many heads can be removed after training without major loss and that some layers can be reduced to a single head, while Voita et al. showed that the most important heads are often the most confident/specialized and that the majority of heads can be pruned with little quality loss under an L0-style gating method. For TinyRustLM, the practical takeaway is to prune heads with global saliency and keep late, specialized, or cross-token-structure heads conservative.

Layer dropping / depth reduction is powerful but meaningfully riskier if done post hoc. LayerDrop showed that a model trained with structured layer dropout can be evaluated at smaller depths, and ShortGPT later argued that many LLM layers are redundant enough for direct removal based on a block-influence score. In practice, this means layer dropping is best used in two cases: either the source model was trained with depth robustness in mind, or you are willing to do recovery training or distillation after removing layers. For TinyRustLM, depth reduction has excellent latency upside because it cuts the full per-token stack, but it is coarse and should be applied only after finer-grained FFN/head/KV decisions have been made.

Low-rank factorization sits in the middle on accuracy risk and high on runtime-implementation risk. ASVD shows that activation-aware SVD can compress LLM weights post-training, LoSparse combines low-rank and sparse approximation, and Low-Rank Prune-And-Factorize reports better compression-performance tradeoffs when pruning is used to create factorization-friendly structure. However, on a scalar WASM runtime, factorization turns one matrix multiply into two. So it only helps latency when the retained rank is well below the algebraic break-even point, and it helps engineering simplicity only if the exporter materializes a smaller dense model or the runtime gains explicit factorized-tensor kernels. Accuracy can be good with conservative ranks, but deployment value depends heavily on runtime support.

The highest-risk mainstream technique in this list is vocabulary trimming. It can produce very large savings on small models because embeddings become a larger fraction of total parameters as models shrink, but it changes the tokenizer/model contract and can damage broad-domain behavior. Recent work on vocabulary reduction for SLMs found that embedding layers are a real bottleneck in smaller models and that vocabulary reduction can materially lower memory footprint, but quality is task-sensitive and often benefits from fine-tuning afterward. Another study found that language-targeted vocabulary trimming can reduce memory use and improve speed, but the gains are inconsistent and diminish for larger models. In other words, vocabulary trimming is valuable only when the deployment domain, language mix, and tokenizer contract are tightly controlled.

Finally, unstructured sparsity should rank last for TinyRustLM browser inference. It can improve storage at very high sparsity and can work well in specialized sparse runtimes, but the strongest browser-adjacent production evidence is still narrow: TensorFlow’s XNNPACK latency-oriented sparse pruning policy is explicitly restricted to 1×1 Conv2D patterns, not generic transformer linear layers. ONNX can represent sparse tensors, but the practical browser path for transformer matmuls remains much less mature than dense execution. For TinyRustLM’s current scalar/WASM runtime, block sparsity is the only sparse path that is plausibly worth engineering for execution speed.

Recommended Hugging Face to .slm compression pipeline

The recommended pipeline is a dense-first structural rewrite pipeline. Start from an FP16 or FP32 Hugging Face base model, not a merely quantized deployment checkpoint. If the source is a PEFT/LoRA model, merge adapters into the base model first so the compressor sees the actual deployed weights. Hugging Face’s PEFT docs explicitly recommend merge_and_unload() when the goal is a standalone model with no adapter latency, and safe_merge exists to catch NaNs during merging.

Then run a calibration pass over a held-out internal compression set to collect the saliency signals needed for structured pruning. For layers, use a depth-sensitivity signal such as block influence or layer-drop survivability. For attention heads, combine at least two views: a direct importance estimate in the Michel style and a confidence/specialization or entropy-style estimate in the Voita style; if memory allows, add MINI-LLM-like gradient sensitivity or SlimLLM-style holistic head scoring. For FFN channels, compute channel-level importance, not individual-weight importance, and in SwiGLU blocks prune channels jointly across W1, W3, and the matching rows/columns in W2. That respects the actual TinyRustLM layer order and avoids “ghost width” where tensors keep original shapes while only storing more zeros.

Apply compressions in this order unless profiling proves a better model-specific sequence. First, tie embeddings if the architecture permits it and you have no evidence that a distinct LM head is required. Second, convert MHA to GQA with a conservative KV-head count, typically not all the way to MQA on the first pass. Third, prune FFN channels to a target width allocation per layer. Fourth, prune attention heads using global whole-head ranking, not flat per-layer quotas. Fifth, drop layers only after the cheaper structural redundancies are already removed. Sixth, apply low-rank factorization only on the heaviest surviving matrices and only where the chosen rank is small enough to help both bytes and runtime. Seventh, consider vocabulary trimming only for domain-locked .slm targets such as English-only, medical-only, or one-product assistant deployments. This order front-loads the changes with the best accuracy-per-byte payoff and leaves the most destabilizing changes last.

After each structural stage, run a short recovery phase. For TinyRustLM, the efficient recovery recipe is not full retraining; it is continued training or distillation on modest data. Minitron reports exactly this style of pipeline, pruning width/head/embedding structure and then continuing training with distillation. LLM-Pruner similarly uses LoRA-based recovery after pruning, and ZipLM uses distillation to maintain accuracy on compressed models. For a TinyRustLM exporter, the recovery loss should combine teacher-token KL, standard next-token loss on an unseen-but-non-secret recovery corpus, and optional hidden-state matching for layers you retained.

The export rule should be strict: produce a new executable model, not a masked old model. If you prune heads, export smaller Q/K/V/O tensors or a GQA-configured checkpoint. If you prune FFN channels, export narrower W1/W3/W2. If you drop layers, renumber layers contiguously in the final artifact. If you trim vocabulary, rewrite tokenizer and embedding/output rows accordingly. If you cannot express a change without hidden masks, that transformation is not ready for .slm export yet. This is the single most important anti-cheating rule.

Runtime compatibility and safe .slm encoding

For TinyRustLM, the cleanest compatibility rule is to separate transforms into canonical dense shrinks and new executable tensor kinds. Canonical dense shrinks include layer dropping, FFN-width reduction, head pruning when materialized as smaller tensors, GQA conversion, vocab trimming, and untied-to-tied embedding export when the runtime can represent an alias cleanly. These changes can often remain within the spirit of the current .slm contract because the runtime only needs final dimensions, final tensor payloads, tokenizer data, and checksums. The public .slm docs emphasize explicit model dimensions, bounded validation, fixed header/tensor directory structure, tokenizer inclusion, and checksum binding.

Only factorized and block-sparse tensors truly require a format extension. The current public .slm description exposes a 108-byte header target and 64-byte tensor entries, but I did not find public documentation for native low-rank or sparse tensor kinds. So the safe rule is: keep SLM1 dense and canonical; introduce a new version or explicit extension block only when the runtime has new kernels that understand those tensor kinds. Do not overload SLM1 by stuffing proprietary pruning traces into the header.

A good extension design is minimal and runtime-facing. The runtime needs to know only: the final architecture dimensions, whether embeddings are tied, the number of query and KV heads, the tokenizer spec/hash, and the storage kind of each tensor. It does not need head-importance scores, channel rankings, calibration prompts, teacher outputs, optimizer states, or proprietary decisions used during conversion. If provenance is desirable, store only an opaque hash of an external signed provenance bundle, not the bundle itself. That preserves IP and keeps the artifact ABI focused on execution.

The compatibility matrix should be explicit. A dense-only runtime should accept: contiguous layer-reduced models, FFN-narrowed models, GQA models, tied-embedding models, and vocab-trimmed models, provided every tensor shape matches the declared architecture. A factorization-aware runtime may additionally load low-rank tensor pairs. A sparse-aware runtime may additionally load block-sparse tensors. If the runtime lacks a declared tensor kind, loading must fail at admission time before scratch, KV cache, or logits buffers are committed. That matches TinyRustLM’s documented load-then-commit transaction model.

File-size, memory, and sparsity economics

For TinyRustLM’s decoder block, the current public execution order is attention followed by a SwiGLU FFN: Q/K/V → attention → W_O and then W1/W3 → SwiGLU → W2. If hidden size is d, FFN width is m, query-head count is H, and KV-head count is G, then a rough per-layer parameter model is attention ≈ 2d² + 2d²(G/H) for GQA and FFN ≈ 3dm for SwiGLU. The important consequence is that FFN pruning removes weights from three matrices per layer, while GQA mainly shrinks K and V projections plus the KV cache.

That gives a useful first-order impact model:

• Layer drop: removing a fraction p of layers cuts about p of stacked-layer bytes and about p of per-token compute, because the full attention+FFN block disappears on every token.
• FFN channel pruning: pruning a fraction p_f of FFN width saves about 3 L d m p_f parameters across L layers and removes comparable FFN matmul work. This is usually the biggest byte win in a modern decoder.
• Head pruning: pruning one head in a standard MHA layer saves about 4 d (d/H) parameters from the Q/K/V/O projections at that layer, plus corresponding attention work. Michel et al. showed that modest fractions can be removed with little damage, but not every layer can collapse to one head safely.
• GQA conversion: switching from H KV heads to G KV heads saves about 2 L d² (1 - G/H) parameters in the K/V projections and reduces KV-cache memory by exactly the factor G/H. For example, 32 → 8 KV heads cuts KV-cache bytes by 75%.
• Low-rank factorization: replacing an M × N matrix by two dense factors of rank r stores r(M + N) parameters instead of MN, so it is only a byte win when r < MN / (M + N). On TinyRustLM’s scalar runtime, it is only likely to be a latency win when r is well below that threshold.
• Embedding tying: if the LM head is independent, tying removes one V × d matrix. If the model is already architecturally tied, there is nothing further to save.
• Vocabulary trimming: shrinking vocabulary from V to V' saves (V - V') d embedding parameters and, if embeddings are untied, another (V - V') d output-head parameters. It also reduces logits computation roughly in proportion to V'/V. The gains are especially large on SLMs because the embedding fraction grows as model size shrinks.

At the .slm file level, structural compression scales across all supported storage modes. A 20% reduction in actual stored parameters removes about 20% of the quantized payload too. Against TinyRustLM’s 32 MiB budget, that is roughly 6.4 MiB saved before tensor-directory and tokenizer overheads, whether the remaining weights are dense q8_0, dense q4_0, or a supported sparse/factorized kind. That is exactly why structural compression is more durable than “quantize harder” for this runtime.

For sparse storage, unstructured CSR/CSC is rarely attractive against dense q4/q8. Rust sparse libraries such as nalgebra-sparse support COO/CSR/CSC formats, which are useful interchange formats, and ONNX also defines sparse tensors. But for a quantized LLM weight matrix, the index overhead dominates unless sparsity is very high. As a rule of thumb, unstructured CSR with int32 column indices costs roughly 5 bytes per nonzero at q8 and about 4.5 bytes per nonzero at q4 before row-pointer overhead, so it beats dense q8 only below roughly 20% density and dense q4 only below roughly 11% density. Those are storage break-even points, not execution break-even points.

That is why the only sparse format I would recommend for a future .slm execution extension is block sparse row. BSR is specifically intended for sparse matrices with dense submatrices and is documented as more efficient than CSR/CSC for many arithmetic operations. In a Rust/WASM runtime, fixed-size block sparsity also preserves regular memory access and SIMD-friendliness far better than scalar-coordinate sparsity. So use CSR/CSC/COO for exporter internals and debugging, but use BSR-like storage for any runtime sparse execution path.

Required validation checks for structurally compressed .slm files

TinyRustLM already documents several baseline checks: header validation, tensor-directory validation, tokenizer checksum, layout checksum, explicit dimension checks, in-file bounds checks for tensor entries, and “no NaN/Infinity payloads.” Those should remain mandatory for compressed artifacts.

On top of that, structurally compressed .slm files need additional semantic validation:

• Canonical architecture consistency. The declared num_hidden_layers, hidden_size, intermediate_size, num_attention_heads, num_key_value_heads, and vocab_size must exactly match the physical tensor shapes in the artifact. No exported tensor may retain the old size with hidden masks or dead rows/columns.
• GQA validity. num_attention_heads must be divisible by num_key_value_heads; K/V tensor shapes and KV-cache layout must match the grouped-head geometry; if the model was converted from MHA, grouped K/V heads must be constructed consistently, such as by group mean-pooling.
• SwiGLU FFN integrity. If FFN channels are pruned, the same intermediate channels must be removed coherently across W1, W3, and W2; otherwise the exported block is structurally misleading even if shapes happen to multiply.
• Embedding/tokenizer integrity. For vocab-trimmed models, tokenizer metadata inside the artifact must match the embedding rows exactly. If embeddings are tied, the alias relationship must be explicit and hashable; if untied, row counts must still match vocabulary size exactly. Hugging Face’s resizing and special-token caveats are relevant here.
• Factorized tensor integrity. If a tensor is stored as low-rank factors, the loader must verify the logical output shape, factor rank, finite values, and multiplication order. If the rank is not actually compressive, the exporter should reject it as misleading rather than presenting it as “compressed.”
• Sparse tensor canonicality. For block-sparse tensors, block size must divide the logical matrix dimensions; block-column indices must be sorted; duplicate blocks must be forbidden; and zero blocks should not be stored. These are standard canonical-format expectations for block sparse matrices.
• Quality-claim separation. The artifact should not contain unverifiable statements like “95% of baseline quality.” If benchmark receipts exist, they should be external, signed, bound by hash to the model artifact, and ignored by the runtime if missing or mismatched. This prevents a .slm file from becoming self-attesting marketing.

One additional anti-cheating rule is worth making explicit: the exporter should record both stored parameter count and effective executable parameter count, and they must be equal for dense canonical models. If a sparse or factorized extension is used, executable semantics must be declared by tensor kind. A model must never claim structural compression while relying on hidden dense expansion during load. That rule is partly a design recommendation, but it follows directly from TinyRustLM’s strict bounded-loading philosophy.

No-cheating benchmark design

A defensible benchmark for structurally compressed .slm models needs three physically separate datasets: a calibration set for importance scoring, a recovery/distillation set for post-pruning repair, and a final hidden holdout for reporting. The hidden holdout must be unseen by the pruning logic, unseen by the recovery stage, and ideally unpublished until benchmark lock. That is the only clean way to prevent tuning the converter against the test set. HELM is a useful reference here because it emphasizes standardized, multi-metric evaluation and public release of prompts/completions for transparency, though for your final acceptance gate you should keep the true holdout private until the model is frozen.

The final evaluation should mix exact, semantic, and preference-style metrics. For exact metrics, measure hidden-corpus perplexity, multiple-choice accuracy, exact-match or F1 on QA where relevant, and code-pass metrics if code generation matters. For semantic metrics on open-ended tasks, use an embedding-aware metric such as BERTScore against hidden references where references exist. BERTScore was proposed specifically because it correlates better with human judgment than older overlap metrics.

For open-ended chat quality where references are weak, use pairwise judging with bias controls rather than a single-reference score. MT-Bench is the right template: randomized answer order, hidden model identity, fixed judging rubric, and auditing for position, verbosity, and self-enhancement bias. The MT-Bench paper reports that strong judges can exceed 80% agreement with humans, but it also documents the exact biases you need to guard against. For TinyRustLM acceptance, I would require both LLM-as-judge scoring and a stratified human audit sample on the hidden holdout.

To keep the benchmark honest, each run should emit a receipt bundle: model hash, tokenizer hash, compression recipe ID, runtime version, prompt set hash, deterministic sampling config where applicable, raw completions, and all metric outputs. This is a HELM-style transparency move, but with one additional rule: the true holdout prompt text stays sealed until model lock, then is published alongside results so others can verify the receipts after the fact.

The acceptance policy should be scenario-based rather than one-number based. A practical release gate for TinyRustLM would be: no hidden-task catastrophic failures, bounded perplexity regression on the hidden corpus, bounded pairwise-loss rate against the baseline model, and no increase in hallucination-sensitive failure classes on truthfulness or retrieval-grounded prompts. Vocabulary-trimmed models should have stricter domain-scope declarations than architecture-pruned models because they change the valid token universe itself.

Recommended deployment stance

For TinyRustLM today, the highest-confidence compression stack is:

1. merge LoRA if present;
2. tie embeddings when valid;
3. convert MHA to conservative GQA;
4. prune FFN channels with holistic channel scoring;
5. prune attention heads conservatively;
6. drop a small number of redundant layers;
7. optionally apply ASVD-style low-rank factorization only to the heaviest surviving matrices and only if the runtime truly benefits;
8. trim vocabulary only for domain-locked deployment targets;
9. export a canonical smaller model, not a masked original.

If you want one sentence of policy: prefer structural changes that the current dense TinyRustLM runtime can execute natively, and postpone sparse/factorized execution formats until the runtime has explicit kernels for them. That policy best matches the current .slm contract, the current Rust/WASM execution model, and the research evidence on what kinds of compression reliably turn into real runtime wins.

Open questions and limitations

The public TinyRustLM documentation clearly describes the current .slm container and runtime behavior, but I did not find a public executable spec for native factorized tensors or native sparse tensor kinds. The report therefore treats those as proposed extensions rather than existing .slm features.

I also did not find public TinyRustLM benchmark receipts comparing dense structural rewrites against sparse execution kernels in the browser. So the recommendations on block sparsity for Rust/WASM are based on the current TinyRustLM runtime constraints plus general sparse-format and sparse-runtime evidence, not on a published TinyRustLM sparse-kernel benchmark.