Executive Answer

Full report

On this page

A valid submodel must satisfy the transformer’s graph closure: it must include all necessary inputs, outputs, and intermediate nodes so that the model can compute end-to-end without missing pieces. In practice, this means only structured reductions (e.g. pruning whole heads, neurons, or layers) that preserve dimensional consistency and dependencies will yield a working subnetwork. Every removed component must be “closed” by adjusting connected layers (e.g. reshaping weight matrices, re-aligning residuals, and recalibrating normalization). In contrast, naïve slicing – arbitrary deletion of weights, attention heads, or vocabulary rows – typically violates these constraints. Such direct cuts break residual connection alignment or drop critical embedding/output mappings, causing the model to fail catastrophically (e.g. repeating tokens or garbling output).

In short, a useful submodel must be explicitly designed or recovered to form a coherent graph; one cannot simply “take the useful parts” without retraining or structural repair. As many studies show, purely removing components causes large accuracy drops unless followed by careful recovery or re-training. Valid submodels arise from methods that ensure closure (structured pruning with fine-tuning, nested supernet training, etc.), while invalid slicing (random head/weight drops, ad hoc vocabulary trims) reliably produces nonsense.

Assumptions and Terminology

We assume a standard Transformer-based language model (with embedding layers, multi-head self-attention, residual connections, layernorm, feed-forward networks (FFNs), etc.) and consider subnetworks defined by choosing a subset of its parameters. Graph closure means the chosen parameters must form a complete computational subgraph: if a tensor’s output is kept, all of its inputs and preceding operations must also be kept (or equivalently, the graph of computation is closed under ancestry). This ensures inputs to each layer have the expected dimensionality and that residual pathways and attention projections remain valid.

• Residual dimensions and layernorm: We assume standard residual-add connections and layernorm. Removing any part of a layer (e.g. some channels) must preserve the overall hidden dimension so that residual additions remain well-defined. In practice we treat each residual block’s output size as fixed.
• Attention projections: We denote Q/K/V weight matrices and the output projection. In multi-head attention, the hidden size is split into head subspaces. Removing an entire head requires slicing Q/K/V projections and output projections by the same subspace indices. In multi-query or grouped-query attention, keys/values are shared across heads, so heads cannot be independently deleted without adjusting the shared KV dimensions.
• Rotary positional embeddings: If rotary embeddings are used (as in LLaMA), they apply a sinusoidal rotation across each pair of dimensions. Removing dimensions breaks the rotary pattern, so either all rotary dimensions must be kept or a consistent subspace removed.
• FFN gating (GLU/GELU): Many modern models (LLaMA, GLM) use gated FFNs (e.g. GEGLU) with two linear projections (gate_proj and up_proj) feeding into an elementwise product. These projections are 1-to-1 paired: to remove neurons from the FFN, exactly the same neuron indices must be removed in both projections. If this coupling is violated, the gating breaks and output collapses.
• Embedding and vocabulary: The input embedding matrix and (tied) output projection share parameters. Removing tokens (vocabulary rows) changes the dimensionality of these matrices. A specialist submodel with reduced vocabulary must handle unknown tokens – typically by mapping them to a special token or falling back to a larger model. Tokenizer dependencies: A reduced vocabulary may be incompatible with the original tokenizer; either the tokenizer must be adapted or all out-of-vocabulary words must trigger fallback. Tied embeddings mean any removal of embedding columns (hidden dims) must be mirrored in the output projection.
• Champion vs specialist: We assume a “champion” general model (large, many parameters) plus smaller “specialists.” Each specialist may be a pruned/distilled submodel or an adapter, always with fallback to the champion for unsupported cases (e.g. unknown tokens).

Taxonomy of Subnetwork and Compression Methods

1. Post-hoc deletion (one-shot pruning): After training, remove components (entire layers/heads/neurons) based on some criterion. No further training or recovery is performed. Example: ShortGPT removes redundant layers by similarity analysis. Typically yields severe quality loss unless the model happened to have very redundant structure.
2. Prune-and-recover (structured pruning): Iteratively remove structural units (layers, heads, channels, neurons) and then recover accuracy via fine-tuning or mask optimization. This includes many “post-training” pruning schemes for LLMs. Recovery may be classical fine-tuning or specialized mask-learning (e.g. FPT’s least-squares mask tuning). Training required: access to training or calibration data and gradient-based updates.
3. Train-time elastic (universal/supernet training): Train the model to support multiple capacities a priori. Techniques include Once-for-All and Universally Slimmable Networks (US-Nets). A single “super-model” is trained with switches or masks so it can run at varying widths, depths, or token budgets. Training uses the sandwich rule (optimize largest, smallest, and random sub-networks per batch) and often in-place distillation from the largest to smaller subnets. No retraining is done per sub-model at deployment; the weights were jointly learned.
4. Nested checkpoints (sandwich or matryoshka networks): During a single training run, checkpoints are saved at multiple depths or widths. Alternatively, nested training like Flexible Transformer or Matryoshka networks load progressively larger sub-models and prune or grow them. This requires tracking multiple sets of BN/LayerNorm statistics (or using post-hoc BN correction). Training interference: smaller subnets share weights with larger ones, so the “sandwich” training stabilizes all widths.
5. Conditional execution (dynamic skipping): Modify the model to allow skipping or dropping computations at run-time. Examples include layer-dropping (stochastic depth, early exit branches) and Mixture-of-Experts (MoE) (only activate a subset of experts per input). Training involves learning gating functions or exit classifiers. At inference, only the relevant parts execute. No static weight removal is done – instead, components are conditionally unused. (This can be viewed as a form of elasticity along the depth or expert axes.)
6. Independent distillation (separate student models): Train one or more smaller student models from a large teacher. These are entirely new networks (often same architecture but fewer layers/dims, or different specialized architecture) that distill knowledge from the champion. Training requirements: full distillation (soft-label or logits matching) or task-specific fine-tuning. Resulting models are standalone and do not share weights at runtime with the champion.

Each class has different training needs: (2) requires fine-tuning data and iterations, (3–4) require specialized multi-configuration training regimes, (5) requires gating losses during training, and (6) is essentially standard supervised (or distillation) training.

Transformer Graph-Closure by Tensor Family

To define a valid subnetwork, every tensor and weight we retain must have all its dependencies present. Below we analyze the key tensor groups in a transformer:

• Residual/hidden dimensions: Every transformer block adds its output to a residual. Thus the hidden dimension $d_{\text{model}}$ is fixed. Removing neurons or heads must preserve that entire dimension size. For example, if we remove some attention heads (each of size $d_{\text{head}}$), we must either reduce $d_{\text{model}}$ accordingly across all layers (reshaping every weight matrix) or recast the model with a smaller $d_{\text{model}}$. A partial removal in one layer without adjusting others breaks the addition of residuals.
• Normalization layers: LayerNorm (or its equivalents) operate on the full hidden dimension. If we change $d_{\text{model}}$, we must rebuild or re-parameterize the LayerNorm weights and running statistics for the new shape. Similarly, BatchNorm would require recalculating statistics for any altered channels. (Slimmable networks often use “switchable” or post-training recalculated normalization.)
• Attention projections (Q/K/V/O): Multi-head attention has projections $W_q,W_k,W_v,W_o$ of shapes $(d_{\text{model}}\times d_{\text{model}})$. These are typically implemented as (model_dim→heads×head_dim). Removing a head means slicing out its corresponding columns from the query/key/value projections and the corresponding rows from the output projection. In grouped-query attention, multiple heads share key/value matrices: deleting a head still leaves its shared KV vectors needed by others, so naive head removal is invalid unless KV are duplicated or group restructured. In multi-query attention (one KV pair for all heads), heads are fully interdependent, so you cannot drop an output dimension without adjusting all queries.
• Rotary positional embeddings: Rotary embeddings apply a fixed rotation matrix to each pair of hidden dimensions. If hidden dims are removed, one must remove the corresponding rows/columns from the rotary matrices, but the sinusoidal pattern ties dimensions in pairs. Thus one can only remove pairs of rotary dims or none; arbitrarily cutting a single dim would desynchronize the rotation (the math expects even-length chunks). In practice, rotary often uses exactly the first $d$ dims; subsetting those uniformly may be possible if careful, but generic head slicing will break them.
• FFN gates (GLU/GEGLU): The LLaMA-style FFN has two parallel projections (gate_proj and up_proj) each mapping $d_{\text{model}}\to d_{\text{ffn}}$. Their outputs are multiplied elementwise (gating). To prune FFN neurons, the same neuron indices must be pruned in both projections. If $i$th neuron is removed in gate_proj but not up_proj (or vice versa), the product is misaligned. The GLU-aware pruning blog shows that removing inconsistent neurons leads to nonsense output (“Paris is the capital of of France… the the the…”).
• Embedding layers: The input embedding matrix (vocab×$d_{\text{model}}$) and the output (“LM head”) projection ($d_{\text{model}}$×vocab) are often tied. Removing vocabulary rows (words/tokens) requires removing the same row/column in both matrices. Removing hidden dims (columns of the embedding) must be done consistently in both the input embedding and all projection matrices (including LayerNorm and FFN weights) to keep shapes aligned. A subnetwork that drops tokens (e.g. specializing in a domain) must also handle any input that falls outside the reduced vocabulary, usually by mapping it to an “unknown” token or deferring to the champion model.
• Tokenizer/Vocabulary dependencies: A model’s tokenizer maps text to token IDs. If a submodel uses a reduced tokenizer or merges rare tokens, its embeddings and output layer shrink. However, text outside that domain may not tokenize properly. Common practice is to use the full-tokenizer and map out-of-vocab to a fallback token. This means a specialist can share the champion’s tokenizer and simply have an embedding row for “unknown” (typically zeroed), but it still must carry the full mapping logic. In all cases, changing the vocabulary size invalidates the final linear layer’s shape, which must be reshaped (e.g. a $d\times |V|$ weight matrix becomes $d\times |V'|$) and often requires rebalancing logits (e.g. adding an “other” class).

In summary, any submodel must cut tensors in ways that keep all linear algebra shapes consistent. This usually entails removing or preserving whole units (heads, neurons, channels) with compensating shape edits, rather than slicing arbitrary portions of weight matrices. Missing any dependency (e.g. leaving a LayerNorm expecting a dimension that was removed) will break the graph.

Architecture Survey (0.3B–2B range)

Below is a (non-exhaustive) table of representative small language models (roughly 0.3–2B parameters) with their key architecture figures, license, and publication. This illustrates the landscape for TinyRustLM’s target scale:

(Notes: Architectures are broadly similar: multi-head self-attention with comparable depth and width. Licenses vary: LLaMA-2/3 are under Meta’s research licenses (CC-BY-NC or CC-BY-SA). BLOOM uses its RAIL license (non-commercial). Apache 2.0 licenses (e.g. GPT-NeoX, Falcon) are permissive. Always check model-specific terms.)

Training/Recovery and Failure Modes

• Complete component removal (layers/heads/channels): Removing entire layers or heads yields a smaller model architecture, but generally causes large quality drops unless countermeasures are taken. Recovery typically means fine-tuning or iterative re-training of the pruned model on data. Failure mode: no-recovery removal leads to high perplexity and nonsensical outputs (as [41] shows with ShortGPT/SliceGPT on non-LLaMA models). Even with recovery, aggressive layer removal can still underperform because the model may re-learn redundant mappings rather than improve efficiency.
• Neuron/channel pruning: Cutting FFN neurons or embedding channels without recovery similarly breaks accuracy. The model can only partially adapt by redistributing weight to remaining neurons. Recovery: requires training (gradient updates) to re-optimize the reduced network, or sophisticated mask search (e.g. COMP’s mask tuning). Without recovery, outputs degrade (especially rare-word handling suffers if embedding rows are removed).
• Vocabulary reduction: Truncating the vocabulary (e.g. removing rare or domain-irrelevant tokens) invalidates the LM head. The shape of the final linear layer must be changed, and softmax outputs reinterpreted. Recovery: usually done by retraining the projection on text from the specialist domain. Failure: if done without adjustment, the model cannot produce scores for removed tokens, leading to fallback (outputting <UNK> or nothing).
• Sparse vs structured removal: Fine-grained (unstructured) sparsity (zeroing weights) typically requires retraining to work well, and even then it only saves storage, not computation (dense kernels still run). Structured sparsity (e.g. N:M patterns) may enable some hardware speedups but still generally needs retraining with special regularizers. Failure: straightforward magnitude pruning of individual weights (unstructured) without fine-tuning yields accuracy loss similar to dense pruning, and the resulting sparse model may not be faster without specialized libraries.

In summary, any removal beyond trivial adjustments demands some form of recovery training or calibration. Purely “prune and evaluate” is rarely effective for substantial size reductions. Common failure modes include output repetition, collapsed generation, and inflated perplexities.

CPU, SIMD, and WebAssembly (WASM) Implications

Language models on CPU/SIMD or in-browser WASM have different performance tradeoffs:

• Dense kernels vs sparse representations: CPUs and WebAssembly engines generally execute dense matrix multiplies efficiently (especially with SIMD instructions). If pruning only reduces parameter count but the matrix shapes stay the same (e.g. masked sparse weights), runtime doesn’t improve: the matrix multiply still iterates over the full shape, checking zero entries. In contrast, structured pruning (reducing hidden size or channels) yields smaller dense mats, directly saving compute. Unstructured/N:M sparsity requires extra logic: e.g. storing bitmasks or index lists. Decoding these on the fly in WASM or general CPU code adds overhead (per-matrix-block masks, bit-twiddling, or gather/scatter operations) that can negate arithmetic savings.
• SIMD utilization: SIMD (vector) units work best on contiguous data. Channel pruning (keeping full contiguous columns/rows) is SIMD-friendly, as it just shortens vectors. Block sparsity or N:M patterns might allow semi-structured vectorization (e.g. NVIDIA’s 2:4 sparsity uses fixed masks to utilize tensor cores), but on general CPU there is no standard support for, say, 2-out-of-4 patterns – one would need to insert shuffle operations to realign nonzeros. Unstructured sparsity is the worst: no guarantee of contiguous nonzeros, so SIMD lanes frequently do wasted work or need zero-masking. In short, on CPUs/WASM: channel and block pruning can still use optimized GEMM kernels, whereas fine-grained sparsity typically does not speed up without custom kernel support.
• Index/mask overhead: Every sparse scheme needs metadata. Unstructured: each nonzero weight may require an index or mask bit, doubling memory for flags and complicating the multiply (need to check mask or load index pointers). N:M (e.g. 2:4) can encode mask bits compactly, but still requires a decode step for each block of 4 (like a small lookup table for each 4-element block). Block sparse (e.g. 4×4 blocks) requires row/column offsets per block. All of these increase scratch or unpacking time per layer. On the other hand, channel pruning has essentially no runtime metadata beyond the new dimension size – it’s pure dense math.
• Branching and early exit: Conditional execution methods (like early-exit classifiers or conditional skips) introduce branches into the inference loop. A branch misprediction penalty on CPU can offset its compute savings. In WASM, dynamic branching is allowed but may still stall pipelines. However, early-exit can be implemented as a simple if-statement on a confidence metric (cheap) followed by skipping layers (big savings), which can still be net-positive if the branch conditions are simple.
• WebAssembly constraints: WASM currently supports standard SIMD instructions (128-bit vectors), so it can accelerate dense math but not specialized sparse kernels. Memory is also more precious in-browser. Storing sparse metadata in WASM heap eats RAM and potentially triggers garbage collection if large. Thus in a WASM runtime like TinyRustLM, we expect structured reductions (channel, layer cutting) to yield real gains, whereas unstructured sparsity or N:M patterns likely increase code complexity and reduce speed.

Overall, structured compression (depth or channel) is most effective on CPU/WASM because it shrinks the main dense operations. Sparse formats must pay for index/mask overhead that often outweighs saved multiplications on these platforms. We also note that SIMD CPU speedups depend on block alignment (e.g. keeping dimensions a multiple of the vector width, typically 4 or 8 floats).

Honest Byte and Memory Accounting

We denote:

• $N_p$ = total number of model parameters (weights, biases, embedding entries).
• $B$ = bytes per parameter (e.g. 4 bytes for FP32, 2 for FP16, 1 for INT8).
• $N_{\text{shared}}$ = number of parameters shared by multiple sub-networks (deduplicated storage).
• $N_{\text{sparse\_mask}}$ = number of mask bits/entries needed for sparsity metadata.
• $d_{\text{model}}$ = model hidden size; $L_{\max}$ = max sequence length; $D_k=D_v=d_{\text{model}}/n_{\text{heads}}$ for attention heads.

Formulas:

$$\text{ParamBytes} = N_p \cdot B.$$ This includes all weight matrices, embeddings, biases, etc., as if each weight were independently stored.

• Logical parameter bytes (the “dry” count):

• Physical shared bytes: If several submodels share weights (e.g. an elastic network at different widths) or use weight sharing, the stored bytes may be less. For example, a “full model” plus a half-width model share the first half of the channels; those parameters are stored only once. Let $N_{\text{shared}}$ be the count of parameters physically stored once that serve multiple logical roles. Then the stored bytes = $N_{\text{shared}} \cdot B + (N_p - N_{\text{shared}}) \cdot B$. If no sharing, this equals ParamBytes; if full sharing among subnets, this equals ParamBytes of the largest net.

$$\text{KVBytes} = 2 \cdot L_{\max} \cdot d_{\text{model}} \cdot B.$$ (Factor 2 for keys and values). Note: If heads are pruned or $d_{\text{model}}$ is reduced, $d_{\text{model}}$ in the above reduces accordingly. Other dynamic memory (“scratch”) includes space for matrix multiply buffers and partial activations; worst-case this can be $O(d_{\text{model}}^2)$ temporarily, but we do not specify here.

• Runtime resident bytes: At inference, memory must hold (a) the static model parameters (as above) plus (b) dynamic state such as KV cache. A full KV cache of length $L_{\max}$ requires (for self-attention) storing keys and values for each token:

• Sparse metadata: For unstructured sparsity, one might store a bitmask per weight: $$\text{MaskBytes} = \lceil N_p / 8 \rceil.$$ For N:M sparsity, masks compress more (e.g. 2 bits per 4 weights = 0.5 bit per weight). Block sparsity (e.g. blocks of $b\times b$) needs indices per block: if there are $N_b$ blocks, overhead is $N_b \cdot \lceil\log_2(\text{possible positions in block})\rceil$. These bytes add on top of the weight bytes.

• Additional: Model storage often deduplicates sub-networks. For an elastic model family with $k$ widths, if weights are fully shared, the on-disk archive size is just that of the largest model plus small config data. If using separate checkpoints, sum all.

No invented throughput or memory numbers are given; these formulas show how each category contributes to the footprint. Crucially, note that reducing parameter count does not necessarily shrink Runtime resident bytes unless it actually shrinks $d_{\text{model}}$ or $L_{\max}$. A sparse model may have fewer nonzero weights but still occupies the same matrix shape (and thus the same dense allocation) unless special sparse GEMM is used.

Vocabulary & Embedding Reduction

For narrow specialists, it is tempting to drop unused tokens. Considerations:

• Tokenizer compatibility: Using a subset tokenizer requires either retraining the tokenizer or mapping out-of-domain tokens to an “unknown” ID. Sharing the champion’s tokenizer means carrying all tokenization logic, including punctuation and multilingual symbols, even if the specialist rarely uses them. A compromise is a two-stage approach: tokenize with full tokenizer, then filter vocabulary at the embedding level.
• Tied embeddings and output: If embeddings and the output softmax matrix are tied (common in many LMs), reducing vocabulary means resizing both matrices consistently. One must remove the corresponding rows/columns in the tied matrices. If they are not tied, one can potentially reduce only the input or only the output table, but this breaks symmetry and usually hurts performance.
• Unknown-domain fallback: A narrow specialist missing tokens for other domains will output an UNK or garbage for unseen words. In TinyRustLM’s design, such fallback should be handled by switching to the champion. This implies the specialist must detect out-of-domain input (e.g. a token not in its vocab) and defer (this can be done at the API level rather than in-model).
• Multilingual/code tokens: If the specialist is, say, English-only, we could drop tokens for other languages to shrink vocab. But this biases the model: it will be very poor on any input mixing languages. If the champion must cover those cases, the routing logic needs to catch them. Rare programming tokens (braces, operators) could also be omitted from an NLP specialist, reducing its embedding table size.
• Tying specialists to champion vocab: One approach is to keep the full champion vocabulary in the specialist model but freeze most embeddings (treat them as effectively unused) and not load them into memory. This complicates size accounting: “logical” vocab size is full, but “physical” storage for specialist need not reserve all embeddings if we lazy-load. However, our constraints treat static weights and embeddings as separate quotas. In practice, a smaller specialist should skip heavy embedding rows; the simplest technical solution is to have a smaller vocabulary and rely on champion fallback for unknowns.
• Performance tradeoff: Reducing vocab often has diminishing returns: the top ~50K tokens cover most English text. A specialist could keep, say, the top 20K tokens (especially if its tasks are specialized) to save memory. But any token outside that is OOV and must trigger fallback. If all tasks of the specialist are in-domain, this is acceptable; otherwise it risks losing context.

In summary, vocab pruning for a specialist is a form of structured pruning on the embedding/output weights. It must be paired with tokenizer handling (or fallback logic). It saves memory and compute (smaller softmax, smaller embedding matmul), but requires careful design to handle any tokens outside the specialist’s scope.

Expert Pruning, Merging, and Routed MoE

Expert Pruning vs Offloading: In MoE models, removing experts (entire expert sub-networks) is akin to channel or head pruning. _Expert pruning_ physically deletes experts and must adjust the router so it no longer selects them. This typically causes large performance loss if not handled. An alternative is offloading/cold storage: keep all experts in the model definition, but only load “hot” (frequently used) experts into RAM; inactive ones remain on disk/SSD and are fetched on demand. Offloading does not change the model’s structure – the router still sees all experts – but moves memory burden off the device. It can reduce resident memory without any re-training, at the cost of I/O latency.

Expert Merging (REAM) vs Pruning: The recent REAM approach shows merging weights of pruned experts yields better retention than outright deletion. In merging, two or more experts’ parameters are averaged or combined to form a new “super-expert,” preserving some knowledge from each. This produces a smaller expert set with less accuracy loss than pruning one expert completely. With pruning, the router often must be recalibrated or fine-tuned because its previous load balancing is disrupted. REAM effectively does “soft pruning”: experts are still used (in merged form) rather than dropped.

Router Recalibration: Anytime experts are pruned or merged, the gating network (router) must be adjusted. If the router is left unchanged, it may continue directing tokens to now-missing experts. Studies find that router retraining is essential for effective MoE compression: updating only the router (keeping experts frozen) often recovers much of the lost performance. (One can think of this as re-balancing which token goes to which remaining expert.) Full fine-tuning could be done, but a targeted Router Knowledge Distillation (training the router to match the old distribution) can suffice.

Load balancing: MoE routers often include a load-balancing loss to spread tokens across experts. Pruning disrupts this balance, so the router may collapse to using a few experts. Recalibration (or retraining with a balancing term) is needed to avoid degenerate load distributions.

Knowledge recovery and rare domains: Rare-domain or niche expertise often lives in a few “cold” experts. Pruning or merging them can disproportionately harm tasks involving those domains. Since rare-domain data is sparse, the router might scarcely have trained on it, making recovery harder. Expert merging (averaging specialists into generalists) may soften the loss, but some rare knowledge is inevitably diluted. There is no free lunch: removing experts means some knowledge is lost or will require heavier distillation into the remaining experts.

In summary, removing experts from MoE is nontrivial: one must retrain or recalibrate the router to use the new expert set effectively. Offloading (disk) is a pragmatic memory-saving step that preserves model semantics, whereas true pruning requires either retraining or a merging strategy (like REAM) to maintain quality.

Negative-Control Experiments

To illustrate the pitfalls of naive slicing, one should design an experiment where a dense model is arbitrarily truncated without recovery and evaluate its degradation. For example:

• Layer-by-layer removal: Take a 1B or 7B model and remove every 2nd transformer layer (or remove 20% of layers chosen uniformly at random). Do not fine-tune. Evaluate on standard benchmarks (language modeling or QA). We expect the model’s output to break down. Indeed, prior work shows that such “slice-and-run” yields enormous perplexity jumps.

• Attention-head pruning control: Delete half of the attention heads in each layer (for simplicity, contiguous halves). Again no retraining. On generation tasks the model should produce incoherent text. (One can compare base vs pruned answers to a simple prompt. The GLU-pruning blog provides a vivid example: pruning without respecting the gate structure turned “Paris is the capital of France” into “Paris is the capital of of France… the the the”.)

• Vocabulary slicing: Restrict the vocabulary to the top 10K tokens of a 30K-word model, without changing weights. On questions involving rarer words, the output will fail (e.g. outputting UNKs or nonsense). This will highlight that any text containing dropped tokens leads to garbage, rather than “filling in” missing words.

In all these controls, no recovery or fine-tuning is applied, so the poor quality is directly attributable to invalid slicing. The objective is to keep these failures visible, not hiding them behind output correction. For instance, one could plot perplexity or accuracy for various prune ratios: prior results show catastrophically rising perplexity for blind layer removal. Such a control experiment serves as a baseline, confirming that naive deletion is not viable without expensive downstream fixes.

TinyRustLM Decision Tree

For TinyRustLM’s constraints (browser-local, small footprint with champion+specialists), we propose:

1. Full Champion: If device memory and latency allow running the full champion reliably, use it for all queries. This avoids complexity. Only resort to submodels if constraints force it.
2. Domain check: If the input/query is from a well-defined narrow domain (language, code, specific genre) and we have a specialist for that domain:

• Use the specialist if it improves efficiency. Ensure the text tokens are largely within the specialist’s vocabulary; otherwise fallback to champion.
• Specialists should be independently distilled models trained on that domain (if labeled or unlabeled domain data is available). They carry a reduced architecture (e.g. fewer layers or tokens) tailored for speed.

1. Memory/Licensing limits: If model size must shrink (due to device constraints), consider a structurally pruned variant of the champion: retrain it via structured pruning + fine-tuning to a slightly smaller footprint. If retraining data isn’t available, consider mask-tuning or reward-based recovery. Stop pruning once quality drop exceeds a threshold. This is a pruned descendant approach.
2. Elastic subnetwork: If multiple capacity targets are needed simultaneously, or if on-device one wants flexibility (e.g. degrade quality gracefully under memory pressure), train a universal (slimmable) model once-for-all. In deployment, choose runtime width/depth based on device state. This requires a pre-trained elastic checkpoint (a single .slm artifact storing one weight set for all sub-sizes). Use this when you expect to need multiple execution modes (e.g. low-latency vs high-accuracy modes).
3. Adapter or PEFT: If only a small tweak is needed for a new task or domain, consider a trained adapter (LoRA) on top of a frozen base. This keeps the full champion but adds only a few small matrices per layer, reducing storage. Adapters are best when you want a quick adaptation without full retraining.
4. Specialist model: If a domain requires massive simplification (e.g. extremely limited token set or behavior), and you can train offline, build an independently distilled specialist. This is a separate smaller model (possibly with a smaller tokenizer). It trades universality for efficiency. Use it for that domain only.

Stop rules: We would stop compressing when (a) the model’s quality on held-out tasks falls below an acceptable threshold; or (b) further parameter reductions yield diminishing latency/memory gains. For nested training, we would stop adding subnetwork capacities when they no longer improve the Pareto front of latency vs accuracy. For pruning, stop when fine-tuning no longer recovers loss. In all cases, the champion remains the ultimate fallback if the submodel fails or produces untrustworthy output.

Unknowns and Further Experiments

Certain questions remain that likely require local prototyping:

• How does each pruning pattern actually affect first-token latency vs decode speed on WASM? (This would need benchmarking of kernels on the target platform.)
• What are the quality trade-offs for token-level pruning in practice? (One might run small-scale tests on a 2B model to see how much vocab can be cut before accuracy degrades beyond tolerance.)
• Can in-place distillation from a champion to a submodel in WASM be done efficiently during inference? (Probably not, but it’s an open idea.)
• Are there hybrid schemes (e.g. low-rank plus small dense remainder) that outperform pure channel pruning at TinyRustLM scales? Only empirical tests will say.

These and other platform-specific behaviors (cache effects, JIT behaviors in WASM, etc.) must be measured.

Annotated Bibliography

• Mengzhou Xia et al., “CoFi: Structured Pruning Learns Compact and Accurate Models” (ACL 2022). Proposes coarse+fine structured pruning for transformers, pruning layers, heads, and FFN dims with distillation recovery. Shows that removing neurons/heads/layers without recovery degrades accuracy. License: ACL (open access).
• Pere Martra et al., “Making LLMs Smaller Without Breaking Them: A GLU-Aware Pruning Approach” (Hugging Face Blog, Nov 2024). Demonstrates pruning in a 1B LLaMA model while respecting GLU gating and embedding structure. Contains illustrative examples of failed and successful pruning (e.g. output collapse when gating is misaligned). Blog (no formal license, CC BY likely).
• Xiaoxuan Niu et al., “COMP: Lightweight Post-Training Structured Pruning for On-Device LLMs” (arXiv 2025). Surveys structured vs unstructured pruning trade-offs. Introduces COMP (Hybrid layer+neuron pruning) with mask tuning recovery. Reports that structured pruning usually needs fine-tuning. License: ArXiv preprint (CC BY 4.0).
• Blanco et al., “REAM: Merging Improves Pruning of Experts in Mixture-of-Experts Models” (arXiv 2025). Presents REAM algorithm grouping and merging MoE experts instead of deleting them. Empirically shows merging preserves quality better than pruning alone. License: ArXiv preprint.
• Anonymous, “Is Retraining-Free Enough? The Necessity of Router Calibration for MoE Compression” (arXiv 2026). Argues that effective MoE pruning must involve recalibrating the gating network. Introduces Router-KD method (freeze experts, train router) to recover performance. License: ArXiv preprint.
• “Expert Streaming (Offload) Proposal” (GitHub issue, MLX, 2026). Proposal for on-demand loading of MoE experts from SSD for large models. Shows how offloading inactive experts allows running huge MoE models in limited RAM. License: Open source issue discussion.
• Ruisi Cai et al., “AdaPerceiver: Transformers with Adaptive Width, Depth, and Tokens” (arXiv 2025). Introduces a unified adaptive transformer and “Once-for-All” training across depth, width, and token axes. Relevant for elastic multi-dimension networks. License: ArXiv preprint.
• Jiahui Yu & Thomas Huang, “Universally Slimmable Networks” (ICCV 2019). Defines slimmable networks and the sandwich rule/in-place distillation for training one net at arbitrary widths. Although on vision models, the training principles directly apply to transformers. License: ICCV (IEEE, typically CC BY-NC). Also arXiv.
• Jacqueline Wang et al., “Position: Current Model Licensing Practices...” (ICSE 2024). Analyzes LLM licenses; notes that LLaMA’s licenses restrict derivative use. Cites Meta’s Llama2 license which forbids using Llama data to improve non-Llama models. License: ICSE (ACM).
• Min et al., “They’ve Stolen My GPL-Licensed Model!…” (ArXiv 2024). Survey of model licensing. Quotes Llama 2 Community License text granting rights to create derivatives (illustrating derivative permissions) and discusses CC-by, NC terms. License: ArXiv preprint.
• Google US20230419079A1, “Mixture of Experts Neural Networks” (patent, granted 2024). Describes MoE layers with gating subsystems that select and combine experts. Relevant for understanding patented aspects of MoE architectures. License: U.S. Patent (public).

Each source above is cited where relevant in the discussion. All dates and licenses are given where available; primary sources include conference/journal papers, arXiv preprints, and documentation in public repositories.