Bounded Training Strategy and Distillation Architecture for TinyRustLM

Full report

On this page

Recommended Decision and Primary Falsification

The deployment of a local, in-browser small language model (SLM) via the TinyRustLM ecosystem requires establishing a rigorous boundary between structural software engineering and model capability. The current prerelease architecture successfully executes WebAssembly (Wasm) and Windows .NET binaries, yet it lacks a qualified default model. Historical training runs of one and three epochs yielded validation losses of 4.2476 and 4.0866, respectively, but resulted in conversational degradation, including mishandling the user's name. This inverse relationship between loss and capability indicates a profound disconnect between the optimization objective and the inference-time conversational interface.

The recommended decision is to implement a bounded, code-invariant Low-Rank Adaptation (LoRA) strategy focusing strictly on Supervised Fine-Tuning (SFT), optionally followed by Simple Preference Optimization (SimPO). This approach completely bypasses the 200,000-record scale corpus until a 1,024-record CPU pilot validates the inference template constraints. The baseline model under consideration, Qwen3-0.6B, is architecturally sound for the target hardware profile but suffers from known chat template rendering inconsistencies that cause catastrophic history-tracking failures in strict inference environments.

The strongest reason this recommendation could fail is the phenomenon of Interface Injury, masked mathematically as capability loss1. If the local Rust inference engine parses the conversational prompt incorrectly—specifically regarding the injection or omission of structural tokens like \<think\> and \</think\> tags—the training loop will optimize for a token distribution that the Wasm inference engine can never faithfully reproduce. In this scenario, the model achieves artificially low training loss by memorizing broken templates, while outputting conversational static at runtime. The smallest experiment that could falsify the recommendation is taking a raw, untrained Qwen3-0.6B payload, manually injecting a five-turn mock conversation matching the exact byte-stream of template.template2, and observing the logits. If the model outputs garbage on the second turn, the hypothesis is falsified: training is the wrong lever, and the Rust runtime parser must be fixed before any parameter updates occur.

Project Context and Fact Base

To establish a bounded training pipeline, it is necessary to separate project-supplied facts, externally verified facts, hypotheses, and locally unverified conditions.

The project-supplied facts mandate that TinyRustLM executes via a Rust/Wasm pipeline in the browser and a Windows .NET companion. The system requires a working default model obtained automatically via MiniModel.org, which is explicitly authorized to supply initial server seeds. The current artifact composition tightly binds exactly six files: model.slm2, tokenizer.tokenizer2, template.template2, sampling.sampling2, prompt.prompt2, and composition.acg2. There is no established qualified default model or proven live initial seed in the supplied engineering checkpoint, and retired compatibility paths (like SLM1) must not be revived.

Externally verified facts confirm the structural parameters of the rejected baseline candidate, Qwen3-0.6B. It is a dense, decoder-only transformer with approximately 600 million parameters (0.44B non-embedding). The architecture relies on 28 hidden layers, Grouped Query Attention (GQA) with 16 query heads and 8 key-value (KV) heads, a hidden size of 1024, an intermediate MLP size of 3072, and tied embeddings2. The model supports a 32,768-token context length and utilizes RoPE positional encoding extended to a 1M base frequency5. Crucially, for in-browser execution, WebAssembly traditionally imposes a 32-bit linear memory limit, capping addressable space at 4GB6. However, the Wasm Memory64 proposal, currently supported in modern browsers, expands this ceiling to 16GB, avoiding the necessity to severely cripple or truncate the model to force it into standard Wasm memory profiles7.

A locally unverified condition—yet highly probable based on external tracking—is the presence of a specific chat template bug in the Qwen3-0.6B Hugging Face configuration. Known integration issues reveal that the template renders trailing assistant turns differently depending on whether subsequent user messages exist, inadvertently injecting or dropping \\n\\n byte pairs9.

The primary hypothesis driving this report is that the conversational degradation observed in the historical 4.0866-loss training run was not caused by catastrophic forgetting of general world knowledge, but rather by Template Drift. The format in which the model was historically fine-tuned likely deviated at the byte level from the template.template2 utilized by the Wasm inference runtime.

Comparison of Training Interventions

To deliver a working first-product default, a minimum viable approach must be selected from the landscape of post-training interventions. Each paradigm carries specific prerequisites and varying degrees of suitability for the TinyRustLM constraints.

Supervised Fine-Tuning (SFT) over the full parameter space requires high-quality prompt-completion pairs and substantial VRAM, typically scaling to 16 bytes per parameter just to hold the FP32 optimizer states10. This option is rejected for the primary phase. Full parameter updates risk catastrophic forgetting of the base model's reasoning capabilities and demand hardware footprints that are incompatible with rapid, potentially CPU-bound local pilots.

Low-Rank Adaptation (LoRA) or Quantized LoRA (QLoRA) requires the base weights and optimized kernels for low-rank matrix multiplication. This is the accepted primary intervention. LoRA freezes the original base weights and updates only a rank-decomposed set of matrices, strictly limiting the parameter update space. This isolates causal gains (e.g., learning the correct user name formatting) from broad parameter noise, restricts the optimizer memory footprint to a fraction of a gigabyte, and allows rapid iteration over the 1,024-record pilot.

Preservation Regularization demands a frozen copy of the base model in memory to compute Kullback-Leibler (KL) divergence against the training policy. This is conditionally rejected for the immediate pilot. While theoretically useful for preventing capability drift, if the base model mishandles reasoning tokens or suffers from the template.template2 bug, strict preservation KL will actively protect the defect, penalizing the optimizer for correcting the model's interface geometry.

Sequence-Level Distillation (off-policy) relies on vast datasets of teacher-generated sequence trajectories. This is rejected for the first-product problem. Off-policy distillation induces severe train-test mismatch; the student is forced to match teacher trajectories but lacks the self-correction mechanisms to recover when it inevitably deviates into its own latent space during local autoregressive generation11.

Token-Level Distillation (on-policy) requires live access to a massive teacher model's logits during the student's training loop, alongside complex tokenizer alignment algorithms. This is strictly rejected. It violates the constraint of establishing a simple, bounded local training environment, as it requires loading an impossibly heavy teacher model (e.g., 70B parameters) into the local or pilot pipeline, vastly exceeding any hypothetical local hardware budget.

Standard Preference Optimization (such as Direct Preference Optimization, or DPO) requires preference datasets comprising chosen and rejected responses, along with a frozen reference policy model. This is rejected due to the hardware overhead of maintaining the reference model in memory during optimization, effectively doubling the activation and parameter footprint.

Simple Preference Optimization (SimPO) serves as the accepted alternative phase. SimPO requires preference datasets but completely eliminates the need for a reference model. It formulates the reward directly from the policy model's length-normalized average log-probability13. This makes it significantly more compute- and memory-efficient while still allowing the system to strongly penalize specific defects (e.g., misnaming the user) without brute-forcing supervised data.

Derivation of Training Objectives and Mechanics

The chosen training objective must be precisely defined and configured to prevent the model from optimizing the wrong mathematical levers, a frequent cause of superficially low loss paired with degraded conversational output.

During standard SFT, the objective is to minimize the negative log-likelihood of the target tokens. However, the masking strategy is critical. Prompt masking is an absolute non-negotiable requirement. The loss must only be calculated on the tokens belonging to the assistant's completion. If the prompt tokens (the user's input) are not masked, the model will expend its limited gradient capacity attempting to predict what the user will say next, rather than learning how to respond. This directly causes the "endless babbling" or repetitive hallucination loops commonly observed in poorly tuned models.

To maximize throughput on constrained hardware, sequence packing is employed. Multiple independent training examples (e.g., prompt-completion pairs) are concatenated into a single continuous sequence up to the model's maximum context boundary. To preserve statistical independence, the attention mask must be strictly modified into a block-diagonal format. Token [Figure omitted from source export] from training record A must have its attention score forced to negative infinity with respect to token [Figure omitted from source export] from training record B, preventing cross-contamination of contexts. Furthermore, loss normalization must be executed on a per-token basis rather than a per-sequence basis14. Normalizing per-sequence artificially inflates the gradient magnitude of extremely short responses, warping the model's learned length prior and causing it to systematically output truncated, unhelpful answers.

Padding tokens must carry a label of \-100 (under standard neural network framework semantics) to ensure they are ignored by the Cross-Entropy loss calculation. The precise SFT Cross-Entropy objective for a packed sequence of length [Figure omitted from source export] containing a masked prompt of length [Figure omitted from source export] is derived as:

[Figure omitted from source export]

Precision must be maintained carefully. Forward and backward passes should utilize bfloat16 (BF16) to prevent underflow, while optimizer states (the momentum and variance tracking inside AdamW) must remain in FP32 to maintain the numerical stability of the minute gradient updates15. Given the small batch sizes necessitated by local pilots, gradient accumulation is required. If the hardware can only process a batch size of 1, accumulating gradients over 16 steps simulates an effective batch size of 16\. Gradients must be mathematically scaled (divided by the accumulation factor) prior to the optimizer step. Additionally, global gradient clipping is mandatory. The L2 norm of the gradients should be clamped (typically to a threshold of 1.0) to prevent anomalous, high-magnitude gradient spikes from permanently scarring the frozen base model's representations through the LoRA matrices16.

For the subsequent preference optimization phase, SimPO leverages a length-normalized reward formulation. Given a context [Figure omitted from source export], a winning completion [Figure omitted from source export], and a losing completion [Figure omitted from source export], the implicit reward is:

[Figure omitted from source export]

The SimPO loss objective incorporates a target reward margin [Figure omitted from source export] to enforce a definitive separation between preferred and rejected behaviors, driving the model away from producing the user's incorrect name:

[Figure omitted from source export]

where [Figure omitted from source export] scales the reward magnitude and [Figure omitted from source export] represents the sigmoid function13.

Preservation KL: Protection versus Defect Preservation

The existing pinned plan relies on same-base preservation KL. The direction and application of this KL divergence dictate whether it acts as a protective shield for general capability or a restrictive anchor that preserves the rejected base model's defects.

Preservation KL computes the Kullback-Leibler divergence between the logits of the training policy model ([Figure omitted from source export]) and a frozen copy of the original base model ([Figure omitted from source export]). The total loss becomes a weighted sum:

[Figure omitted from source export]

This mechanism protects general capability by anchoring the model's logits on vast expanses of pre-trained world knowledge, syntactic structures, and logical reasoning pathways that are absent from a narrow 1,024-record conversational pilot. However, it preserves defects if the base Qwen3-0.6B model inherently hallucinates template tags, fails instruction-following, or exhibits the known \<think\> token rendering bugs. In these cases, forcing the student to match the base model's exact probability distribution will actively resist the SFT updates designed to fix the interface, leading to stagnant optimization.

The direction of the KL divergence fundamentally alters the learning dynamics. Forward KL, expressed as [Figure omitted from source export], is inherently mode-covering. It penalizes the student heavily if it assigns low probability to any token supported by the teacher. Because a sub-1B parameter SLM lacks the capacity to memorize the full distributional complexity of its training source, attempting to cover all modes spreads probability mass into the "void" between valid modes, resulting in hallucination and incoherent, average phrasing12.

Conversely, Reverse KL, expressed as [Figure omitted from source export], is mode-seeking17. It encourages the student to concentrate its probability mass on the single most likely trajectory of the teacher, dropping alternative modes entirely. While this ensures coherent and stable generative paths, it can lead to "mode collapse," where the model loses all conversational diversity and repeatedly outputs the exact same robotic phrasing18.

To diagnose and manage the loss weighting ([Figure omitted from source export]), a static, frozen recipe must be avoided. The weight must be dynamically monitored. If the task loss ([Figure omitted from source export]) plateaus prematurely while the KL loss drops toward zero, it indicates that the model has abandoned learning the new conversational data in order to satisfy the rigid preservation constraint. The proposed divergence from the existing frozen recipe is to implement a linear warmup for [Figure omitted from source export], annealing it from 0 to its maximum value over the first 10% of training steps, allowing the model to adapt its interface before the regularization penalty fully anchors it.

Distillation Constraints and Tokenizer Mismatch

If the engineering roadmap eventually transitions from self-correction to true knowledge distillation (e.g., distilling reasoning pathways from a larger 32B model into Qwen3-0.6B), the pipeline will encounter the severe obstacle of Tokenizer Mismatch.

Standard token-level white-box distillation assumes that logit position [Figure omitted from source export] in the teacher corresponds perfectly to logit position [Figure omitted from source export] in the student. When tokenizers diverge, this alignment shatters. A larger model might tokenize a complex word into a single subword, while Qwen3's specific 151,669-token byte-level BPE tokenizer might split it into three separate bytes2. Attempting direct KL divergence across these mismatched vocabularies results in suppressive gradients that destroy the student's semantic mapping.

To execute distillation safely, the system must employ a deterministic cross-tokenizer projection framework, such as the X-Token methodology19. This involves three distinct steps. First, Dynamic Programming (DP) span alignment must be used to group disparate tokens from both models into text-consistent chunks that decode to the exact same underlying string20. Second, a sparse projection matrix [Figure omitted from source export] is constructed to map the student-token probabilities into the teacher's vocabulary space. Finally, the P-KL objective formulation is applied. Instead of attempting to partition tokens into common and uncommon sets, P-KL projects the student's full distribution via matrix [Figure omitted from source export] and computes the KL divergence directly20.

This methodology preserves the "dark knowledge" (the residual probability mass distributed across alternative, semantically valid tokens) without penalizing the student for structural tokenization differences. Furthermore, hidden-reasoning exclusion must be managed. If the teacher utilizes internal \<think\> reasoning traces that the tiny student lacks the parameter budget to replicate, the teacher's confidence over these spans must be masked out of the distillation objective, restricting the divergence calculation solely to the finalized, user-facing output. In such cases, sequence-level targets (where the student trains on the finalized text of the teacher) become more defensible than strict logit distribution matching, as they avoid forcing the student to emulate cognitive architectures it does not possess.

Diagnosing the Loss versus Conversation Paradox

The phenomenon where a three-epoch training run achieves a lower mathematical loss (4.0866) than a one-epoch run (4.2476) yet yields profoundly worse conversational behavior is a classic presentation of Interface Injury, compounded by training-evaluation mismatch.

Perplexity and Cross-Entropy loss are evaluated via teacher-forcing; the model predicts the next token given a sequence of perfect, ground-truth history. However, during live in-browser execution, inference is autoregressive. The model must consume its own generated tokens. If the model slightly overfits to a specific, rigid formatting quirk in the training data, its training loss will plummet. Yet, at runtime, a single deviated token throws the model off its memorized manifold, leading to cascading errors that the teacher-forcing loss metric completely fails to capture.

This discrepancy stems from several specific failure modes. The foremost is Template Drift. The official Qwen3-0.6B chat template relies on complex Jinja rendering logic that dynamically altering trailing assistant turns based on the presence of subsequent user messages9. If template.template2 and prompt.prompt2 in the TinyRustLM ecosystem enforce strict, static byte-sequence matching, the model will encounter an out-of-distribution sequence on the very second conversational turn. It expects a specific \\n\\n structure that the Wasm parser drops. The model then hallucinates or crashes, despite having excellent theoretical perplexity.

The second failure mode is Label Stickiness, a direct symptom of interface injury. When a base model undergoes poorly bounded fine-tuning, the fragile interface that maps latent semantic knowledge onto output tokens is damaged before the underlying knowledge itself degrades. Research on Qwen3 linearization demonstrates that models suffering from this injury will systematically ignore the semantic content of a prompt and instead stick to structural labels—for instance, repeatedly outputting "A" in multiple-choice scenarios regardless of how the options are rotated, or infinitely repeating XML tags without generating content1. The model knows the answer, but the "wiring" to output it has been severed by overzealous gradient updates.

Finally, catastrophic forgetting of the stop token causes seemingly nonsensical conversational decay. If the 1,024-record dataset is improperly packed and fails to consistently append the exact End-Of-Sequence (EOS) token defined in tokenizer.tokenizer2, the model's loss will decrease regarding text generation, but it will never learn when to stop emitting tokens in the browser, leading to endless, trailing babble that ruins the user experience.

Hardware Cost Estimation and Memory Formulas

Moving from arbitrary trials to a bounded engineering discipline requires explicit resource estimation for the 1,024-record CPU pilot and any future optional GPU scaling. No local GPU is confirmed, so the analysis spans both domains.

The memory footprint of training is dictated by the optimizer, the precision format, and the parameter count. For a model with [Figure omitted from source export] parameters trained using AdamW in standard mixed precision, the total static memory requirement is driven by four primary tensors21:

1. Model Weights (BF16): [Figure omitted from source export] bytes.

2. Parameter Gradients (BF16): [Figure omitted from source export] bytes.

3. Master Weights (FP32): [Figure omitted from source export] bytes (required for stable accumulation).

4. Optimizer States (AdamW FP32): [Figure omitted from source export] bytes (comprising 4 bytes for the first moment/mean, and 4 bytes for the second moment/variance).

Summing these components, full parameter fine-tuning demands a static memory floor of [Figure omitted from source export] bytes10. For the Qwen3-0.6B model ([Figure omitted from source export]), the math resolves to:

[Figure omitted from source export]

Adding dynamic activation memory (approximately 1.5 GB for a context of 1,024 and batch size of 1), full fine-tuning requires over 11.1 GB of RAM/VRAM.

Implementing LoRA drastically alters this equation. By freezing the base weights, the FP32 master weights, gradients, and AdamW optimizer states are required solely for the injected low-rank matrices, which typically constitute less than 2% of the total parameter count22.

• Frozen Base Weights (BF16): [Figure omitted from source export] GB.
• LoRA Weights, Gradients, and States: [Figure omitted from source export] GB.
• Total LoRA Static Memory: [Figure omitted from source export] GB. This comfortably fits within tight memory envelopes, making the 1,024-record pilot highly viable.

Wall-clock execution costs on CPU hardware expose the futility of running the 200,000-record corpus locally. On an explicit hypothetical hardware setup—an 8-core consumer desktop CPU without AVX-512 extensions—matrix multiplication for a 0.6B parameter model will yield approximately 3 to 5 tokens per second during the combined forward/backward pass. Processing the 1,024-record pilot, assuming an average length of 1,024 tokens per record, requires processing roughly 1,048,576 tokens. At 4 tokens per second, a single epoch requires approximately 72 hours of continuous CPU execution. While acceptable for a weekend pilot run to validate binary formats, scaling this to 200,000 records would require over 1.6 years. Therefore, execution of the scale corpus inherently requires a dedicated GPU (e.g., an RTX 3090 24GB or cloud equivalent), which must be formally budgeted and authorized once the pilot proves successful.

Ablation Matrix and Experiment Control

To isolate causal gains from noise and prevent an uncontrolled combinatorial search, the engineering team must execute the following strict ablation matrix on the 1,024-record pilot. All outcomes must be preserved, avoiding the trap of selectively archiving only favorable seeds.

Checkpoint Selection and Code-Invariant Adapter Merging

The management of model payloads in D:\\LLMs\\TinyRustLM requires strict hygiene. Periodic historical model piles and duplicated adapters scattered across multiple folders must be prohibited.

The retention rule dictates maintaining exactly one resumable payload directory (Staging). Once a replacement model passes the acceptance gate, the adapter must be merged into the base model to yield a standalone, distributable model.slm2 file, after which all superseded historical payloads must be immediately deleted.

The merging process itself presents a severe risk if the target deployment relies on a quantized format (e.g., 4-bit integer weights). Merging a high-precision FP16 or BF16 LoRA adapter directly into quantized base weights necessitates re-quantizing the entire merged matrix. This mathematically re-derives the E2M1 code plane, injecting fresh quantization error that can permanently destroy the delicate alignments learned during the SFT phase, effectively deleting the adaptation23.

To prevent this, the deployment must mandate Scale-QLoRA / Code-Invariant Adapter Merging24. Under this paradigm, the LoRA adapters do not update the integer codes. Instead, they learn low-rank corrections applied strictly to the native per-block scaling factors (which exist in FP16). When the adapter is merged, the E2M1 code plane remains entirely frozen, and only the scale bytes are overwritten. This guarantees a bit-exact, code-invariant checkpoint that preserves the exact conversational behaviors learned during training, requiring no revalidation of the quantization grid prior to browser deployment.

Required Decision Artifacts and Implementation Sequence

The following artifacts provide the actionable blueprint for the engineering team.

Provisional Training Recipe (Assumptions: CPU/Low-VRAM GPU, BF16, LoRA rank 16)

• Base: Qwen3-0.6B (Frozen).
• Target: template.template2 flattened to static ChatML.
• Method: LoRA SFT (Rank 16, Alpha 32).
• Optimization: AdamW (FP32 states), Learning Rate 2e-5, Cosine Decay schedule.
• Batching: Micro-batch size 1, Gradient Accumulation 16\.
• Masking: Strict prompt masking (labels set to \-100). Loss normalized per-token.

Two Alternatives:

1. If conversational coherence is achieved but specific facts (like the user name) fail: Execute SimPO over 500 targeted preference pairs, bypassing standard SFT for those specific behaviors.

2. If LoRA fails to map the template: Switch to full parameter SFT exclusively on the embedding and language model head layers, keeping the core transformer blocks frozen.

Pseudocode for the Critical Training Loop

 

 

 

Python

def compute\_masked\_sft\_loss(logits, targets, mask\_indices):     \# Enforce strict prompt masking to prevent conversational looping     targets\[mask\_indices\] \= \-100     \# Normalize per-token, ignoring the masked indices     loss \= cross\_entropy(         logits.view(-1, vocab\_size),          targets.view(-1),          ignore\_index=-100     )     return loss

def execute\_bounded\_training\_loop(model, dataloader, optimizer, accumulation\_steps=16):     model.train()     optimizer.zero\_grad()          for step, batch in enumerate(dataloader):         inputs \= batch\['input\_ids'\]         targets \= batch\['labels'\]         prompt\_mask \= batch\['prompt\_mask'\]                  with autocast(dtype=bfloat16):             outputs \= model(inputs)             loss \= compute\_masked\_sft\_loss(outputs.logits, targets, prompt\_mask)             \# Scale loss to account for gradient accumulation             loss \= loss / accumulation\_steps                      loss.backward()                  if (step \+ 1) % accumulation\_steps \== 0:             \# Enforce global gradient clipping to protect base representations             clip\_grad\_norm\_(model.parameters(), max\_norm=1.0)             optimizer.step()             optimizer.zero\_grad()

Evaluation Protocol and Broad-Regression Checks

Before any browser default can be claimed, the model.slm2 payload must undergo an automated regression check. This protocol involves submitting 50 static, pre-computed reasoning prompts (evaluating math, logic, and summarization) to both the original base model and the new merged model. The perplexity variance between the two outputs must remain below 0.2. A variance greater than this indicates that the LoRA adapter has caused catastrophic forgetting, triggering an immediate rejection of the payload.

What to Stop Doing

1. Stop executing multi-epoch scale training (the 200,000-record run) until the Wasm inference integration is proven flawless via the 1,024-record pilot.

2. Stop relying on training loss as an acceptance gate. Training loss measures dataset memorization, not interface integrity or runtime conversational fluidity.

3. Stop ignoring the chat template bug. The dynamic Jinja logic associated with Qwen3's \<think\> tags must be flattened into a static structure inside template.template2 to prevent the Rust parser from failing on multi-turn context.

Compact Experiment-Lesson Template

For every iteration executed within D:\\LLMs\\TinyRustLM, the operating engineer must record the outcome using this exact format to prevent institutional memory loss:

• Question: Does increasing LoRA rank from 8 to 16 prevent the model from hallucinating the user's name during Turn 3?
• Exact Inputs: D:\\LLMs\\TinyRustLM\\Pilot\_1024\_v1.jsonl, Base \= Qwen3-0.6B.
• Method: LoRA SFT, BF16, Rank 16, Alpha 32, LR 2e-5, 1 Epoch.
• Result: Validation loss 4.10. Automated inference outputs the name correctly on 94/100 test queries.
• Uncertainty: It remains unverified if the 6% failure rate stems from inadequate dataset diversity or inherent context-window truncation limits.
• Decision: Promote Rank 16 configuration to the standard staging baseline.
• Reusable Lesson: Rank 8 lacks sufficient parameter capacity to successfully map the required persona formatting over the Qwen3 base logic; rank 16 provides the necessary dimensionality.
• Evidence Identity: Git commit hash \[a1b2c3d\], resulting composition.acg2 SHA256 \[hash\].

Works cited

1. Diagnosing and Repairing Interface Injury in Attention-to-KDA, https://arxiv.org/pdf/2608.02689

2. Qwen3 Technical Report \- arXiv, https://arxiv.org/html/2505.09388v1

3. README.md · Qwen/Qwen3-0.6B-Base at, https://huggingface.co/Qwen/Qwen3-0.6B-Base/blob/2dbaa23833ca4f21ecbf38999b27ba8d3cebedde/README.md

4. config.json · Qwen/Qwen3-0.6B-Base at main \- Hugging Face, https://huggingface.co/Qwen/Qwen3-0.6B-Base/blob/main/config.json

5. Qwen3-0.6B: Compact Dense Transformer LLM \- Emergent Mind, https://www.emergentmind.com/topics/qwen3-0-6b

6. Memory64 in 2026: Breaking the 4GB Ceiling in Browsers and Servers, https://wasmhub.dev/blog/memory64-in-2026

7. Memory64: Unlocking WebAssembly's True Potential with 16GB of, https://www.scichart.com/blog/memory64-unlocking-webassemblys-true-potential-with-16gb-of-in-browser-memory/

8. Wasm 3.0 Completed \- WebAssembly, https://webassembly.org/news/2025-09-17-wasm-3.0/

9. Qwen3-0.6B bundle: second message on any conversation fails with, https://github.com/google-ai-edge/LiteRT-LM/issues/3443

10. GRPO VRAM & GPU Sizing: Policy, Reference and Reward \- Lyceum, https://lyceum.technology/magazine/grpo-vram-gpu-sizing-policy-reference-and-reward/

11. MiniLLM: Knowledge Distillation of Large Language Models \- arXiv, https://arxiv.org/html/2306.08543v3

12. A Survey of On-Policy Distillation for Large Language Models \- arXiv, https://arxiv.org/html/2604.00626v1

13. SimPO: Simple Preference Optimization with a Reference-Free, https://arxiv.org/html/2405.14734v1

14. keval-sha/medgemma-cardiac-training-plan \- Hugging Face, https://huggingface.co/keval-sha/medgemma-cardiac-training-plan

15. GPU Memory Calculator Guide for Deep Learning (2026) \- Lyceum, https://lyceum.technology/magazine/gpu-memory-calculator-deep-learning/

16. Stable Low-Precision Training with Combined Adam–Madam Updates, https://openreview.net/pdf?id=NKT7LFio6e

17. MiniLLM: On-Policy Distillation of Large Language Models \- arXiv, https://arxiv.org/pdf/2306.08543

18. Beyond Mode Collapse: Distribution Matching for Diverse Reasoning, https://arxiv.org/html/2605.19461v1

19. X-Token: Projection-Guided Cross-Tokenizer Knowledge Distillation, https://arxiv.org/html/2605.21699v1

20. NVIDIA Introduces X-Token: Projection-Guided Cross-Tokenizer KD, https://www.marktechpost.com/2026/05/29/nvidia-introduces-x-token-projection-guided-cross-tokenizer-kd-that-outperforms-gold-by-3-82-average-points-on-llama-3-2-1b/

21. A Practical Journey into Distributed Training with PyTorch and Ray, https://debnsuma.github.io/my-blog/posts/distributed-training-from-scratch/

22. GPU VRAM Requirements to Fine-Tune LLMs in 2026 \- Spheron, https://www.spheron.network/blog/gpu-vram-requirements-fine-tune-llm-2026/

23. CellFill — Bounded In-Cell Learning for Bit-Identical, Revocable, https://arxiv.org/html/2608.20873v1

24. Code-Invariant Adapter Merging for Native 4-bit Microscaling LLMs, https://arxiv.org/pdf/2609.04526

25. Code-Invariant Adapter Merging for Native 4-bit Microscaling LLMs, https://arxiv.org/abs/2609.04526