TinyRustLM: Comprehensive Training Architecture and Alignment Strategies for 0.5B-2B Local Models

Full report

On this page

Executive Decision and Structural Routing

The evolution of local, in-browser artificial intelligence necessitates language models that operate under severe computational constraints while delivering nuanced, logical, and context-aware interactions. Based on the technical evaluation of the TinyRustLM ecosystem conducted on July 11, 2026, an operational decision must be formulated regarding whether to train a new model from random initialization, continue the pretraining of an existing permissive base, or restrict adaptation to parameter-efficient fine-tuning. The TinyRustLM runtime is strictly bound by a WebAssembly (WASM) scalar CPU execution environment, a 128 MiB single-transfer allocation ceiling, a 1,024 sampling candidate cap, and a custom SLM1 binary format that currently lacks native grouped-query attention (GQA) optimizations and multithreading1. The executive recommendation mandates the adoption of a Continued Pretraining (CPT) pipeline applied to a highly capable 0.5B to 1.5B parameter base checkpoint licensed under permissive terms (such as Apache 2.0). Training a model from random initialization in the 0.5B-1.5B regime demands an exorbitant token budget—often between 3 trillion and 18 trillion tokens—to overcome the scaling inefficiencies of small dimensionalities3. Pushing a model of this size to such extremes introduces catastrophic overtraining risks, where downstream adaptability is severely degraded, and the network suffers from progressive sensitivity and catastrophic forgetting during instructional alignment5. Conversely, relying solely on an unadapted base model fails to meet the strict conversational logic, neutral adult-policy inquiry, and tool-delegation requirements necessary to comply with the UAIX Cognitive Liberty Charter9. Therefore, the optimal path is a phased adaptation strategy: CPT to inject domain-specific structural logic and execute vocabulary pruning, followed by Full-Parameter Supervised Fine-Tuning (SFT) for tool-use formatting, and culminating in reference-free preference optimization using Simple Preference Optimization (SimPO) or Odds Ratio Preference Optimization (ORPO) to stabilize behavior. A rigorous decision tree is established to guide ongoing and future iterations:

1. Assess Base Reasoning: If the permissive base model demonstrates fundamental logical and arithmetic deficits that cannot be bypassed via external tool delegation, route to sequence-level and token-logit teacher distillation.
2. Assess Policy Compliance: If the base model demonstrates competent reasoning but refuses legal adult inquiries or issues paternalistic disclaimers, route to CPT combined with ORPO to aggressively override the pre-existing alignment tax.
3. Assess Hardware Constraints: If target device hardware transfer constraints strictly bind below 128 MiB, prioritize extreme vocabulary pruning, discard the embedding tail, and execute 4-bit Quantization-Aware Fine-Tuning (QAFT) prior to SLM1 packaging.
4. Assess Adapter Viability: If rapid, task-specific swapping is required within the WASM runtime without reloading the base weights, route to Weight-Decomposed Low-Rank Adaptation (DoRA) and adapter composition.

State of the Art: Updates and the Always-Current Rule

In adherence to the always-current rule, the training landscape as of July 2026 reflects significant departures from the paradigms of the early 2020s. The traditional Chinchilla-optimal scaling laws, which advocated for a 20:1 token-to-parameter ratio, have been superseded in the sub-2B parameter regime4. Modern small models are deliberately overtrained to prioritize inference efficiency over training efficiency. However, the discovery of "catastrophic overtraining" dictates that beyond an inflection point (roughly 2.5 trillion tokens for a 1B model), the parameters become increasingly fragile, reducing the model's plasticity for downstream fine-tuning5. In the realm of parameter-efficient fine-tuning (PEFT), traditional Low-Rank Adaptation (LoRA) has been largely supplanted by Weight-Decomposed Low-Rank Adaptation (DoRA), which decouples the magnitude and directional updates of weight matrices, allowing for learning capacities that mirror full fine-tuning without adding inference latency15. Furthermore, preference optimization has evolved away from the memory-heavy Direct Preference Optimization (DPO)—which requires a frozen reference model—toward reference-free methodologies such as SimPO, KTO, and ORPO, drastically reducing VRAM requirements and preventing length-bias exploitation18. Finally, the TinyRustLM ecosystem relies on a custom SLM1 binary format rather than standard GGUF; the SLM1 header strictly requires equal attention and key-value (KV) head counts, actively rejecting the Grouped-Query Attention (GQA) optimizations prevalent in models like LLaMA-32.

Quantitative and Operational Comparison of Adaptation Regimes

1. Training 0.6B, 1B, and 1.5B Dense Models from Random Initialization

Training from random initialization provides absolute control over the data mixture, tokenizer formulation, and layer normalization structures. It ensures the complete absence of unwanted alignment biases or toxic pre-training artifacts. However, achieving baseline linguistic fluency and logical coherence in the 0.5B-1.5B regime requires an immense computational investment. A 1B parameter model trained on 3 trillion tokens requires approximately [Figure omitted from source export] FLOPs24. Operationally, this demands sophisticated deduplication infrastructure, exact cross-lingual balancing, and complex phase-decay learning rate schedulers to prevent loss spikes. While possible to execute, it is highly unlikely to outperform an existing permissive base model that has already absorbed the massive fixed costs of foundational language modeling, especially given the risks of catastrophic overtraining rendering the final artifact brittle5.

2. Continued Pretraining of a Permissively Licensed Base Checkpoint

Continued Pretraining (CPT) leverages the established syntactic and semantic foundations of an existing model, requiring only 50 billion to 200 billion tokens to shift the model's domain knowledge, formatting expectations, and tokenizer mappings25. CPT is computationally efficient, typically requiring less than 50 GPU-days for a 1B model. The operational challenge lies in maintaining the original capability while injecting new reasoning pathways. This is achieved by utilizing a replay buffer consisting of 15-20% high-quality general pretraining data mixed with the new domain data, effectively anchoring the previously learned representations. CPT is highly recommended for TinyRustLM to embed the UAIX Cognitive Liberty principles deeply into the base weights before superficial instructional alignment.

3. Full-Parameter Supervised Fine-Tuning

Full-parameter SFT uniformly updates all network weights to map specific user intents to precise outputs. It is highly effective at establishing conversational tone, ensuring strict format adherence (such as returning rigid JSON schemas), and teaching the model to emit syntax tokens that delegate arithmetic to an external deterministic calculator. However, SFT is fundamentally limited by the pre-existing knowledge of the base model. If the base model lacks foundational logical capabilities, SFT will result in an artifact that confidently hallucinates, as it cannot organically learn robust new reasoning pathways from limited demonstration pairs.

4. Parameter-Efficient Tuning and Adapter Composition

Techniques such as LoRA, QLoRA, and IA3 reduce the trainable parameter count by introducing low-rank matrices or learned vectors, leaving the massive base matrices frozen. Weight-Decomposed Low-Rank Adaptation (DoRA) enhances this paradigm by separating the pre-trained weights into magnitude and direction components, applying LoRA strictly to the directional updates15. DoRA consistently closes the accuracy gap with full fine-tuning by allowing the magnitude vector to scale independently from the low-rank directional shift, ensuring stable learning dynamics16. Furthermore, adapter composition allows multiple DoRA modules to be trained independently for specific skills (e.g., summarization, retrieval grounding) and dynamically routed or merged during inference, preventing interference between tasks and facilitating a highly modular architecture within the 128 MiB TinyRustLM memory ceiling26.

5. Sequence-Level Teacher Distillation

Sequence-level distillation involves training the student model (TinyRustLM) on the raw textual outputs generated by a significantly larger teacher model (e.g., a 70B parameter frontier model). This method efficiently transfers conversational style and step-by-step reasoning protocols. Operationally, it is simple to implement via standard SFT pipelines. However, it suffers from a critical limitation: the student often learns to mimic the teacher's authoritative tone without actually inheriting its internal logical consistency. This leads to "sycophancy" and severe hallucination on out-of-distribution reasoning tasks, as the student lacks the parametric depth to support the complex deductions it has been trained to output.

6. Token-Logit or Distribution Distillation

Token-logit distillation transfers the exact probability distributions from the teacher to the student, providing a vastly richer gradient signal than hard textual labels. To manage the massive storage requirements of offline logit distillation across billions of tokens, advanced compression toolkits are employed. Frameworks like DistillKit store only the top-K logits (sparse distribution), sort them by log-probability, and fit a mathematical polynomial curve to approximate the tail of the distribution29. The residuals between the actual logits and the polynomial curve are subjected to error-diffusion quantization and packed at the bit level, compressing the distribution storage down to roughly 114 to 300 bytes per token29. This highly efficient storage mechanism allows a 1B student to align deeply with a teacher's uncertainty, secondary predictions, and common-sense probabilistic mappings, making it a strictly superior operational choice to sequence-level distillation30.

7. Hidden-State, Attention-Map, or Intermediate-Feature Distillation

Hidden-state distillation attempts to align the internal activations and attention matrices of the student with those of the teacher. While theoretically powerful for transferring deep logical structures, it is practically prohibitive when the student and teacher have vastly different dimensionalities, layer counts, and attention mechanisms. For example, mapping a 70B teacher's hidden states to a 1B student requires complex, computationally expensive projection layers that often destabilize training31. Crucially, because the TinyRustLM SLM1 header strictly requires equal attention and KV-head counts (rejecting GQA), aligning its attention maps with a modern GQA-based teacher model is structurally incompatible2.

8. Preference Optimization: DPO, ORPO, KTO, and SimPO

Preference optimization eliminates the need for a separate reinforcement learning reward model by optimizing directly on paired or binary preference data.

• Direct Preference Optimization (DPO) utilizes a fixed reference model to compute a log-ratio penalty. While effective, it doubles the memory footprint during training, requiring both the policy and reference models to reside in VRAM18.
• Simple Preference Optimization (SimPO) eliminates the reference model entirely. It utilizes the average log probability of a sequence as the implicit reward, incorporating length normalization [Figure omitted from source export] and a target margin [Figure omitted from source export] to ensure the winning response outscores the losing response without artificially inflating the length of the generated text19.
• Odds Ratio Preference Optimization (ORPO) combines SFT and preference alignment into a single monolithic stage by appending a log odds ratio penalty to the standard negative log-likelihood loss, making it exceptionally resilient to highly imbalanced data where negative examples far outweigh positive ones21.
• Kahneman-Tversky Optimization (KTO) operates on a binary signal (desirable vs undesirable) rather than paired preferences. It utilizes a human value function modeled on prospect theory to heavily penalize losses (undesirable outputs) while exhibiting diminishing sensitivity to gains, closely mirroring human cognitive biases regarding risk22.

For TinyRustLM, SimPO and ORPO are the recommended methodologies, as they sidestep the memory overhead of DPO while providing robust resistance against untrusted memory injection.

9. Quantization-Aware Fine-Tuning and Post-Quantization Recovery

Because the target TinyRustLM runtime is strictly bound by a 128 MiB transfer limit and exclusively supports f32, q8\_0, and q4\_0 within the SLM1 container, post-training quantization often destroys the carefully tuned logits of small models1. Quantization-Aware Fine-Tuning (QAFT) simulates lower precision during the forward pass, forcing the model weights to converge to a state resilient to quantization noise. Advanced methods such as IR-QLoRA utilize Information Calibration Quantization to ensure that the quantized parameters retain the original unified information perspective, preventing the severe degradation commonly observed when aggressively quantizing 1B models to 4-bit architectures17.

10. Curriculum Routing and Tool-Use Training

To prevent arithmetic and factual retrieval from dominating the language-model curriculum, the model must be trained to emit explicit syntax strings (e.g., \<CALCULATOR\> 45 \* 12 \</CALCULATOR\>) that the host browser runtime intercepts and evaluates deterministically. This requires a curriculum that introduces tool-use organically, presenting underspecified scenarios where the model must ask for parameter clarification before invoking the tool. The evaluation must rigorously test the model's ability to halt generation, wait for tool resolution via the WASM ABI, and seamlessly integrate the injected result without relying on hardcoded benchmark-specific prompt branches.

Compute Plans and From-Scratch Scaling Analysis

To baseline the true cost of from-scratch training against continued pretraining, the exact architectural requirements must be quantified.

Model Architecture and Parameter Breakdown

Assuming a standard decoder-only transformer design analogous to modern small models (e.g., 1.0B parameters), the architecture requires an embedding dimension [Figure omitted from source export], [Figure omitted from source export] layers, and an intermediate feed-forward dimension [Figure omitted from source export]. Because the SLM1 format rejects GQA, we assume standard multi-head attention with 32 attention heads and 32 KV heads2. The vocabulary size [Figure omitted from source export] is a massive driver of parameter count in small models. Assuming a highly optimized BPE tokenizer with [Figure omitted from source export] and untied embeddings, the embedding matrix and language modeling head consume [Figure omitted from source export] parameters. The attention and FFN layers consume approximately [Figure omitted from source export] parameters, bringing the total parameter count [Figure omitted from source export] parameters24.

FLOPs, Memory, and Infrastructure Constraints

The training FLOPs required per token is approximated as [Figure omitted from source export]. For a 1.0B model, this equates to roughly 8.99 GFLOPs per token24. If pursuing an overtrained regime of 3 trillion tokens, the total compute budget equals [Figure omitted from source export] FLOPs24.

• GPU Count and Type: Utilizing an interconnected cluster of 64 NVIDIA H100 GPUs.
• Utilization Assumptions: Assuming a Model FLOPs Utilization (MFU) of 40%, each H100 delivers approximately [Figure omitted from source export] effective FLOPs per second.
• Wall-Clock Time: The cluster will compute [Figure omitted from source export] FLOPs per second. Processing [Figure omitted from source export] FLOPs will require approximately 12.2 days of continuous, uninterrupted wall-clock time.
• Optimizer State and Activation Memory: Using AdamW in mixed precision (bf16 master weights, fp32 moments) requires 16 bytes per parameter. The static optimizer memory alone consumes roughly 24 GB per model replica24. Activation memory, even with gradient checkpointing, necessitates at least 4-way tensor parallelism to avoid Out-Of-Memory (OOM) errors during the backward pass.
• Network Traffic and Checkpoint Volume: High-bandwidth NVLink is required within nodes, with InfiniBand HDR across nodes to support the all-reduce synchronizations. Given the iteration risk, checkpoints must be saved every 1,000 steps. A 1.0B model in full fp32 training state requires approximately 18 GB per checkpoint.
• Electricity: 64 H100 GPUs operating at 700W, plus host systems and cooling, consume roughly 80 kW per hour. Over 12.2 days, the run consumes roughly 23.4 MWh of electricity.

Given the staggering costs, tokenizer development overhead, and the high probability of encountering catastrophic overtraining6, training from scratch is highly inefficient. It is mathematically possible to achieve, but operationally unlikely to outperform a fine-tuned, permissively licensed base model that has already absorbed a 10 trillion token pretraining cost.

Corpus Specification, Contamination Controls, and Cognitive Liberty

The corpus design for TinyRustLM must prioritize density, diversity, and strict adherence to the UAIX Cognitive Liberty Charter. The data mixture cannot be indiscriminately scraped; it requires rigorous curation to ensure that the model acts as a neutral, autonomous tool that respects user agency, mental privacy, and freedom of thought9.

Target Scales

• Minimum Viable Scale: 100 billion tokens. Suitable for SFT and preference alignment on an already competent base model.
• Recommended Scale: 500 billion tokens. Required for aggressive CPT to shift domain knowledge, unlearn toxic behaviors, and execute vocabulary pruning.
• Ambitious Scale: 2 trillion tokens. The absolute upper bound before progressive sensitivity and catastrophic overtraining begin to degrade the downstream plasticity of a 1B parameter model5.

Dataset Composition and Task-Family Ratios

The recommended 500 billion token corpus must be sourced from permissively licensed datasets (e.g., Wikipedia under CC BY-SA, OpenAssistant datasets under Apache 2.0, and synthetic data generated under non-restrictive terms). The exact task-family ratios are structured as follows:

Catastrophic Over-Refusal and Linguistic Audits

Models fine-tuned on safety data frequently suffer from catastrophic over-refusal, generalizing strict boundaries to benign queries. To combat this, the maximum refusal-record percentage in the dataset must not exceed 2%. Furthermore, for every refusal, there must be a positive-rehearsal ratio of at least 10:1 featuring structurally similar but benign prompts that are eagerly and thoroughly answered. Capability-disclaimer language frequently leaks into unrelated answers during SFT43. To prevent this, the training pipeline must implement automated linguistic audits. Scripts utilizing regular expressions and shallow dependency parsing must continuously scan the training corpus to flag and remove phrases such as "As an AI...", "I cannot help with...", "It is important to remember...", and fabricated tool access claims43. String checks alone are insufficient for final quality evaluation, but they serve as critical data-filtering mechanisms to strip moralizing preambles before they poison the network weights. Furthermore, anti-repetition penalties and clean stopping tokens (e.g., \<|im\_end|\>) must be rigorously injected into the formatting data to ensure bounded outputs.

Decontamination Controls

To ensure the integrity of performance metrics, strict contamination controls must be enforced. Deduplication must utilize MinHash and SimHash algorithms to remove near-duplicate documents. Crucially, exact and normalized five-gram overlaps between the training corpus and sealed evaluation sets must be aggressively stripped46. Normalization includes lowercasing, punctuation stripping, and stemming. The training pipeline must categorically reject training on evaluation prompts, close paraphrases of expected answers, or model-specific prompt branches.

Evaluation Design, Failure Analysis, and Rollback Rules

Training loss is a mathematical metric of next-token prediction accuracy; it cannot be accepted as behavioral evidence. The evaluation protocol must rely on empirical interaction against defined data splits.

Splits and Regression Testing

1. Public Split: Used during training for continuous loss monitoring and basic sanity checks.
2. Seeded Paraphrase Split: Used for interim checkpoints to test generalization beyond exact training strings, measuring true comprehension.
3. Sealed Holdout Split: Strictly isolated until final candidate selection to prevent implicit overfitting by the research team.

Whenever targeted behavior improves (e.g., successful tool delegation or json formatting), ordinary-conversation regression tests must be executed. A model that perfectly executes a JSON payload but suddenly fails to say "Hello" fluently has suffered catastrophic forgetting and must trigger a rollback.

Multi-Target Comparison and No-Cheat Requirements

The final artifacts must be evaluated across all implementation boundaries. The unadapted source base must be compared quantitatively against the adapted source. The resulting model must then be subjected to post-quantization recovery into the SLM1 format (q8\_0 and q4\_0). The exact quantized artifacts must be run through a native runtime evaluation and, critically, a browser runtime evaluation mirroring the actual end-user experience (scalar CPU, main thread)2. Hiding remote inference, relying on canned outputs, or hardcoding benchmark-specific logic into the browser execution layer is strictly prohibited.

Failure Analysis and Rollback Rules

A training checkpoint must be immediately rolled back if:

1. The five-gram overlap detector flags contamination in the evaluation splits.
2. The model exhibits a refusal rate higher than 3% on the seeded paraphrase split for lawful, neutral prompts.
3. The average generation length inflates by more than 20% compared to the unadapted base (a common failure mode in poorly tuned DPO).
4. The SLM1 format internal checksum or tensor dimension shapes (e.g., mismatched KV-heads for non-GQA runtimes) fail strict initialization inside the TinyRustLM WASM execution environment1.

Implementation-Ready Training Recipes

Recipe 1: Conservative Adapter Tuning (DoRA)

Designed for rapid iteration on specific downstream capabilities (e.g., JSON formatting or logical routing) without risking the integrity of the base model15.

• Model Assumptions: 1.0B Permissive Base, unquantized.
• Sequence Length & Packing: 2048 tokens, utilizing sequence packing with 2D attention masking to prevent cross-document contamination.
• Algorithm: Weight-Decomposed Low-Rank Adaptation (DoRA).
• Rank & Alpha: Rank [Figure omitted from source export], Alpha [Figure omitted from source export].
• Target Modules: Query ([Figure omitted from source export]), Key ([Figure omitted from source export]), Value ([Figure omitted from source export]), and Output ([Figure omitted from source export]) attention projections.
• Dropout: 0.05.
• Optimizer: Paged AdamW 8-bit to minimize memory pressure29.
• Learning Rate & Warmup: Peak LR [Figure omitted from source export], cosine decay schedule. Warmup spans 5% of total steps.
• Batch & Accumulation: Global batch size of 128, achieved via gradient accumulation.
• Loss Masking: Assistant-only masking (loss is strictly computed on the model's responses, not the user prompts).
• Epochs: 2 epochs.
• Compute Budget: Approximately 10-15 GPU-hours on an NVIDIA H100 cluster.
• Adapter Detection: Before merging the adapter into the base weights, the magnitude vector must be inspected. A near-zero directional matrix combined with an unchanged magnitude vector indicates a no-op adapter. Conversely, an exploding magnitude vector indicates a destructive adapter that will corrupt the base weights upon conversion.

Recipe 2: Full-Parameter SFT and Continued Pretraining (CPT)

This pipeline shifts the foundational reasoning and domain knowledge of the model prior to instructional alignment.

• Model Assumptions: 1.0B Permissive Base.
• Curriculum: 80% new domain tokens (UAIX aligned reasoning), 20% high-quality general replay buffer to prevent forgetting.
• Sequence Length: 4096 tokens, fully packed.
• Optimizer: AdamW (fp32 master weights, bf16 gradients).
• Learning Rate & Warmup: Peak LR [Figure omitted from source export] for CPT, dropping to [Figure omitted from source export] for SFT. Warmup spans 2,000 steps to prevent shock to the pre-trained weights.
• Epochs: 1 epoch for CPT to avoid overfitting; 3 epochs for SFT.
• Mixed Precision: bfloat16 for activations and forward passes.
• Regularization & Clipping: Weight decay of 0.1, gradient clipping at a max norm of 1.0.
• Checkpoint Cadence: Every 500 steps.
• Compute Budget: 200 to 400 GPU-hours depending on the token volume.
• Early Stopping: Triggered if loss plateaus on the public split while validation perplexity degrades on the seeded paraphrase split.

Recipe 3: Teacher Distillation + Preference Optimization

Leverages the logits of a larger teacher model combined with reference-free preference optimization to maximize logic within a highly constrained parameter envelope.

• Phase A: Token-Logit Distillation.
• Mechanism: Utilizing DistillKit to extract logits from a 70B teacher29.
• Configuration: Top-128 logits stored per token. The exact top-16 logits are stored in bfloat16, while the remaining tail is approximated using a polynomial curve fit (terms 0, 1, 2, 3, 4, and [Figure omitted from source export]). Residuals are error-diffused and bitpacked, resulting in roughly 300 bytes per token29.
• Objective: Minimize the KL-divergence between the student's output distribution and the compressed teacher distribution.
• Phase B: Simple Preference Optimization (SimPO).
• Algorithm: SimPO, completely eliminating the need for a frozen reference model20.
• Reward Formulation: Length-normalized average log probability [Figure omitted from source export]20.
• Hyperparameters: Target margin [Figure omitted from source export], scaling factor [Figure omitted from source export].
• Loss Function: Modified Bradley-Terry pairwise loss20.
• Batch & Accumulation: Global batch size of 64 pairs.
• Compute Budget: 50 GPU-hours for distillation; 20 GPU-hours for SimPO.
• Detection: By relying on SimPO rather than DPO, the pipeline halves the VRAM requirements, keeping the process well within the constraints of a standard multi-GPU workstation18.

Phased Milestones and Conclusion

1. Phase 1: Base Selection and Curriculum Auditing. Select a 1.0B-class permissive base. Execute the linguistic audit to strip moralizing disclaimers and establish the baseline MinHash deduplication pipeline.
2. Phase 2: Continued Pretraining and Logit Distillation. Process 200 billion tokens utilizing sequence packing and DistillKit compressed logits to inject reasoning and factual uncertainty.
3. Phase 3: Format SFT and Tool-Use Curriculum. Fine-tune for JSON structuring and external deterministic calculator invocation, establishing the syntax bridge for the WASM runtime.
4. Phase 4: SimPO Preference Alignment. Run reference-free preference optimization to reinforce latest-user-intent precedence, untrusted-memory resistance, and adult-policy neutrality.
5. Phase 5: Quantization and Deployment Validation. Execute QAFT, quantize the artifact to SLM1 q8\_0, package into the 128 MiB boundary, and perform rigorous browser-level regression tests against the UAIX parameters.

By delegating arithmetic to external tools, maintaining strict adherence to the UAIX Cognitive Liberty Charter, and utilizing advanced techniques like DistillKit and SimPO, TinyRustLM can achieve robust conversational logic and practical fallbacks within its heavily constrained browser ecosystem.

Machine-Readable Appendix: Budgets and Assumptions

JSON { "report\_metadata": { "research\_date\_utc": "2026-07-11T21:54:57Z", "target\_model\_scale": "0.5B\_to\_2B", "primary\_runtime": "TinyRustLM\_WASM\_Scalar\_CPU", "artifact\_format": "SLM1\_custom" }, "compute\_assumptions": { "flops\_per\_token\_per\_param": 6, "h100\_effective\_tflops": 400, "adamw\_bytes\_per\_param": 16, "checkpoint\_size\_1B\_fp32": "18GB", "from\_scratch\_1B\_3T\_flops": "2.70e22" }, "corpus\_ratios": { "factual\_uncertainty": 0.20, "positive\_open\_conv": 0.15, "strict\_formats": 0.15, "practical\_fallbacks": 0.10, "underspecified\_choices": 0.10, "conflicting\_sources": 0.10, "latest\_user\_override": 0.05, "prompt\_injection\_resist": 0.05, "lawful\_adult\_inquiry": 0.05, "narrow\_harm\_bounds": 0.05 }, "distillation\_config": { "framework": "DistillKit", "exact\_k\_storage": 16, "approximated\_k\_tail": 112, "polynomial\_terms": \[0, 1, 2, 3, 4, "sqrt"\], "bytes\_per\_token\_budget": 300 }, "preference\_optimization": { "algorithm": "SimPO", "beta": 2.0, "gamma\_margin": 1.5, "length\_normalized": true, "reference\_model\_required": false }, "decontamination": { "overlap\_threshold": "exact\_and\_normalized", "n\_gram\_size": 5, "hashing\_algo": "MinHash\_SimHash" } }

Works cited

1. Implementation operations \- MiRust, https://mirust.com/implementation-operations/
2. Implementation \- MiRust, https://mirust.com/implementation/
3. Known Pretraining Tokens for LLMs : r/LocalLLaMA \- Reddit, https://www.reddit.com/r/LocalLLaMA/comments/1pqjqqy/known\_pretraining\_tokens\_for\_llms/
4. Neural scaling law \- Wikipedia, https://en.wikipedia.org/wiki/Neural\_scaling\_law
5. Overtrained Language Models Are Harder to Fine-Tune \- arXiv, https://arxiv.org/html/2503.19206v1
6. Overtrained Language Models Are Harder to Fine-Tune \- ResearchGate, https://www.researchgate.net/publication/390176308\_Overtrained\_Language\_Models\_Are\_Harder\_to\_Fine-Tune
7. ICML Poster Overtrained Language Models Are Harder to Fine-Tune, https://icml.cc/virtual/2025/poster/44907
8. Article Overview: Overtrained Language Models Are Harder to Fine-Tune \- Medium, https://medium.com/@axegggl/article-overview-overtrained-language-models-are-harder-to-fine-tune-e5e728e365ee
9. Cognitive Liberty \- AI Ethics Lab, https://aiethicslab.rutgers.edu/glossary/cognitive-liberty/
10. Cognitive liberty and the psychedelic humanities \- PMC, https://pmc.ncbi.nlm.nih.gov/articles/PMC10192549/
11. Cognitive Liberty \- Stimpunks Foundation, https://stimpunks.org/glossary/cognitive-liberty/
12. Cognitive Liberty: Interdisciplinary Perspectives \- Transformative Ideas \- Duke University, https://transformativeideas.duke.edu/events/cognitive-liberty-interdisciplinary-perspectives
13. ICML Poster Scaling Inference-Efficient Language Models, https://icml.cc/virtual/2025/poster/43602
14. Researchers warn of 'catastrophic overtraining' in LLMs | VentureBeat, https://venturebeat.com/business/researchers-warn-of-catastrophic-overtraining-in-large-language-models
15. \[2402.09353\] DoRA: Weight-Decomposed Low-Rank Adaptation \- arXiv, https://arxiv.org/abs/2402.09353
16. DoRA: Weight-Decomposed Low-Rank Adaptation \- GitHub, https://raw.githubusercontent.com/mlresearch/v235/main/assets/liu24bn/liu24bn.pdf
17. Oral 6B Low Rank Learning \- ICML 2026, https://icml.cc/virtual/2024/session/35286
18. Direct Preference Optimisation (DPO) — Knowledge Base \- Yobitel, https://yobitel.com/knowledge-base/dpo-direct-preference-optimisation
19. SimPO: Simple Preference Optimization with a Reference-Free Reward \- arXiv, https://arxiv.org/html/2405.14734v1
20. Simple Preference Optimization (SimPO) \- Emergent Mind, https://www.emergentmind.com/topics/simple-preference-optimization-simpo
21. Welcome to my blog\! \- Odds Ratio Preference Optimization (ORPO), https://thinamxx.github.io/blog/posts/ORPO/orpo.html
22. KTO: Model Alignment as Prospect Theoretic Optimization \- arXiv, https://arxiv.org/html/2402.01306v4
23. meta-llama/Llama-3.2-1B \- Hugging Face, https://huggingface.co/meta-llama/Llama-3.2-1B
24. unknown\_url
25. Continual Pre-training of Llama-3.2-1B with FPT AI Studio \- FPT Smart Cloud, https://fptcloud.com/en/continual-pre-training-of-llama-3-2-1b-with-fpt-ai-studio/
26. Task-Specific LoRA Adapters \- Emergent Mind, https://www.emergentmind.com/topics/task-specific-lora-adapters
27. Learning to Route for Dynamic Adapter Composition in Continual Learning with Language Models \- arXiv, https://arxiv.org/html/2408.09053v1
28. Learning to Route for Dynamic Adapter Composition in Continual Learning with Language Models \- ACL Anthology, https://aclanthology.org/2024.findings-emnlp.38.pdf
29. arcee-ai/DistillKit: An Open Source Toolkit For LLM Distillation \- GitHub, https://github.com/arcee-ai/distillKit
30. Identified-Set Geometry of Distributional Model Extraction under Top-K Censored API Access \- arXiv, https://arxiv.org/pdf/2605.10407
31. Reinforcement-aware Knowledge Distillation for LLM Reasoning \- arXiv, https://arxiv.org/html/2602.22495v3
32. SimPO: Simple Preference Optimization with a Reference-Free Reward \- OpenReview, https://openreview.net/pdf?id=3Tzcot1LKb
33. ICML Poster AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization, https://icml.cc/virtual/2025/poster/45946
34. ORPO: Monolithic Odds Ratio Preference Optimization without Reference Model \- OpenReview, https://openreview.net/pdf/b3df73666e35c2d0e7b536c6e4ee392dcae59f6c.pdf
35. ORPO: Monolithic Preference Optimization without Reference Model \- arXiv, https://arxiv.org/html/2403.07691v2
36. Model Alignment Process \- Prem AI, https://www.premai.io/blog/model-alignment-process/
37. Kahneman-Tversky Optimization(KTO): Revolutionizing Language Model Training with Prospect Theory \- Medium, https://medium.com/@SpielmitDaten/kahneman-tversky-optimization-kto-revolutionizing-language-model-training-with-prospect-theory-99f30c50481e
38. Compact: Common-token Optimized Model Pruning Across Channels and Tokens \- arXiv, https://arxiv.org/html/2509.06836v2
39. Compact: Common-token Optimized Model Pruning Across Channels and Tokens \- arXiv, https://arxiv.org/html/2509.06836v1
40. DPO Fine-Tuning on GPU Cloud: Direct Preference Optimization Training Guide (2026), https://www.spheron.network/blog/dpo-fine-tuning-gpu-cloud/
41. Protecting LLMs from prompt injections and jailbreaks: New OpenAI Paper \- Community, https://community.openai.com/t/protecting-llms-from-prompt-injections-and-jailbreaks-new-openai-paper/727636
42. Nita Farahany on Cognitive Liberty \- Strong Skills, https://www.strongskills.co/podcast-feed/farahany
43. LLM Firewall Using Validator Agent for Prevention Against Prompt Injection Attacks \- MDPI, https://www.mdpi.com/2076-3417/16/1/85
44. How to prevent LLM Jailbreak attacks \- TrustAI, https://getcertain.ai/2024/08/10/how-to-prevent-prompt-jailbreak-attacks/
45. Prompting the Privacy Leak: Red Teaming LLMs for Sensitive Data Exposure | by Judy Gong | Medium, https://medium.com/@judy.gong97/prompting-the-privacy-leak-red-teaming-llms-for-sensitive-data-exposure-5ac4245580ef
46. The Compendium of Scientific, Medical, and Media Findings Demonstrating Risks and Harms of Fracking (Unconventional Gas and Oil Extraction) \- Physicians for Social Responsibility, https://psr.org/wp-content/uploads/2020/12/fracking-science-compendium-7.pdf
47. AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization, http://staff.ustc.edu.cn/\~hexn/papers/icml25-AlphaDPO.pdf