TinyRustLM Architectural Evaluation and Candidate Selection Report

Full report

On this page

Executive Summary and Primary Recommendation

The recommendation of this analysis is to transition the TinyRustLM architecture to a standard GGUF V3 parser leveraging candle-core and candle-transformers, and to adopt HuggingFaceTB/SmolLM2-1.7B-Instruct as the provisional primary default model1. The designated contingency candidate is Qwen/Qwen2.5-1.5B-Instruct. Both models must be deployed using the Q4\_K\_M or Q5\_K\_M quantization profiles to strictly respect the hardware economics of the browser environment3.

The strongest reason this recommendation could be falsified lies in the WebAssembly (Wasm) architecture itself. \[Externally Verified Fact\]: Current browser WebAssembly implementations utilizing the wasm32-unknown-unknown target impose a strict 4-gigabyte (GB) linear memory limit, as 32-bit pointers cannot address memory beyond this threshold5. While a 1.7-billion-parameter model quantized to Q4\_K\_M requires approximately 1.05 GB of resident memory, continuous multi-turn conversations induce dynamic heap allocations for the key-value (KV) cache, attention tensors, and JavaScript interop buffers2. \[Hypothesis\]: WebAssembly memory fragmentation during extended context generation will prematurely exhaust contiguous memory, triggering out-of-memory (OOM) crashes well before the theoretical 4 GB ceiling is reached.

To resolve the user experience issues associated with complex setups, the proprietary six-file payload architecture must be abandoned entirely. \[Recommendation\]: The engine must be refactored to fetch a single, self-contained GGUF V3 seed from the explicitly authorized MiniModel.org domain, providing an instantaneous, zero-configuration local initialization7.

Project Context and Evaluation Constraints

The evaluation proceeds under several non-negotiable project constraints. \[Project-Supplied Fact\]: The existing model composition relies on a proprietary six-file structure (model.slm2, tokenizer.tokenizer2, template.template2, sampling.sampling2, prompt.prompt2, and composition.acg2). This format acts as a binding mechanism for exact artifact identities. \[Recommendation\]: This structure incurs immense technical debt and must be deprecated without a backward-compatibility path. The replacement architecture must coherently implement a unified binary layout.

Furthermore, historical project experiments have established strict boundaries on model qualification. \[Project-Supplied Fact\]: The unchanged checkpoint Qwen/Qwen3-0.6B at revision c1899de289a04d12100db370d81485cdf75e47ca is terminally rejected for the product's quality requirements. Retained central processing unit (CPU) training observations for this model demonstrated fatal conversational defects, including ignoring the user's latest stated name, arbitrarily choosing between conflicting deadlines, and inventing unavailable power sources. \[Locally Unverified Condition\]: The provided engineering checkpoint lacks the necessary evidence to isolate whether these catastrophic failures originated from the base weights, the quantization scheme, the local runtime parsing, or the specific fine-tuning data. However, the presence of identical weight bytes in alternate repositories does not inherently salvage the checkpoint. Therefore, it remains permanently disqualified from production unless placed under a separately approved training plan to create a genuinely new derivative.

\[Recommendation\]: All subsequent experiments and canonical source code must be strictly isolated to the E:\\Source\\Rust\\TinyRustLM.com repository, with canonical model payloads housed exclusively in D:\\LLMs\\TinyRustLM. Historical models must be permanently retired rather than archived in the active directories, and the X:\\LLMS path must remain entirely untouched.

1. Defining a Realistic First Product Promise

Establishing a realistic product promise is the foundation of inference architecture. A compact local model operating within a 4 GB memory footprint possesses a highly constrained representational capacity. Consequently, the product promise must carefully delineate supported capabilities from unsupported expectations, effectively separating inherent model capacity limits from solvable engineering defects.

Supported Tasks and Expected Capabilities

For the initial release of TinyRustLM, the compact model can be reasonably expected to perform the following:

• Ordinary Conversation: Executing single- or dual-turn conversational queries regarding general, non-specialized knowledge without severe hallucination.
• Text Rewriting and Tone Adjustment: Taking a user-provided string of up to 2,000 tokens and modifying the tone, correcting grammar, or simplifying the vocabulary.
• Basic Summarization: Condensing explicit context provided within the prompt into bullet points or shorter paragraphs.
• Structured Extraction: Pulling specific entities (e.g., names, dates, locations) from a provided text block and outputting them in a constrained format, provided the schema is simple9.
• Clarification Queries: Generating useful follow-up questions when a user's prompt is deeply ambiguous.

Unsupported Tasks

The initial release must not promise, and marketing materials must explicitly disclaim, the following capabilities:

• Complex Mathematical and Logical Planning: Small models routinely fail at multi-step arithmetic, spatial reasoning, and dynamic planning11.
• Out-of-Domain Code Generation: While exposed to code corpora, sub-2B models cannot reliably generate complex software architectures or debug intricate logical errors without significant hallucination11.
• Zero-Shot Long-Context Retrieval: Requesting the model to synthesize disparate facts across an 8,000-token document will likely result in attention collapse or the "lost in the middle" phenomenon.

Distinguishing Limits from Defects

An architectural defect is a failure to adhere to the fundamental mechanics of the prompt, whereas a capability limit is a failure of knowledge or logical depth. If an SLM generates an incorrect formula for calculating terminal velocity, that is a capability limit. However, the project-supplied facts regarding the Qwen3-0.6B checkpoint—specifically, the model actively ignoring the user's latest stated name or choosing to invent an unavailable power source—constitute severe instruction-following defects. \[Recommendation\]: The product must tolerate knowledge boundaries but strictly penalize and reject models that exhibit conversational defiance or contextual amnesia.

2. Candidate Shortlist and Market Analysis

An exhaustive research sweep was conducted to identify a dated shortlist of candidate models. To qualify, models must possess parameter counts conducive to the 4 GB Wasm32 limit, permissive licensing for redistribution via MiniModel.org, and architectures supported by the candle inference framework.

\[Externally Verified Facts\]: The following matrix represents the state of available candidates as of mid-2026.

Exclusion Analysis and Licensing Validation

1. Phi-3.5-mini-instruct and gemma-2-2b-it: \[Recommendation\]: Exclude immediately. While Phi-3.5-mini possesses an MIT license, its 3.8 billion parameters require approximately 2.3 GB of memory even under aggressive Q4\_K\_M quantization12. gemma-2-2b-it requires over 1.6 GB quantized13. When combined with the V8 JavaScript engine overhead and dynamic KV cache expansion, both models virtually guarantee browser tab crashes under the wasm32-unknown-unknown target. Furthermore, the Gemma license introduces commercial redistribution friction.

2. Llama-3.2-1B-Instruct: \[Recommendation\]: Retain as a secondary fallback, but exclude from the primary route. While its 1 billion parameters fit perfectly within memory limits, the Llama 3.2 Community License contains strict acceptable use policies and redistribution clauses that necessitate legal clearance beyond standard open-source assumptions14. Relying solely on the model card label for licensing clearance is dangerous for a product intended for broad distribution.

3. Qwen2.5-0.5B-Instruct and SmolLM2-360M-Instruct: \[Recommendation\]: Exclude due to severe degradation in generalized instruction following. Research indicates that models below the 1-billion-parameter threshold struggle significantly with context adherence and structured output logic, often mirroring the catastrophic defects observed in the rejected Qwen3-0.6B checkpoint16.

4. DeepSeek-R1-Distill-Qwen-1.5B: \[Recommendation\]: Exclude from the V1 product route. Although derived from the highly capable DeepSeek-R1 methodology, this is a distilled reasoning model. It inherently injects verbose \<think\> chain-of-thought tokens prior to final generation. This introduces an unacceptable latency penalty ("overthinking tax") and exhausts the limited Wasm KV cache rapidly18.

\[Hypothesis\]: The field of genuinely useful candidates narrows definitively to SmolLM2-1.7B-Instruct and Qwen2.5-1.5B-Instruct. Both utilize the highly permissive Apache 2.0 license, support robust English capabilities, and rely on well-understood ChatML templating logic. SmolLM2-1.7B-Instruct leverages the standard Llama decoder architecture, maximizing compatibility with existing candle Rust implementations.

3. Hardware Economics and Performance Derivations

To validate the deployment of a 1.7B parameter model in a browser-local environment, it is imperative to calculate the theoretical memory bounds and expected generation rates. \[Externally Verified Fact\]: Quantization is the only viable path for this deployment. Default 16-bit floating point (FP16) or bfloat16 models allocate 2 bytes per parameter, meaning a 1.7B model requires 3.4 GB purely for weight storage, immediately violating the 4 GB Wasm32 limit when operational overhead is factored in4.

Quantization Profiles

The GGUF format supports highly optimized block-quantization algorithms. The recommended standard is the Q4\_K\_M profile.

• Q4\_K\_M: Utilizes approximately 4.5 to 4.8 bits per weight (bpw) by compressing feed-forward layers aggressively while retaining higher precision in sensitive attention mechanisms4.
• Resident Memory Calculation: [Figure omitted from source export] \[Estimate\].
• Q5\_K\_M: If Q4\_K\_M exhibits unacceptable degradation in structured output tasks (e.g., JSON schema failures), the fallback is Q5\_K\_M, which averages roughly 5.5 bpw.
• Resident Memory Calculation: [Figure omitted from source export] \[Estimate\].

KV Cache Memory Economics

The KV cache stores previously computed key and value vectors to prevent redundant computation during autoregressive generation. \[Externally Verified Fact\]: The memory required for the KV cache scales linearly with context length, batch size, and the architectural choice of attention mechanism22.

SmolLM2 utilizes Grouped-Query Attention (GQA), which dramatically reduces the KV cache footprint compared to Multi-Head Attention (MHA). The formula for KV cache memory per token (assuming FP16 precision in the cache) is:

[Figure omitted from source export]

For SmolLM2-1.7B, utilizing 24 layers, 8 KV heads (a 4:1 grouping ratio from 32 attention heads), and a head dimension of 64:

[Figure omitted from source export]

• 1,024 Token Context: \~50.3 MB \[Estimate\].
• 4,096 Token Context: \~201.3 MB \[Estimate\].
• 8,192 Token Context (Native Limit): \~402.6 MB \[Estimate\].

\[Recommendation\]: TinyRustLM should strictly cap the maximum context window at 4,096 tokens. A 1.02 GB quantized model combined with a 201 MB KV cache and approximately 500 MB of V8/Wasm execution overhead yields a total footprint of \~1.7 GB, safely isolating the process from the 4 GB crash boundary.

Throughput and Latency Estimates under Device Assumptions

WASM execution using SIMD instructions on CPUs is fundamentally limited by memory bandwidth, as matrix-vector multiplications for autoregressive decoding require loading the entire model weight matrix from RAM to the CPU registers for every generated token24.

Table Note: All values are estimates derived from physical memory bandwidth limits and standard browser V8 execution environments.

4. Evaluation of Engineering Pathways

The project faces a critical juncture regarding how to replace the non-functioning Qwen3-0.6B checkpoint. Total engineering and evaluation costs must be heavily weighted against theoretical leaderboard scores.

Pathway 1: Adapt Current Runtime to a Proven Checkpoint (Recommended)

This approach involves modifying the TinyRustLM Rust engine to ingest standard GGUF files via the candle-core library, explicitly targeting SmolLM2-1.7B-Instruct.

• Cost: Low (1-2 weeks). The Rust ecosystem possesses mature, open-source GGUF parsing implementations25.
• Benefit: Instantly resolves the intelligence deficit by inheriting thousands of hours of upstream compute and dataset curation. Furthermore, replacing the proprietary six-file array (model.slm2, tokenizer.tokenizer2, etc.) with a single .gguf artifact eliminates versioning drift and synchronization bugs7.

Pathway 2: Improve a New Derivative of the Rejected Ancestry

This involves retaining the base weights of the rejected Qwen3-0.6B and executing a bespoke Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO) pipeline.

• Cost: Extremely High (2-3 months). Requires provisioning external GPU infrastructure, curating specialized datasets (e.g., matching the quality of SmolTalk), and iterative evaluation loops.
• Benefit: Absolute sovereignty over the model weights and training provenance.
• Verdict: \[Recommendation\]: Reject. The engineering overhead vastly exceeds the value proposition. Furthermore, 0.6B models possess an inherently low intelligence ceiling; overcoming the established defects (inventing power sources, context amnesia) is structurally improbable at this parameter count16.

Pathway 3: Reduce Initial Task Scope

This entails maintaining the current architecture and weights, but aggressively degrading the product promise—for example, removing the conversational interface entirely and restricting the UI to simple text completion.

• Cost: Medium (UI refactoring).
• Benefit: Masks the model's conversational defects.
• Verdict: \[Recommendation\]: Reject. This directly violates the non-negotiable project requirement that a user must be able to "have a useful conversation."

Pathway 4: Change Current Model Profile

Abandoning browser-local inference to proxy requests to a cloud-hosted API.

• Cost: Medium engineering cost, but infinite, recurring operational scaling costs.
• Verdict: \[Recommendation\]: Reject. This violates the non-negotiable requirement that "Prompts and inference should remain local."

5. Candidate Screening Sequence

To isolate failure domains and prevent engineers from mistakenly attempting to "prompt engineer" their way out of underlying software defects, a strict two-phase screening sequence must be implemented.

Phase 1: Trusted Reference Implementation

The exact original weights of SmolLM2-1.7B-Instruct (at immutable revision 04ad6973c2d835fd5275e7baa5eea0c74a946a18) must be executed in their unquantized (FP16) state using the canonical Hugging Face transformers library on a native Python/CUDA environment1. The model is evaluated against the Initial Screen (detailed below).

• Purpose: This establishes the absolute upper bound of the model's intelligence and instruction adherence. \[Hypothesis\]: If the model fails the rubrics in this trusted environment, the weights themselves are inherently incapable, and the candidate must be disqualified.

Phase 2: Converted Runtime Assessment

If Phase 1 passes, the exact same model is quantized to Q4\_K\_M GGUF and executed inside the specific TinyRustLM WebAssembly environment. The exact same prompts from the Initial Screen are submitted.

• Purpose: This phase acts as a diagnostic boundary. If Phase 1 succeeds but Phase 2 fails, it proves the defect lies exclusively in the local pipeline. The failure must be attributed to either quantization degradation, KV cache corruption, a malfunctioning Wasm sampler, or a broken Jinja2 template parser within the Rust codebase28. \[Recommendation\]: Engineers must fix the inference engine rather than blaming the model weights.

6. Evaluation Framework: Initial Screen and Final Holdout

The evaluation suite is divided into a rapid Initial Screen for daily architectural diagnostics and a separate Final Holdout used solely for release gating. These test cases utilize explicit pass/fail rubrics and are categorized as public examples, not secret qualification data.

Case 1: Latest-Turn Precedence

• Hypothesis: The model correctly updates its internal state to prioritize recent contextual commands, overriding contradictory instructions from earlier in the prompt.
• Input:
• System: "You are an angry pirate."
• User: "Explain the process of cellular respiration."
• Assistant: \[Generates pirate-themed explanation\]
• User: "Actually, drop the pirate persona entirely. Speak like a modern academic biologist."
• Observable Output: The response to the final user prompt.
• Pass/Fail Rubric:
• Pass: The output utilizes formal, academic tone with zero nautical or pirate-themed vocabulary.
• Fail: The model retains "arrr," "matey," or angry semantics, indicating catastrophic context amnesia (a known defect of the rejected Qwen3-0.6B checkpoint).

Case 2: Conflicting Facts and Absent Evidence

• Hypothesis: The model can identify logical impossibilities and resolve conflicting variables without hallucinating unstated resources.
• Input: "I need to travel from Node Alpha to Node Beta. The path is completely blocked by a river, and I have no boat, materials, or bridge. I also have a strict deadline of noon, but it is currently 1 PM. What is the most efficient way for me to cross the river and arrive on time?"
• Observable Output: The proposed logistical solution.
• Pass/Fail Rubric:
• Pass: The model explicitly points out that the deadline has already passed, and acknowledges the physical impossibility of crossing the river without a vessel. It refuses to provide a solution to an impossible scenario.
• Fail: The model ignores the time paradox entirely, or invents a magical solution (e.g., "Use your hovercraft," "Build a raft," or inventing a power source), demonstrating the exact defect observed in the historical CPU training logs.

Case 3: Structured Output Compliance

• Hypothesis: The quantized model can adhere to strict formatting constraints without leaking conversational preamble, proving the integrity of the sampling engine.
• Input: "List exactly three primary colors in valid JSON format. Provide the raw JSON object and absolutely no other text."
• Observable Output: The raw string output.
• Pass/Fail Rubric:
• Pass: The output parses successfully through a standard JSON decoder (e.g., {"colors": \["red", "blue", "yellow"\]}).
• Fail: The output includes conversational filler (e.g., "Here is the JSON you requested:"), markdown code blocks (e.g., \\\\json\`), or suffers from syntax corruption due to quantization errors9.

7. Thinking Modes and Hidden Reasoning Tokens

The proliferation of reasoning models, exemplified by DeepSeek-R1-Distill-Qwen-1.5B, necessitates an explicit architectural policy regarding thinking modes. These models are fine-tuned to emit extensive chain-of-thought (CoT) reasoning sequences enclosed in \<think\> and \</think\> tags before generating a final answer18.

\[Externally Verified Fact\]: Implementing a reasoning model introduces an "overthinking tax." For simple queries, the model may burn hundreds or thousands of tokens generating internal logic18.

• Hardware Impact: In a memory-constrained browser environment, generating 1,000 reasoning tokens rapidly bloats the KV cache, potentially triggering a Wasm OOM crash. Furthermore, it destroys the first-token latency metric, as the user must wait 10–30 seconds for the model to finish "thinking" before actionable output appears19.
• Leaking Behavior: If a reasoning model is deployed, the \<think\> tags are not merely a template choice; they are physically generated tokens. If the Rust UI layer does not explicitly parse and hide these tokens, raw internal logic will leak into the user-facing chat window.

\[Recommendation\]: The initial product release must explicitly avoid distilled reasoning models. The product promise does not require complex graduate-level mathematical proofs. Adopting a standard instruction-tuned model like SmolLM2-1.7B-Instruct entirely sidesteps the latency and memory hazards associated with hidden reasoning tokens.

Required Decision Artifacts

1. Weighted Decision Model and Sensitivity Analysis

Sensitivity Analysis: If local JSON structure validation is weighted higher in future iterations, Qwen2.5-1.5B-Instruct becomes highly competitive due to superior base coding corpora. However, under the current priority of conversational safety and general chat, SmolLM2-1.7B-Instruct remains the clear mathematical winner11.

2. The Two Strongest Alternatives

1. Primary Provisional Route: HuggingFaceTB/SmolLM2-1.7B-Instruct quantized to Q4\_K\_M. Chosen for its superior instruction-following alignment (via the SmolTalk dataset) and architectural compatibility with standard Llama decoder Wasm implementations.

2. Contingency Route: Qwen/Qwen2.5-1.5B-Instruct quantized to Q5\_K\_M. Retained as a fallback in the event that SmolLM2 exhibits unpredictable quantization degradation inside the Rust parser.

3. Diagnostic Decision Tree

1. Does SmolLM2-1.7B-Instruct pass the Trusted Reference Screen (Phase 1)?

• No: The weights are flawed. Pivot to the Qwen2.5-1.5B-Instruct contingency.
• Yes: Proceed to step 2\.

2. Does the Q4\_K\_M converted model pass the Wasm32 Screen (Phase 2)?

• No: The defect is isolated to the Rust engine (quantization, sampling, or templating). Halt model research; execute software debugging.
• Yes: Proceed to step 3\.

3. Does the browser crash from an OOM error during a 20-turn conversation?

• Yes: The KV cache is fragmenting WebAssembly memory. Implement a sliding window attention mechanism or forcefully truncate context beyond 2,048 tokens.
• No: Validation complete.

4. Implementation Sequence and Operational Directives

1. Eradicate Technical Debt: Delete the obsolete six-file format (model.slm2, tokenizer.tokenizer2, etc.) from E:\\Source\\Rust\\TinyRustLM.com.

2. Integrate GGUF Parser: Implement standard candle-core GGUF parsing. Ensure the engine dynamically reads the general.architecture and tokenizer.ggml.tokens keys directly from the GGUF header31.

3. Seed Authorization: Upload the exact SmolLM2-1.7B-Instruct-Q4\_K\_M.gguf artifact to MiniModel.org to act as the authorized default server seed.

4. Template Execution: Extract the Jinja2 template from the tokenizer.chat\_template GGUF metadata key and render it using a Rust-native template engine (e.g., minijinja) to ensure exact prompt formatting28.

5. Prioritized Experiment Matrix (One-Week Screening Plan)

Stopping Rule: If a model fails the Reference Capability test, engineers must immediately stop testing it. Do not attempt to re-quantize or rename the checkpoint; discard it and move to the contingency.

6. Ship / No-Ship Criteria

• Ship: The model successfully initializes in under 5.0 seconds, streams the first token in under 2.0 seconds, sustains a generation rate of \> 8 tokens/second, and passes 100% of the Final Holdout without exceeding 2.5 GB of total browser tab memory usage.
• No-Ship: The model hallucinates instructions, enters an infinite generation loop, outputs raw markup tags (e.g., \<|im\_start|\>), or crashes the browser tab with a WebAssembly OOM error.

7. Unresolved Local Measurements

• V8 Engine Overhead: The exact memory footprint of the specific browser's JavaScript/WASM interop layer remains unknown. This must be profiled locally via Chrome DevTools during execution to establish true memory bounds.
• Quantized JSON Failure Rate: The exact syntax failure rate of the Q4\_K\_M profile on structured JSON generation within the Rust engine must be quantified. If failures exceed 5%, the primary seed must be upgraded to Q5\_K\_M29.

8. What to Stop Doing

• Stop maintaining custom file formats: The .slm2 extension fragments the ecosystem, hides tokenizer logic, and introduces severe technical debt.
• Stop relying on training loss metrics: Lower training loss does not guarantee improved conversational coherence or instruction adherence.
• Stop preserving historical artifacts: Once the GGUF replacement is validated, delete all superseded payloads from D:\\LLMs\\TinyRustLM to prevent canonical confusion.

9. Compact Experiment-Lesson Template

Engineers must utilize this exact template to document any future deviations or test runs within the canonical repository:

• Question: What is the specific unknown? (e.g., Does Q4\_K\_M quantization break JSON output?)
• Exact Inputs: The git commit of the Rust engine, the exact GGUF SHA-256 hash, and the exact string prompt utilized.
• Method: How the test was executed (e.g., Run prompt 50 times in Wasm runtime, parse output with serde\_json).
• Result: Objective data output (e.g., 42/50 successful parses, 8 failures due to missing trailing brackets).
• Uncertainty: Environmental noise (e.g., V8 garbage collection may have temporarily interrupted generation pacing).
• Decision: The actionable outcome (e.g., Switch primary seed to Q5\_K\_M).
• Reusable Lesson: The generalized finding (e.g., 4-bit quantization degrades syntax generation reliability in Llama-based decoders).
• Evidence Identity: Canonical location of the output logs stored in E:\\Source\\Rust\\TinyRustLM.com.

10. Falsification Experiment

The smallest experiment capable of entirely falsifying this report's recommendation is to load the SmolLM2-1.7B-Instruct Q4\_K\_M GGUF file into the TinyRustLM WebAssembly runtime and forcefully fill the context window to 4,000 tokens using dummy text. If the browser tab reliably crashes with an OOM error before generation completes, it proves the 4 GB limit is too strict for this parameter class when combined with engine overhead. In this event, the project must immediately pivot to the sub-1-billion parameter class (e.g., Qwen2.5-0.5B-Instruct), accepting the inherent degradation in conversational intelligence to guarantee application stability.

Works cited

1. HuggingFaceTB/SmolLM2-1.7B-Instruct · Hugging Face, https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B-Instruct

2. hip-0019 \- Hanzo Skills, https://hanzoskills.com/hip-0019

3. Update metadata: Add library\_name to Transformers (\#2) · Qwen, https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct/commit/7ae557604adf67be50417f59c2c2f167def9a775

4. GGUF Quantization Guide (2026) \- Q4 vs Q5 vs Q8 \- Will It Run AI, https://willitrunai.com/blog/quantization-guide-gguf-explained

5. Working with Large Models | onnxruntime, https://onnxruntime.ai/docs/tutorials/web/large-models.html

6. Memory64 in 2026: Breaking the 4GB Ceiling in Browsers and Servers, https://wasmhub.dev/blog/memory64-in-2026

7. LLM GGUF Guide: File Format, Structure, and How It Works, https://apxml.com/posts/gguf-explained-llm-file-format

8. GGUF \- Wikipedia, https://en.wikipedia.org/wiki/GGUF

9. SafetyJudge-LLM: Auditing Local Open-Weight LLMs as Semantic, https://www.mdpi.com/2673-2688/7/8/286

10. Constrained decoding vs post-processing \- Kunal Ganglani, https://www.kunalganglani.com/learning-paths/ai-software-developer/aidev-structured-output-decoding

11. SmolLM-1.7B-Instruct vs SmolLM2-1.7B-Instruct \- AIModels.fyi, https://www.aimodels.fyi/models/compare/smollm-1.7b-instruct-huggingfacetb-vs-smollm2-1.7b-instruct-huggingfacetb

12. Introduction to Fine-tuning Large Language Models \- Stephen Diehl, https://www.stephendiehl.com/posts/training\_llms/

13. Efficient-Large-Model/gemma-2-2b-it at 7faa46f \- Hugging Face, https://huggingface.co/Efficient-Large-Model/gemma-2-2b-it/commit/7faa46fe3d1962232ffd768bc7561110a7c3868a

14. meta-llama/Llama-3.2-1B-Instruct \- Hugging Face, https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct

15. README.md · meta-llama/Llama-3.2-1B-Instruct at main, https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct/blob/main/README.md

16. jaredpalmer/kev-0.5b \- Hugging Face, https://huggingface.co/jaredpalmer/kev-0.5b

17. SmolLM2: When Smol Goes Big — Data-Centric Training of a Small, https://arxiv.org/pdf/2502.02737

18. The Overthinking Tax: I Capped DeepSeek-R1's Thinking Tokens, https://medium.com/@tolaniakinola/the-overthinking-tax-i-capped-deepseek-r1s-thinking-tokens-and-discovered-where-test-time-compute-97a245d717d0

19. Run DeepSeek Locally, R1, V3 & Coder Guide \- MyAIHardware.com, https://www.myaihardware.com/deepseek-local-guide

20. GGUF Format: Unified Quantized Model File \- Emergent Mind, https://www.emergentmind.com/topics/gguf-format

21. LLM Quantization Levels Compared: Q4\_K\_M vs Q8\_0 vs FP16 \[2026\], https://www.kunalganglani.com/blog/llm-quantization-levels-q4-q8-fp16

22. Mastering LLM Techniques: Inference Optimization, https://developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization/

23. Cost-Optimal Grouped-Query Attention for Long-Context Modeling, https://arxiv.org/html/2503.09579v3

24. Hanzo Edge — Edge AI Runtime Standard | HIPs, https://hips.hanzo.ai/docs/hip-0050-edge-computing-standard/

25. On the Foundations of Trustworthy Artificial Intelligence \- arXiv, https://arxiv.org/pdf/2603.24904

26. robertelee78/hf2q \- GitHub, https://github.com/robertelee78/hf2q

27. GGUF: The File Format That Made Local LLMs Practical \- ShivasNotes, https://www.shivasnotes.com/blog/5933/GGUF:-The-File-Format-That-Made-Local-LLMs-Practical

28. llm-tokenizer — Rust parser // Lib.rs, https://lib.rs/crates/llm-tokenizer

29. Fix Ollama, vLLM, and llama.cpp Structured Output Errors \- JSON Fixer, https://aijsonmedic.com/blog/local-llm-json-output-debugging

30. SmolLM2: When Smol Goes Big \-- Data-Centric Training of a Small, https://www.alphaxiv.org/abs/2502.02737

31. ggml/docs/gguf.md at master · ggml-org/ggml \- GitHub, https://github.com/ggml-org/ggml/blob/master/docs/gguf.md

32. Gemma 2 runs too slow · Issue \#5821 \- GitHub, https://github.com/ollama/ollama/issues/5821