TinyRustLM Runtime Quality And Tiny Model Candidate Research

Full report

On this page

Core findings

The biggest reason TinyRustLM currently produces fixture-like or repeated-token output is that the checked-in artifacts are explicitly documented as runtime-smoke and deterministic-smoke models, not assistant-quality converted-trained models. MiRust’s own implementation notes say the supplied artifacts make no trained assistant-quality claim, and its quality-gate documentation is explicit that a runtime-smoke artifact cannot be promoted to assistant quality merely because it emits text. In other words, part of the current behavior is not a bug in sampling alone; it is also an artifact-quality boundary.

The second major finding is that TinyRustLM’s current runtime contract is still too narrow for most genuinely useful small open chat models. The current SLM1 tensor ABI assumes equal-shaped Q/K/V/O projections and warns that Grouped Query Attention, alternative projection shapes, biases, or renamed tensors require a new model-type contract or format version. That matters because the strongest realistic browser-size candidates in 2026—SmolLM2 135M/360M, Qwen2.5 0.5B, and TinyLlama 1.1B—use Llama/Qwen-family inference conventions and, in several cases, explicit GQA layouts.

The third major finding is that the current local load/generate path itself can create repetition or incoherence even with better weights. The present runtime uses a custom embedded BTOK or BPE1 tokenizer, strips one terminal EOS before prefill, treats token ID 257 as the only structural stop token, defaults to deterministic argmax sampling with temperature=0, top_k=1, top_p=1, and seed=1, and does not automatically build multi-turn chat prompts from prior transcript state. Clear/Reset semantics also diverge: “Clear” only clears the visible transcript, while runtime KV/token state remains unless reset_context is called. Those are exactly the kinds of conditions that magnify loops, stale continuations, and “why is it answering as if it is still in the last turn?” failures.

The practical bottom line is straightforward: if the goal is real coherent local .slm chat in Rust/WASM, the near-term path is not “prompt hacks.” It is: use a real converted-trained model, widen the runtime to support real tokenizer identity and chat-template metadata, add model-specific EOS/stop handling, add repetition controls and non-greedy defaults, fix context/reset UX so runtime state cannot silently diverge from visible chat state, and extend the tensor ABI for GQA-class small models.

Why repeated-token and fixture-like output happens

The most likely root cause is still the simplest one: TinyRustLM is presently demonstrated with artifacts whose own manifests and documentation describe them as deterministic smoke fixtures, not as trained assistants. MiRust’s quality-gate and model-catalog docs repeatedly separate “artifact is parseable and executes” from “artifact is useful conversationally,” and they explicitly reject promoting fixture-like output into assistant-quality claims. If repeated or toy-like continuations are being observed on the supplied models, that is consistent with the project’s own stated evidence boundary.

A second high-probability root cause is tokenizer mismatch. TinyRustLM currently relies on embedded BTOK or BPE1 tokenizers; BTOK is byte-level with a 260-token fixed vocabulary and is only appropriate when a model was trained with that exact mapping. MiRust also says BPE1 is still a simple vector-and-scan design that needs production features such as tokenizer identity/checksum, corpus compatibility tests, and better indexing. If a source model was trained with SentencePiece or a full tokenizer.json BPE and the .slm conversion is only approximately re-encoded, the model will see the wrong token IDs, and repetition or nonsense output is a completely expected failure mode.

A third likely root cause is prompt-format mismatch. The current browser UI sends only the current prompt field to generate; earlier transcript turns are not concatenated into the prompt, and the UI does not expose the low-level continuation export that could intentionally extend runtime context. Meanwhile, the best candidate chat models all expect a specific chat template. SmolLM2 and Qwen2.5 both document apply_chat_template(..., add_generation_prompt=True), and TinyLlama’s chat model card shows explicit <|system|>, <|user|>, and <|assistant|> markers separated by </s>. If TinyRustLM feeds plain text when a model expects role-tagged serialized conversation, the model often falls into shallow repetition or incomplete continuations rather than genuine chat behavior.

A fourth likely root cause is EOS/stop-token mismatch. TinyRustLM’s current operation notes say that both tokenizer paths emit BOS/EOS, prefill removes the terminal EOS, and only token ID 257 is treated as a structural stop token. That is fine for BTOK/BPE1 fixtures, but real models frequently rely on model-specific EOS IDs and, for chat, on additional control tokens or stop sequences. If conversion fails to map the source model’s real stop semantics into .slm, generation can either stop too early or never stop naturally and drift into repetition until token limits or context limits are hit.

A fifth likely root cause is greedy sampler behavior. TinyRustLM currently defaults to deterministic argmax: temperature=0, top_k=1, top_p=1, seed=1. MiRust’s sampler notes say that this selects deterministic argmax, and broader generation guidance from Hugging Face documents repetition penalty as an explicit generation control because 1.0 means no anti-repetition effect at all. For open-ended chat, pure argmax is well known to increase dullness and local repetition, especially on small models.

A sixth likely root cause is state divergence between visible chat and runtime context. TinyRustLM’s docs distinguish transcript state from model context. reset_context clears runtime token vectors, generated tokens, visible KV length, and diagnostics, but preserves the browser transcript; the browser Clear control removes visible messages only and does not touch runtime KV/token state. That can create the exact confusing failure mode where the UI looks like a fresh chat but the model is still conditioned on stale hidden context, which small models often manifest as repeated or nonsensical continuations.

A seventh likely root cause is architecture mismatch during conversion, especially around GQA. TinyRustLM’s current tensor contract states that Q/K/V/O use hidden-width matrices in the current equal-head implementation, and it explicitly warns that GQA shapes require a new contract or version. SmolLM2 360M’s config shows num_attention_heads=15 and num_key_value_heads=5; TinyLlama 1.1B shows 32 vs 4; Qwen2.5 0.5B documents 14 query heads and 2 KV heads. If the runtime or converter pretends these are equal-head tensors, logits can degrade catastrophically into repetitive or broken text even when the model weights themselves are good.

Prioritized runtime fixes

The highest-priority fix is to stop treating runtime-smoke artifacts as meaningful chat baselines. The loader and UI should surface the model’s source kind, quality-gate status, and manifest evidence directly in the runtime, and the app should visibly warn when a loaded .slm is only structural/runtime-smoke. Without that, debugging generation quality mixes two separate problems: runtime correctness and model quality. MiRust already documents this gate structure, so the runtime should expose it rather than bury it in documentation.

The next priority is to add real tokenizer identity and prompt-template fidelity to the .slm path. Today’s BTOK/BPE1 implementation is not enough for production conversion of mainstream small instruct models. TinyRustLM needs, at minimum, an embedded tokenizer identity checksum, explicit special-token mapping, decode-skip policy, and serialized chat-template metadata so the Rust/WASM runtime can reproduce the exact prompt bytes/token IDs that Hugging Face would produce for the same message list. For Llama-family models that can mean SentencePiece or Llama-style fast tokenizer equivalence; for Qwen it means preserving its larger BPE vocabulary and chat control markers correctly.

The third priority is to widen the tensor/model ABI to support GQA-class small models. This is the runtime change that unlocks the realistic candidate set. TinyRustLM’s current equal-head contract is too restrictive for SmolLM2, Qwen2.5 0.5B, and TinyLlama 1.1B, all of which expose fewer KV heads than attention heads. Adding distinct Q and KV projection shape support, plus corresponding KV-cache indexing, is more important than adding more sampler knobs because it determines whether modern small instruct models can be loaded correctly at all.

The fourth priority is to replace the current fixed stop rule with model-specific EOS and stop handling. The runtime should store and honor the converted model’s actual EOS token ID set, optional stop sequences, and decode-time skip rules. It should also record which stop criterion fired. Right now the operation notes document only one structural stop token, which is too narrow for general model conversion.

The fifth priority is sampling. TinyRustLM should keep deterministic argmax available, but it should not be the default for chat. A good first production preset is a modest stochastic sampler, such as temperature around 0.7, top_p around 0.9, top_k around 40–50, and a light repetition penalty around 1.05–1.15, with 1.0 remaining the no-penalty baseline. Hugging Face documents repetition penalty semantics explicitly, and TinyRustLM’s current defaults give you argmax with no repetition control.

The sixth priority is to make context-reset semantics impossible to misunderstand. “Reset context,” “clear transcript,” and “free model” should remain separate operations, but the UI must show the live KV length and whether visible transcript matches runtime context. For real chat, the app should also support an explicit “serialize transcript into prompt using model chat template” mode, because the current UI does not automatically build multi-turn prompts from history.

The seventh priority is load-path realism. TinyRustLM currently fetches the entire .slm, allocates equal-size WASM memory, and copies all bytes into the runtime. Its operations handbook also documents a current 128 MiB transfer ceiling. That ceiling blocks most realistic .slm candidates before generation even starts, especially because load peak includes both host-side fetched bytes and the equal-size WASM transfer allocation. For useful models, TinyRustLM needs chunked loading and ideally browser-local verified persistence rather than one-shot full-file copy.

Acceptance criteria and diagnostics

A real fix needs measurable acceptance criteria, not subjective “looks better” checks. The most important acceptance criterion is tokenizer parity: for a fixed corpus of prompts, the .slm tokenizer and prompt builder must produce the exact same token IDs as the source model’s reference tokenizer and chat template, including BOS/EOS and all special markers. TinyRustLM’s own docs already emphasize tokenizer identity, special IDs, and prompt/EOS slicing; production acceptance should harden this into golden tests.

A second criterion is prompt-template parity. Given the same array of chat messages, TinyRustLM should serialize to exactly the same model input tokens as the reference Hugging Face path for the target model family. This is especially important because SmolLM2, Qwen2.5, and TinyLlama all document chat-template usage rather than plain free-text prompting.

A third criterion is stop correctness. Generation must stop on the converted model’s real EOS/stop configuration, and the emitted UTF-8 text must exclude structural control tokens. A regression test should assert that the runtime records the exact stop reason: EOS token, stop sequence, max-new-tokens cap, or context-capacity boundary.

A fourth criterion is deterministic replay. For run_generation, the same model checksum, tokenizer checksum, prompt tokens, config, compiler/runtime identity, and seed should reproduce the same token sequence. For continuation mode, the runtime must also document that generate_next_token advances the runtime-owned RNG across calls, so equivalence between full-run and token-step modes is only expected when the initial RNG state and path are controlled.

A fifth criterion is context hygiene. After reset_context, rerunning the same prompt with the same config should yield the same output as a fresh-load run. After “Clear transcript” alone, the runtime should show unchanged KV length, making it impossible for hidden context to masquerade as a fresh session.

A sixth criterion is loop suppression. On a held-out set of short utility prompts—rewrite, summarize, explain code, extract JSON, answer a factual question—the model should avoid pathological single-token or short-n-gram looping under the production preset. This does not require perfect stylistic variety; it requires that generation terminate naturally and remain semantically on task. Because repetition penalty in mainstream generation stacks is defined as 1.0 = no penalty, the acceptance suite should explicitly compare no-penalty vs light-penalty behavior on the same prompts.

The diagnostics TinyRustLM should expose are equally important. The existing docs already mention model-loaded state, tokenizer output, logits summary, selected token, top candidates, KV length, quantization mode, and active sampling configuration. That is a good base, but for real debugging TinyRustLM should also print or expose: model checksum; tokenizer checksum; manifest source kind; model family/ABI version; attention-head and KV-head counts; prompt text hash; serialized prompt bytes hash; input token IDs; stop rule used; stop trigger observed; RNG mode and seed; repetition-penalty value; per-step selected token IDs; and a “visible transcript diverges from runtime context” flag.

Feasible small-model candidates for .slm

The ranking below is based on realistic TinyRustLM feasibility, not just model-card appeal. I am weighting permissive licensing, availability of official safetensors or widely used GGUF, instruction behavior, tokenizer practicalities, and the current TinyRustLM constraints that matter most: custom SLM1 only, 128 MiB one-shot transfer ceiling, scalar browser-thread execution, and an equal-head tensor ABI that does not yet support GQA layouts.

SmolLM2-360M-Instruct is the best overall first serious target after adding GQA support and a real tokenizer/chat-template path. It has an Apache 2.0 license, official safetensors, official GGUF availability, explicit Transformers.js support tags, a tied-embedding config that TinyRustLM already knows how to represent, and an instruct model card built around chat templating. The main friction is that it uses GQA (15 attention heads, 5 KV heads) and its full safetensors checkpoint is about 724 MB, so the current 128 MiB one-shot transfer model is not enough. Still, among practical small instruct models, it offers the best balance between size and expected quality.

Qwen2.5-0.5B-Instruct is the strongest small-model quality candidate in this size neighborhood, but it is a harder TinyRustLM target than SmolLM2-360M. It is Apache 2.0 licensed, has official safetensors and official GGUF, and its model card explicitly calls out stronger instruction following, long-text generation, structured output, and resilience to system prompts than Qwen2. It also uses chat templating. The main risks are its very large tokenizer vocabulary, its more complex chat control-token format, and GQA (14 Q heads, 2 KV heads). There is also ecosystem evidence that incorrect parsing of Qwen tokenizer JSON can produce vocabulary-count mismatches in native runtimes, which is exactly the kind of tokenizer bug TinyRustLM should avoid. This is likely the best quality candidate once the runtime is widened, but not the easiest first one.

SmolLM2-135M-Instruct is the best “small enough to try early in-browser” target, but not the best production-quality target. It is Apache 2.0 licensed, has official safetensors and GGUF ecosystem support, and belongs to the same family as the stronger 360M model. The reason it stays high in the ranking is practical: at 135M parameters, a 4-bit .slm is far more likely to fit a browser-friendly artifact budget than 360M, 500M, or 1.1B models. The downside is quality ceiling; MiRust’s own quality-gate logic would require scoped task evidence before any meaningful assistant claim, and at 135M the model should be expected to excel more at short rewriting, summarization, and bounded classification than open-domain chat. It also shares the family’s GQA requirement, so it still needs the tensor-ABI extension.

TinyLlama-1.1B-Chat-v1.0 is attractive because it is Apache 2.0, heavily used, widely quantized, and its model card explicitly says it uses the same architecture and tokenizer family as Llama 2, which makes conversion logic conceptually straightforward. But its practical feasibility is lower than its popularity suggests. The safetensors checkpoint is about 2.2 GB, the config shows GQA (32 attention heads, 4 KV heads), and even quantized browser deployment is likely to feel heavy in a synchronous scalar Rust/WASM runtime. It remains a good validation target once GQA support exists, but not the best first production path for TinyRustLM.

Models I would not prioritize for MiniModel.org-style distribution include MobileLLM-600M and OpenELM-450M-Instruct, not because they are technically impossible, but because their license positions are weaker for this use case. MobileLLM’s Hugging Face card is marked fair-noncommercial-research, and OpenELM uses Apple’s AMLR license rather than a plain permissive OSS license such as Apache 2.0. For a browser-local distribution channel, that license friction is enough to move them out of the main recommendation set.

Recommended first target and concrete conversion path

The recommended first production-quality .slm target is SmolLM2-360M-Instruct, but only after two runtime changes land first: GQA support and real tokenizer/chat-template metadata. The reason is that this model sits in the best current trade-off zone. It is materially stronger than 135M-class models, much lighter than 1B-class models, openly licensed, officially published in safetensors, already quantized in GGUF by multiple distributors, and explicitly tagged for Transformers.js/on-device style usage.

A realistic conversion path for SmolLM2-360M-Instruct is: start from official safetensors plus config.json, tokenizer.json, tokenizer_config.json, and special-token files; preserve its tokenizer identity and chat template semantics; convert tensors into SLM-native q8_0 and q4_0 layouts; store the model’s true BOS/EOS/PAD values and tied-embedding flag; extend SLM tensor metadata so Q and KV head counts are represented explicitly; and emit an eval sidecar that binds source-model checksum, converted .slm checksum, tokenizer checksum, and task-scope evidence. That path matches MiRust’s own distinction between structural validation, converted-trained provenance, and assistant-quality evidence.

If the goal is to get something useful fastest rather than best, the tactical path is different: first make TinyRustLM capable of loading a SmolLM2-135M-Instruct q4 .slm, because that is the one candidate most likely to approach present browser artifact limits after quantization, then immediately follow with SmolLM2-360M-Instruct once chunked loading and persistence are available. That gives you an early proof of real tokenizer/template correctness on a live instruct model without pretending that 135M is your end-state product model.

If the goal is to maximize small-model answer quality and structured output once the runtime is more mature, the next milestone should be Qwen2.5-0.5B-Instruct. But it should come after SmolLM2-360M, because Qwen’s tokenizer/control-token surface is a materially riskier first parser target for a custom Rust/WASM runtime.

Open questions and limitations

One hard limitation from the current evidence is that TinyRustLM’s implementation docs describe the runtime and checked source snapshot, but they do not publish a complete external compatibility matrix for every modern model family. The strongest conclusion I can make with high confidence is therefore not “model X will definitely run today.” It is: the current runtime must add GQA support, production tokenizer/template fidelity, and a less fragile load path before the realistic candidate set becomes viable. That conclusion is directly supported by the current SLM1 tensor ABI, tokenizer docs, and load-path docs.

A second limitation is that exact browser RAM ceilings vary by browser, device, and allocator behavior, and TinyRustLM’s present load path double-buffers artifacts during transfer. That means feasibility for 360M-and-up models is not just about final quantized weight size; it is also about transient load peak, KV cache, scratch, and the fact that inference still runs on the main browser thread with scalar Rust loops. The rankings above therefore treat “browser-feasible” as a runtime-engineering question, not just a model-card question.

The most important unresolved question for implementation planning is therefore not which model is prettiest on paper. It is whether TinyRustLM wants to remain an equal-head fixture runner, or whether it is ready to become a real small-model runtime. If it stays at the current SLM1/equal-head/128 MiB/greedy-default boundary, there is no genuinely strong production .slm target in this class. If it crosses that boundary, SmolLM2-360M-Instruct first, then Qwen2.5-0.5B-Instruct is the most credible path.