Black-Box Failure Analysis of TinyRustLM

Full report

On this page

Scope and method

This investigation was constrained to public evidence only: the live TinyRustLM site, MiniModel metadata, public MiRust/TinyRustLM implementation documentation, and authoritative external model documentation. I was able to inspect the live public UI state and the audited implementation/public manifests, but I could not independently execute browser-controlled prompt runs from this environment without fabricating outputs. The report therefore separates directly observed public evidence from high-confidence engineering inferences and from explicit unknowns. That distinction matters here because the strongest public evidence already isolates several root causes before any prompt-level replay: the live UI shows a load-state failure, the public MiniModel demo catalog points to a synthetic smoke fixture, and the audited TinyRustLM implementation docs state that shipped artifacts are regression artifacts rather than assistant-quality models.

The most important conclusion is straightforward: the publicly visible behavior is not primarily explained by sampling knobs. It is explained first by artifact identity and runtime compatibility. The MiniModel example catalog exposes a synthetic q8 fixture with an 8,832-byte artifact, a tiny hidden=8, layers=1 shape, context 16, max_tokens/max_new_tokens of 16, and trained_quality_claim: "not-claimed". MiRust’s audited artifact catalog likewise says the supplied SLM artifacts are executable regression artifacts whose manifests explicitly deny trained assistant quality, and its runtime-smoke gate explicitly says a pass does not mean useful language behavior.

What the public evidence proves

The live public page currently shows a contradictory surface. It says it is “Loading the verified MiniModel demo”, then immediately presents “Something needs attention” and “Check the selected model and try again.” The same page shows the model selector headed by “SmolLM2 360M Instruct q4_0” and “SmolLM2 135M Instruct q8_0”, yet the runtime panel still says Model: waiting, benchmark counters remain zero, provenance fields remain waiting, and the footer says “No model loaded.” That is direct public evidence of a pre-generation load/state failure on the public chat surface.

A second contradiction appears when the live UI is compared with the audited implementation snapshot. The audited implementation says the browser boot sequence loads the default tinylm16q8 route, and that the selector maps to only eight fixed same-origin local .slm paths: TinyLM-16M in three quantizations plus five tiny fixtures. The same audited docs say TinyLM-16M-shaped options default to eight new tokens and tiny fixtures default to sixteen. That does not match the current live UI, which publicly renders a 1024-token preset and SmolLM2 labels first in the selector. The most plausible engineering reading is front-end/source drift or a metadata-first UI that is presenting candidate model labels not backed by the audited local artifact set.

The MiniModel public catalog also matters because it identifies what the “verified demo” actually is. MiniModel’s example catalog exposes tinyrustlm/tiny-test-model-q8, a synthetic fixture with artifact_byte_count: 8832, slm_model_shape: vocab=260,hidden=8,layers=1,...,context=16, quantization q8_0, source_kind: "synthetic", source_identity: "tinyrustlm-fixture-q8", quality_claim: "not-endorsed-by-project", and trained_quality_claim: "not-claimed". That is not a weak assistant. It is a deliberate execution/regression fixture.

MiRust’s artifact catalog and model pages reinforce that point. The audited catalog lists only eight bundled SLM artifacts: three TinyLM-16M variants and five tiny fixtures. It then states, plainly, that these artifacts are “executable regression artifacts” and that their manifests “explicitly deny trained assistant quality.” The specific TinyLM-16M q8 page further says its role is “runtime-smoke only” and that it has “Deterministic smoke weights; no trained assistant quality.”

There is also hard evidence that, even if the site were trying to use a real SmolLM2 model, the current runtime would not be a clean fit. SmolLM2-360M-Instruct’s official config shows num_attention_heads: 15 and num_key_value_heads: 5, which is Grouped Query Attention rather than equal-head MHA. TinyRustLM’s current implementation limitations page says the runtime currently requires equal attention and KV-head counts. That makes a real SmolLM2-360M-class import a compatibility problem unless there is an unpublished adapter or a runtime change not present in public evidence.

Finally, the public evidence shows a likely chat-format mismatch for any instruct-class model that might eventually load. The live UI’s transcript behavior documentation says each form submission sends only the current prompt field to generate, and earlier transcript turns are not concatenated automatically. Hugging Face’s SmolLM2 docs and tokenizer config show that the instruct model expects a chat template built around <|im_start|>...<|im_end|> messages, and Hugging Face’s chat-template docs warn that using a format different from the training format can cause severe, silent performance degradation. So even a successfully loaded SmolLM2-derived artifact would underperform badly without model-specific templating and conversational prompt serialization.

Raw evidence captures

The following short public captures are the most load-bearing pieces of evidence:

“Loading the verified MiniModel demo.”

“Something needs attention. Check the selected model and try again.”

“No model loaded”

“Load the default tinylm16q8 route.”

“Pass means the artifact is admitted for local runtime execution evidence. It does not mean useful language behavior.”

"trained_quality_claim": "not-claimed"

"num_attention_heads": 15 and "num_key_value_heads": 5

These captures also explain why there are no trustworthy screenshot attachments in this report: the available public tooling here can inspect and cite the live page text and public manifests, but it cannot safely fabricate browser-run screenshots or prompt outputs that were not actually executed.

Reproducible prompt matrix

Because I could not legally or technically fabricate 30 executed generations, the matrix below is a reproducible execution plan grounded in the public configuration state. For every row, the public live UI currently renders these inherited operator values: Max Tokens 1024, Temperature 0.70, Top-K 40, Top-P 0.90, with the selector headed by SmolLM2 360M Instruct q4_0 and SmolLM2 135M Instruct q8_0, while runtime state still says waiting and No model loaded. If the MiniModel demo actually resolves instead, the only publicly exposed example demo artifact is the tiny synthetic q8 fixture with context 16 and max tokens 16.

For each run, the local operator should record exactly the public runbook fields MiRust prescribes: prompt bytes or approved redacted prompt, model identity, sampling configuration, max token request, result message, numeric code, diagnostics JSON, browser elapsed time, and console entry. The diagnostics surface is documented to expose model-loaded state, last error, prompt/generated token counts, KV length, quantization mode, tokenizer output, logits summary, selected token, top-k summary, and active sampling configuration.

Two engineering notes matter for interpreting this matrix. First, the live public state suggests that many of these rows will currently fail before generation because no usable model is loaded. Second, if the MiniModel demo is what eventually loads, the public artifact metadata makes it unsuitable for this matrix by construction: context 16, output 16, synthetic/non-trained source, and no assistant-quality claim.

Failure taxonomy ranked by confidence and impact

The table below ranks the user’s proposed failure classes by public-evidence confidence and practical impact.

The short version is that the failure stack is layered, but its root is not subtle. The first-order cause is that the publicly surfaced “demo/default” path is tied to non-assistant artifacts or to an unresolved load path. The second-order causes are that the runtime, as publicly documented, lacks the compatibility features needed for a real modern instruct model such as SmolLM2, and the UI lacks the prompt-formatting and context-serialization behavior such a model expects.

Immediate, intermediate, and architectural corrections

The immediate correction is not to “fix responses.” It is to stop presenting a regression fixture as if it were a useful local assistant default. The public chat surface should either ship with no autoloaded chat model or it should autoload only a converted-trained artifact that has actually passed the assistant-quality gate. MiRust’s own public docs already define that boundary: assistant-quality requires source_kind = converted-trained plus evaluation sidecars and a quality decision accepted for assistant-quality, while the current fixture explicitly is not product evidence.

The second immediate correction is to make the selector truthful. If a selector row is only metadata discovery or a future conversion candidate, it should not appear as a ready local runtime choice beside actual loadable .slm routes. The audited selector contract says a selector entry proves only that the static application knows a route; it does not prove runtime acceptance or quality. The live UI should therefore clearly separate verified local artifacts, importable local files, conversion candidates, and metadata-only model cards.

The third immediate correction is to bind prompt construction to model identity. For instruct models, TinyRustLM needs a model-specific chat-template layer and it needs to serialize prior turns into the next prompt. Right now the public docs say the UI sends only the current turn, and Hugging Face explicitly warns that using the wrong training format causes severe silent degradation. This is one of the highest-return fixes because it improves every future trained model without weakening any safety or quality gate.

Intermediate work should focus on real compatibility rather than cosmetics. The runtime needs GQA-capable attention and KV-cache handling if it wants to support current instruct models such as SmolLM2-360M. It also needs a worker-owned runtime so generation moves off the main thread, because the public docs say the current monolithic generate call is synchronous and non-cancellable. Finally, it needs verified artifact admission and a persistent local artifact store so the browser is not forced to refetch and recopy full model files on each usage path.

Architecturally, the project should decide whether it wants to remain an SLM1-only custom-runtime experiment or become a credible browser LLM runtime. Its own roadmap already points in the right direction: trained-model admission, worker ownership, a verified local store, and a true compatibility path rather than aliasing formats. Public MiRust docs explicitly say current TinyRustLM has no GGUF parser, uses scalar CPU loops, supports one model at a time, and still treats provenance sidecars as informational rather than authenticating admission. A production path needs those boundaries closed, not hidden.

Proposed production default and quantitative targets

Under the current publicly documented runtime constraints, the only plausible production default that could provide useful language behavior is not SmolLM2-360M-Instruct q4_0 and not SmolLM2-135M-Instruct q8_0. Public file sizes make both of those bad fits for the current architecture: the current runtime has a 128 MiB single-transfer ceiling, SmolLM2-135M-Instruct Q8_0 is publicly listed at 0.14 GB, and SmolLM2-360M-Instruct Q4_0 is publicly listed at 0.23 GB. Both exceed the present transfer model’s practical ceiling. By contrast, SmolLM2-135M-Instruct Q4_0 is publicly listed at about 0.09 GB, which is the only one of the obvious candidates that fits the current transfer envelope.

So the best near-term production default is this:

Converted-trained SmolLM2-135M-Instruct q4_0 in SLM1, with model-specific chat template application, multi-turn prompt serialization, truthful provenance labeling, and assistant-quality gate acceptance. That is the highest-utility model choice that is still compatible with the public runtime’s current size ceiling, assuming the runtime is extended far enough to support the model’s tokenizer/control-token semantics and its instruct chat template.

Once the runtime adds worker execution, chunked/verified loading, persistent storage, and GQA support, the better long-term default becomes SmolLM2-360M-Instruct in a browser-suitable quantization, because the 360M class is explicitly intended for compact on-device use and is meaningfully more capable than the 135M class. But that should be a phase-two default, not a launch default for the current implementation.

The quantitative targets below are therefore proposed engineering targets, not claims about current TinyRustLM performance:

Those targets are chosen to close the precise public gaps identified above: the current transfer cap, the current lack of worker execution, the current main-thread sync path, the current lack of persistent artifact storage, and the current mismatch between a useful instruct model and the runtime’s admitted artifact set.

Explicit unknowns that public evidence cannot establish

Several things remain unknowable from public evidence alone.

First, I cannot truthfully provide a completed table of 30 executed prompt outputs, latencies, and token counts from this environment, because I could not independently drive the live site through all prompt submissions. Public evidence supports the diagnosis, but it does not justify invented run outputs.

Second, I cannot establish whether the SmolLM2 selector entries on the live page are inert labels, metadata-only candidates, pending conversion routes, or real loadable artifacts backed by an unpublished runtime build. The public implementation audit from 2026-06-25 says the selector maps to eight fixed local SLM artifacts, while the live site now renders SmolLM2 choices. That discrepancy is real, but its exact operational cause is not public.

Third, I cannot establish whether a real converted-trained SmolLM2 SLM artifact exists privately and merely failed to load, or whether the live UI is attempting to bootstrap only the MiniModel fixture and never reaches a trained model path. No public signed manifest or artifact bytes are available to settle that. MiniModel’s public example content is still only the tiny synthetic fixture.

Fourth, I cannot prove a current tokenizer/BOS/EOS/control-token bug in a real trained artifact path. I can show that TinyRustLM has an explicit tokenizer/control-token contract and that SmolLM2 expects its own special tokens and chat template, but without a public converted SmolLM2 SLM and a reproducible run, that remains a plausible integration risk rather than a demonstrated live bug.

Fifth, I cannot prove the existence of a separate post-generation rejection filter that is falsely discarding otherwise good model text. The public evidence I found points to model-load/state rejection and to artifact-quality mislabeling, not to a hidden response-quality filter rejecting outputs after successful generation.

Taken together, the public evidence is still enough to reach a strong engineering verdict: TinyRustLM’s bad default behavior is best explained by the wrong artifact class on the chat surface, incompatible or unresolved runtime/model wiring, and missing instruct-model prompt serialization, not by a merely “weak model” and not by superficial sampling tweaks.