Tokenizer Prompt and Sampling Fidelity for TinyRustLM

Full report

On this page

Executive finding

TinyRustLM can preserve the real behavior of compact instruct models through .slm conversion and Rust/WASM generation only if it treats the tokenizer pipeline, chat template, model configuration, and sampler semantics as authoritative artifacts rather than as metadata to be approximated. The public TinyRustLM app already exposes the right observability hooks for this approach: it surfaces tokenizer import status, prompt and output limits, sampler capability, token console, checksums, provenance, and runtime diagnostics, while the published .slm format explicitly requires tokenizer inclusion, byte fallback, bounded decode, and tokenizer drift checks inside the artifact or a verified sidecar.

The most important architectural risk is that many compact instruct models rely on grouped-query attention and tokenizer/chat-template behaviors that are easy to lose during conversion. SmolLM2-135M-Instruct is a Llama-family model with num_attention_heads: 9 and num_key_value_heads: 3, while Qwen2.5-0.5B-Instruct uses num_attention_heads: 14 and num_key_value_heads: 2; both are therefore GQA models. Yet the public MiRust implementation notes for the inspected June 2026 TinyRustLM source snapshot say the executed path rejected GQA/MQA and required equal attention and KV-head counts. If that remains true in the active converter/runtime, TinyRustLM should refuse conversion of these models instead of silently approximating them. Silent downgrades here would not be “quantization drift”; they would be architectural defects.

That leads to a hard recommendation: TinyRustLM should be release-gated by a differential harness that first proves prompt-byte identity and token-ID identity, then proves first-step logits parity, and only then evaluates seeded multi-token output. If the token IDs do not match, the failure is tokenizer or template related. If the token IDs match but f32 logits diverge, the failure is in model execution. If logits match but seeded text diverges, the failure is in sampling, stop handling, or decode. This decomposition is the cleanest way to preserve real model behavior and to avoid tuning against a single observed rejection such as the reported "test" case.

Source-grounded constraints TinyRustLM must honor

TinyRustLM’s current public contract is intentionally narrow. The .slm format page describes a browser-local WASM runtime with a 33,554,432-byte model budget, explicit support for f32, q8_0, and q4_0, and in-artifact tokenizer support with byte fallback and compiled BPE metadata. The runtime page adds that generation uses preallocated structures, reusable logits, bounded decoding, and a fixed hot-path design, while the app UI exposes the diagnostics necessary to inspect tokenizer import, prompt budgets, output budgets, and sampler-related state.

The public model directory on MiRust also matters because it tells you what TinyRustLM is already using as deterministic fixtures. It lists TinyLM-16M smoke artifacts in f32, q8_0, and q4_0, plus tiny runtime fixtures, a tiny tied-output fixture, and a tiny BPE tokenizer fixture, all marked as deterministic runtime smoke rather than trained assistant-quality models. That is exactly the right base for a conformance suite: small enough for dense regression coverage, but explicit that passing the smoke path does not prove instruct-model fidelity.

The public TinyRustLM UI further shows a real product tension that the test plan must resolve. The selector exposes SmolLM2 360M Instruct q4_0 and SmolLM2 135M Instruct q8_0, and its presets expose temperature, top-k, top-p, seed, and max tokens. Meanwhile, the source-grounded MiRust implementation notes say the inspected runtime lacked GQA/MQA execution and used scalar CPU inference with direct WASM calls on the main thread. Those statements can both be true only if the live site has moved beyond the inspected snapshot, or if the listed models are placeholders requiring a different runtime path. The acceptance protocol should therefore verify the live path empirically instead of assuming parity from the selector.

MiniModel.org is useful here as a provenance layer rather than as a decoding authority. Its public site presents model metadata, manifests, examples, and browser-focused portable model artifacts, while the TinyRustLM app says the verified demo model autoloads from MiniModel.org and that prompts stay in the browser. That makes MiniModel metadata a good place to bind source identifiers, checksums, fixture manifests, and golden-vector records, but not a substitute for exact tokenizer and generation configuration pulled from the reference model files.

Tokenizer and chat-template conformance suite

Tokenizer fidelity must start from the full tokenizer graph

A converter that rebuilds a tokenizer from only vocab.json and merges.txt is not sufficient for instruct-model fidelity. Hugging Face’s tokenizer pipeline is explicitly componentized into normalizers, pre-tokenizers, models, post-processors, decoders, and added tokens, and the BPE model configuration itself includes behavioral fields such as unk_token, continuing_subword_prefix, end_of_word_suffix, fuse_unk, byte_fallback, and ignore_merges. The converter should therefore preserve the entire tokenizer.json graph, plus tokenizer_config.json, special_tokens_map.json, and generation_config.json, and it should hash those bytes into the .slm provenance record.

For byte-level BPE families, preserving the byte pipeline exactly is especially important. Hugging Face documents that the ByteLevel normalizer maps every byte 0–255 into visible Unicode stand-ins, the ByteLevel pre-tokenizer has behavior controlled by fields such as add_prefix_space, trim_offsets, and use_regex, and the ByteLevel decoder reverses that mapping back into the original bytes. That means a converter cannot safely “simplify” whitespace handling, prefix-space rules, or offsets without changing token IDs or decode text.

SmolLM2 appears to sit in this byte-level BPE family. Its model files include tokenizer.json, vocab.json, and merges.txt, and the raw tokenizer vocabulary visibly uses GPT-2-style space markers such as Ġ, plus newline-like symbols such as Ċ, which is the hallmark of byte-level BPE encodings. The model card also instructs users to rely on apply_chat_template() before generation rather than hand-formatting prompts. TinyRustLM should therefore treat SmolLM2 conversion as a byte-level BPE import with exact template rendering, not as a generic “Llama-ish” tokenizer reconstruction.

Qwen2.5 shows why added-token fidelity matters. Its tokenizer_config.json exposes an added_tokens_decoder with per-token flags like lstrip, rstrip, normalized, single_word, and special, and it distinguishes truly special tokens such as <|im_start|> and <|im_end|> from non-special tokens like <tool_call>. If TinyRustLM stores only token strings and IDs but loses those behavioral flags, it can break both encoding and decoding around tool calls, role markers, and stop conditions.

A second easy-to-miss field is clean_up_tokenization_spaces. Qwen2.5 explicitly sets clean_up_tokenization_spaces to false. That is consistent with instruct/chat models that need exact whitespace reproduction, especially around JSON, XML-like tool markup, and code. TinyRustLM should therefore never apply any post-decode whitespace cleanup unless the imported tokenizer configuration explicitly requires it.

Exact template serialization is part of model behavior

Hugging Face’s chat templating documentation is clear that formatting mismatches silently harm model performance and that models should be driven through tokenizer.apply_chat_template() using the chat template embedded in the tokenizer config. The add_generation_prompt flag is specifically designed to append the model-specific assistant-prefix needed to start a response, and using the wrong template can produce valid tokenization while still yielding the wrong behavior.

Qwen2.5’s tokenizer config demonstrates exactly how model-specific this can be. Its template inserts a default system message when the first message is not system, serializes messages with <|im_start|>{role}\n...<|im_end|>\n, has special branches for assistant tool calls and tool responses, and appends <|im_start|>assistant\n when add_generation_prompt is set. It also treats the first system message differently from later system messages. TinyRustLM should preserve the raw Jinja template bytes and the exact rendered prompt bytes in diagnostics, because any “equivalent” hand-coded serializer is likely to drift.

SmolLM2’s public model card does not spell out the full template in the card text, but it does explicitly show tokenizer.apply_chat_template(..., add_generation_prompt=True, tokenize=True, return_dict=True) as the intended inference path. That is enough to establish the rule: for compact instruct models, the chat template is not optional metadata; it is a required part of prompt construction.

The conformance suite should therefore include at least these prompt classes: plain ASCII; leading-space versus no-leading-space inputs; repeated spaces; tabs and mixed newlines; punctuation adjacency; JSON; code fences; XML-like tool tags; empty content; system-user-assistant multi-turn chats; chats without a system message; consecutive tool messages; Unicode NFC/NFD pairs; emoji including ZWJ sequences; CJK; right-to-left text; and invalid UTF-8 byte-handling cases where applicable. That recommendation follows directly from the documented tokenizer component behavior, the explicit system/default logic in Qwen’s template, and TinyRustLM’s own reported invalid-UTF-8 smoke coverage.

BOS, EOS, PAD, UNK, stop behavior, and context budgeting must be model-specific

Special-token handling cannot be normalized across model families. SmolLM2’s config sets bos_token_id: 1, eos_token_id: 2, and pad_token_id: 2. Qwen2.5’s config sets bos_token_id: 151643 and eos_token_id: 151645, while its generation config broadens EOS handling to the set [151645, 151643] and uses do_sample: true, repetition_penalty: 1.1, temperature: 0.7, top_p: 0.8, and top_k: 20. TinyRustLM should therefore import stop-token semantics from the reference generation config instead of assuming a single EOS ID or a single “balanced” default sampler for every model.

The same rule applies to context budgeting. SmolLM2’s config exposes max_position_embeddings: 8192, while Qwen2.5’s tokenizer config exposes model_max_length: 131072 and the model config exposes max_position_embeddings: 32768 with a large rope_theta. TinyRustLM’s own prompt-handling docs say prompts with length greater than or equal to the runtime context are rejected, because generation requires at least one slot beyond the prefill boundary. In production, the effective prompt budget should therefore be min(rendered-template tokens, model context rule, runtime context rule, reserved output budget) rather than a single global truncation threshold.

On decoding, TinyRustLM’s inspected implementation removes one terminal EOS before prefill and treats only token ID 257 as a structural stop token in its deterministic fixture path. That is acceptable for internal fixtures, but it is too narrow for imported instruct models with richer stop conditions or multiple EOS IDs. The import pipeline should therefore map imported stop-token IDs and string stop sequences into a per-model stop table, and it should log both token-level and string-level stop matches.

Differential testing protocol

The six-way comparison should be staged, not collapsed

The correct comparison order is the one you specified, but it should be run as a pipeline with early exits:

1. Reference framework token IDs from the official tokenizer and official chat template.
2. Converted .slm token IDs from TinyRustLM’s imported tokenizer/template path.
3. Reference first-step logits from the official framework after prefill on the exact same token IDs.
4. Rust-native first-step logits from the native TinyRustLM runtime.
5. WASM first-step logits from the browser path.
6. Multi-token seeded output only after steps one through five are within tolerance.

This staging matters because first-step logits are the clean boundary between prompt correctness and decode correctness. MiRust’s generation notes say the prompt is tokenized, terminal EOS is removed for continuation, every prompt token is forwarded sequentially to build the KV cache, and the resulting logits are then sampled for the next token. So if token IDs match but first-step logits do not, the defect must be in execution rather than in prompt serialization.

Golden artifacts should be captured as model-bound records

For each converted model revision, TinyRustLM should capture a “golden prompt pack” containing these exact fields: model source ID; source revision; SHA-256 of all tokenizer-related files; SHA-256 of config.json; SHA-256 of generation_config.json; rendered prompt UTF-8 bytes; rendered prompt SHA-256; token IDs; prompt length; first-step logits as f32; top-32 token IDs with logits and probabilities; stop-token table; and a seeded multi-token continuation trace including the RNG seed and sampling parameters. MiniModel-style manifests are a natural place to store the metadata and checksums, even if the actual tensor/logit blobs live beside them.

Because repository execution was not available here, this report cannot fill in numeric golden vectors for a specific model revision. What it can do, and what is more durable, is specify the exact golden-set schema that should be materialized once per model revision from the reference framework and then treated as the release authority for the Rust and WASM paths. That is consistent with MiRust’s explicit source-of-truth boundary and with TinyRustLM’s provenance-first design.

Recommended tolerance bands

The thresholds below are engineering recommendations for release gates. They are intentionally tight for f32 and progressively wider for lower bit depths. For q6 and q5, which the published TinyRustLM docs do not yet list as supported runtime formats, these thresholds are future-ready rather than current-product claims.

For f32, require exact rendered prompt bytes, exact token IDs, exact prompt length, top-1 next-token match on every case, top-5 set equality on at least 99.9% of prompts, cosine similarity of the full first-step logits vector of at least 0.999999, and max_abs_diff <= 1e-4 between reference and WASM, with a tighter <= 5e-5 target for native Rust. For q8, keep exact prompt parity, require top-1 match on at least 99% of prompts, top-5 overlap on at least 99.5%, cosine similarity of at least 0.9995, and max_abs_diff <= 2e-2 on normalized first-step logits. For q6, require top-1 match at least 98.5%, top-5 overlap at least 99%, cosine at least 0.9990, and max_abs_diff <= 4e-2. For q5, require top-1 match at least 98%, top-5 overlap at least 99%, cosine at least 0.9985, and max_abs_diff <= 6e-2. For q4, require top-1 match at least 96%, top-5 overlap at least 98.5%, cosine at least 0.9970, and max_abs_diff <= 1e-1. These should be evaluated only after prompt parity is perfect.

For seeded multi-token outputs, use two separate gates. Under greedy settings, seeded output must be byte-identical across reference, native Rust, and WASM for all supported precisions. Under stochastic settings, native Rust and WASM should be byte-identical to each other within the same build when logits and candidate ordering match, while reference-framework identity is a stricter optional gate because different frameworks may differ in tie-breaking and floating-point ordering even with the same seed. MiRust’s own docs emphasize that matching seed and settings are necessary but not sufficient across runtimes because floating-point ordering, softmax details, quantized kernels, and candidate tie handling all affect replay.

How to distinguish drift from defects

If rendered prompt bytes differ, the defect is in chat-template serialization or message normalization. If rendered bytes match but token IDs differ, the defect is in tokenizer behavior: normalization, pre-tokenization, added-token handling, byte handling, or special-token insertion. If prompt parity is exact but f32 first-step logits diverge, the defect is in model execution, not quantization.

A GQA defect is the first thing to suspect when the failing model has num_key_value_heads != num_attention_heads and the runtime either rejects loading or produces grossly wrong logits from the first token onward. A RoPE defect usually shows as short-prompt passes and long-prompt failures, with divergence increasing with token position. An RMSNorm defect usually causes smooth, prompt-wide drift even at short lengths. A causal masking defect often passes one-token or two-token cases and fails on multi-token prompts where attention should be blocked. A KV-cache defect often passes first-step logits but diverges on the second sampled token or on generate_next_token continuation. A sampler defect is present when logits and candidate sets match but seeded output diverges anyway.

Quantization drift should look different. It should preserve prompt parity, generally preserve top candidates most of the time, and widen gradually as precision decreases. It should not systematically produce different BOS/EOS behavior, broken tool tokens, collapsed whitespace, or prompt rejection on simple strings. Those are far more consistent with tokenizer, template, or stop-rule defects than with q4/q8 approximation.

Sampler experiment matrix and recommended presets

The experiment matrix should test semantics separately from model quality

The published TinyRustLM implementation notes describe a sampler with deterministic greedy defaults, validated temperature/top-k/top-p, fixed 1,024-candidate storage, top-p truncation, greedy fallback when probability mass becomes non-finite or zero, and a project-owned XorShift64 PRNG. Hugging Face and vLLM document the broader sampling surface that compact instruct models commonly depend on today: temperature, top-k, top-p, min-p, typical sampling, repetition penalty, frequency penalty, presence penalty, seed, stop strings, stop-token IDs, and max-token controls. TinyRustLM should therefore treat the current sampler as the minimum parity surface and expand toward the documented HF/vLLM surface when preserving real instruct-model behavior.

A broad sampler matrix should include: greedy parity runs; temperature sweeps at fixed top-k/top-p; top-k-only sweeps; top-p-only sweeps; min-p-only sweeps; typical-p-only sweeps; repetition-penalty sweeps; presence/frequency penalty sweeps; stop-token versus stop-string sweeps; premature-EOS stress tests; degenerate-loop stress tests; and context-edge tests where the remaining output budget is small. Each of those should be measured against first-step candidate parity and seeded continuation parity, not just “did the answer look good.”

The prémature-EOS tests are especially important because TinyRustLM’s internal fixture path uses a single structural stop token, while real instruct models may have multiple EOS IDs and additional string stops. Qwen2.5’s generation config is a concrete example: it samples by default and uses two EOS IDs. A runtime that collapses that to one structural EOS or strips stop strings incorrectly can terminate early even when the core transformer is correct.

For degenerate loops, the reference design should include explicit pattern-detection diagnostics similar to vLLM’s RepetitionDetectionParams, which detect repeated n-gram patterns and terminate pathological runs early. TinyRustLM does not need to copy vLLM’s exact API, but it should log whether a stop happened because of EOS, max output, string stop, token stop, or loop detection, and it should record the triggering pattern. That is the shortest path to localizing “repetitive nonsense” defects without guessing.

Recommended generation presets

The presets below are engineering recommendations synthesized from TinyRustLM’s current UI surface, model defaults visible in public generation_config.json files, and the parameter semantics documented by Hugging Face and vLLM. They should be treated as defaults to test and tune, not as universal truths.

For chat, use temperature 0.7, top-k 40, top-p 0.9, min-p 0.03–0.05 if implemented, repetition penalty 1.05, presence penalty 0.0–0.1, frequency penalty 0.0–0.1, and a conservative max-output budget such as 384–512 tokens. This tracks TinyRustLM’s public balanced preset closely while adding a mild anti-repetition layer.

For precise, use temperature 0.1–0.2, top-k 20, top-p 0.9–1.0, min-p 0, typical-p 1.0, repetition penalty 1.03–1.08, and a fixed seed for reproducibility. This is the mode to use for evaluation, differential testing, and answerability-sensitive tasks.

For creative, use temperature 0.85–0.95, top-k 80, top-p 0.95, min-p 0.03, typical-p 0.95, repetition penalty 1.01–1.05, presence penalty 0.2, frequency penalty 0.1, and a larger max-output budget. This increases diversity but still bounds the candidate pool.

For code, use temperature 0.15–0.3, top-k 40–60, top-p 0.9–0.95, min-p 0, typical-p 1.0, repetition penalty 1.02–1.05, and no decode whitespace cleanup. Code generation is unusually sensitive to whitespace and delimiter fidelity, so tokenizer and stop-string correctness matter as much as the sampling values.

For JSON, use temperature 0–0.1, top-k 1–10, top-p 1.0, min-p 0, repetition penalty 1.0–1.03, and explicit stop strings or token stops aligned to the target wrapper if the application adds one. For strict JSON extraction, greedy or near-greedy decoding is usually the safest baseline.

Ambiguous-input behavior should prefer clarification over invention

For prompts that are materially underspecified, TinyRustLM should route through a “clarify-first” behavior rather than pushing the model toward a fabricated answer. The generated clarification should be short, should preserve the user’s terminology, should ask only for the missing slot that blocks a reliable answer, and should avoid adding speculative assumptions in the system layer. This is best implemented at the prompt-policy level, but it still depends on exact chat-template rendering because many instruct models treat system and assistant prefixes differently.

A practical rule is: if two or more materially different answers are plausible from the current prompt, the model should ask one targeted follow-up question before answering. The evaluator for this behavior should score not just whether the model asked a question, but whether it asked a useful question. Examples include requesting the target language for translation tasks, runtime/framework details for coding questions, geographic scope for recommendations, or the intended schema for JSON requests. This gate is part of “no answer fabrication,” not an optional UX nicety.

Diagnostics and failure localization

TinyRustLM’s public diagnostics already expose many of the right anchors: tokenizer import, source validation, checksums, prompt limit, output limit, sampler cap, token console, prompt token count, generated token count, scratch usage, and runtime provenance. Those fields should be preserved and expanded rather than replaced, because correctness debugging is faster when the app shows the exact state that affected the run.

For tokenizer and template debugging, the app should log the rendered chat bytes, rendered chat SHA-256, token IDs, special-token map, added-token flags, BOS/EOS/PAD/UNK IDs actually used, and whether cleanup or normalization rules were applied. Without that layer, a simple prompt rejection can look like a model problem when it is really a template, prefix-space, or stop-token mismatch.

For model-execution debugging, the app should log first-step top-N logits before sampling, the post-penalty logits, the post-top-k candidate set, the post-top-p set, the final normalized probabilities, the chosen random draw, the selected token, the current position, RoPE parameters, attention heads and KV heads, and whether the model is in an unsupported GQA/MQA configuration. Those fields make it possible to say “the logits were already wrong before sampling” or “the logits were correct but the filtered candidate set was wrong.”

For stop-rule debugging, the runtime should record the exact termination reason from the set {eos_token, stop_token_id, stop_string, max_output, context_full, loop_detected, error} and, for stop strings, the exact matched byte span. vLLM’s docs are helpful here because they distinguish stop strings from stop_token_ids, define whether stop strings are included in output, and separate EOS ignoring from stop semantics. TinyRustLM should expose the same distinction even if its public API remains smaller.

For the reported "test" rejection case specifically, the right diagnosis path is not to tune that string. It is to inspect: rendered prompt bytes; token IDs; total prompt length after template expansion; whether a terminal EOS was removed correctly; whether the prompt length hit the context boundary rule; whether sampling parameters were accepted by validation; and whether the run failed before or after prefill. Those are precisely the fault lines documented in TinyRustLM’s public runtime and MiRust implementation notes.

Production acceptance thresholds with no answer fabrication

A production-worthy acceptance policy should have four hard gates. The tokenizer gate requires exact rendered prompt bytes and exact token-ID parity on the full conformance suite. The execution gate requires first-step logits to remain within the precision-specific bands above. The sampler gate requires greedy output identity and stable seeded stochastic behavior within a build. The behavior gate requires the model to clarify materially ambiguous prompts instead of inventing missing facts. Any failure in any gate should mark the conversion as unsupported rather than silently shipping degraded behavior.

For tokenizer and prompt fidelity, the acceptance threshold should be absolute: zero mismatches across the conformance suite for rendered prompt bytes, token IDs, special-token insertion, and decode round-trips. This is because tokenizer/template drift is categorical, not probabilistic. A single mismatch can completely change model behavior on an instruct prompt.

For architecture compatibility, the acceptance threshold should also be absolute: if the imported model requires features the runtime does not implement, such as GQA/MQA in the inspected TinyRustLM source snapshot, conversion should fail with an explicit diagnostic. A converter that “sort of runs” the wrong attention shape is not acceptable.

For premature EOS and output budgeting, require zero unexpected early stops on the differential suite outside documented stop conditions, 100% respect for max-output limits, and 100% respect for context boundaries. TinyRustLM’s runtime docs already frame prompt capacity and output limits as correctness concerns rather than UI decorations, and that is the right stance for release gating.

For answer fabrication, require that on an ambiguous-input benchmark the system ask a useful clarification question in at least 95% of cases where the prompt is materially underspecified, and that it emit an explicit limitation or request for clarification rather than a fabricated concrete answer in 100% of evaluator-confirmed no-answer cases. This is an engineering recommendation rather than a public-source threshold, but it follows directly from the broader fidelity goal: preserving real model behavior is not enough if the product layer still turns uncertainty into fiction.

The final operational rule is simple: do not ship approximation when you can ship a truthful refusal. If TinyRustLM cannot yet prove tokenizer parity, template parity, GQA parity, first-step logit parity, and seeded sampler parity for a given compact instruct model, the correct production behavior is to decline that conversion path with a specific diagnostic. For a browser-local Rust/WASM runtime built around bounded validation and provenance, that is not a limitation of the design. It is the design working as intended.