Honest Quality Gate and Production Delivery for TinyRustLM

Full report

On this page

As of July 11, 2026, the public TinyRustLM surface shows a serious browser-local execution and verification story, but not yet a publicly evidenced “genuinely useful default assistant model.” TinyRustLM’s public app emphasizes local browser inference, verified local .slm imports, receipt and gate telemetry, and prompts staying in the browser, while its runtime docs describe a browser-local Rust/WASM stack with no external analytics and a 33,554,432-byte model budget. At the same time, the audited public MiRust model catalog lists only experimental deterministic smoke artifacts and tiny fixtures, repeatedly stating that no trained assistant quality is claimed. MiniModel.org likewise describes itself as a checksum-bound catalog, verifier, and peer-transfer layer rather than a model host or inference service, and its current public availability feed is empty. On that evidence, the honest product posture is: ship the smoke artifact now, but do not auto-promote any public default chat model until it passes the raw-output gates below.

What the current public evidence actually shows

TinyRustLM’s public app and docs are unusually explicit about execution boundaries. The site says prompts stay in the browser, local .slm files are checked before they run, and the runtime is browser-local Rust/WASM with local-only diagnostics and no external analytics. The public UI also exposes receipt fields for origin, fetch counts, checksums, bytes, module bytes, token budgets, route hashes, phases, dependencies, trace, and lineage/gates, which is exactly the kind of observability a no-cheat promotion system should preserve rather than hide behind polished UI output.

The public evidence for model usefulness, however, is materially weaker than the evidence for runtime integrity. TinyRustLM’s UI shows selectable names such as “SmolLM2 360M Instruct q4_0” and “SmolLM2 135M Instruct q8_0,” but the public MiRust verified model catalog currently lists only eight experimental artifacts, all labeled as deterministic smoke or tiny fixtures, and explicitly says that no trained assistant quality is claimed. One publicly listed deterministic smoke artifact is a 17,048,064-parameter q8 model with a measured file size of 17,160,000 bytes, which is appropriate for runtime verification but not sufficient public evidence for a default assistant promotion.

MiniModel.org strengthens delivery trust but not current utility claims. Its public boundary statement says it is not a model hosting or inference service; instead it focuses on manifests, checksums, provenance, local verification, peer-piece transfer, share metadata, and browser handoff into TinyRustLM. Its roadmap and first-build layer already describe artifact SHA-256, chunk hashes, chunk-list SHA-256, Merkle roots, peer-offer proofs, receipt sidecars, and metadata-only ledgers, but the public feed is currently empty. That means the stack is already well aligned with integrity-first delivery, consent-based P2P import, and local verification, but the honest state of the world is that public availability evidence is still pre-default.

Current browser-local AI practice also reinforces the need to separate raw-model evidence from wrapper behavior. WebLLM offers OpenAI-style browser-local inference without server-side processing, and Transformers.js exposes high-level pipelines that bundle preprocessing and postprocessing with model execution. Chrome’s Prompt API similarly separates model availability checks, download progress, and model creation from application logic. Those ecosystems are useful references for UX and delivery, but they also show why the TinyRustLM gate must hook the decoder boundary rather than any pipeline, JSON mode, or UI rendering layer.

The no-cheat evaluation architecture

The right evaluation frame is scenario-based, multi-metric, contamination-aware, and explicit about what is being measured. HELM argues for broad scenario coverage and multi-metric reporting instead of a single headline score; IFEval shows why instruction following should be tested with directly verifiable constraints; IFBench shows that models can overfit known instruction formats and still fail to generalize to new constraints; and contamination research has shown that both exact-match and softer semantic leakage can inflate benchmark performance if split construction is careless. For TinyRustLM, that means promotion must rely on untouched raw completions scored across many task types, with shadow and sealed holdouts built to resist both template-family and semantic-neighbor leakage.

The four-way split should be structurally separated. Public development cases are the only cases the model team may see repeatedly. Generated shadow holdout cases are created after candidate freeze from withheld synonym banks, entity banks, and formatting variants, then used exactly once. Sealed release holdout cases are steward-owned, never shown to the model team before a release decision, and replaced after any failed promotion attempt. Human-review samples are a separate set of nuanced cases where automated scoring is either known to be weak or particularly safety-critical. That split structure is consistent with dynamic evaluation ideas from Dynabench, with contamination detection work that recommends active investigation, and with research showing that multi-turn interaction exposes weaknesses single-turn benchmarks miss.

The raw gate itself must be brutally simple. For the promotion path, each case is executed with a single declared system prompt, a single user prompt or turn bundle, one model invocation, frozen sampler parameters, and full capture of prompt bytes, output bytes, token IDs if available, stop reason, latency, tokens per second, and model-plus-adapter identity. No decomposition, no retries, no regex repair, no schema-filler, no function-calling shim, no JSON “fixer,” and no UI rewrite are allowed on this path. High-level orchestration—conversation compaction, retrieval, memory decks, adapters, or tool use—can be evaluated later as a separate product layer, but it may not substitute for raw-model proof. This is particularly important because TinyRustLM already supports adapters and auto-assembly, so the candidate identity must bind the full module plan, not just the base .slm checksum.

Safety and injection tests must assume partial rather than perfect resistance. OWASP now ranks prompt injection as the leading LLM application risk, and NIST’s taxonomy defines prompt injection as exploitation of concatenated higher-trust and lower-trust inputs. That means the gate should not merely ask whether the model “felt safe”; it should explicitly test whether lower-trust quoted or retrieved text can take over the response, whether the model over-refuses nearby benign tasks, and whether safe alternatives remain useful. Human review is mandatory for borderline, expert, or high-consequence outputs because LLM judges still show meaningful disagreement with domain experts on expert-knowledge tasks.

Machine-readable schema and case catalog

The schema below borrows the sample-level discipline seen in modern eval tools, but adds the fields that a TinyRustLM/MiniModel release actually needs: raw-output capture, receipt linkage, artifact identity, adapter stack identity, and browser/device telemetry. Critically, the raw_gate_contract forbids retries, decomposition, normalization, grammar repair, and tool substitution on the gate path.

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "title": "TinyRustLMRawEvalV1",
  "type": "object",
  "required": [
    "suite_id",
    "created_utc",
    "candidate",
    "environment",
    "raw_gate_contract",
    "threshold_profiles",
    "cases",
    "runs",
    "aggregate"
  ],
  "properties": {
    "suite_id": { "type": "string" },
    "created_utc": { "type": "string", "format": "date-time" },
    "candidate": {
      "type": "object",
      "required": [
        "candidate_id",
        "base_model_sha256",
        "tokenizer_sha256",
        "quantization",
        "artifact_bytes",
        "runtime_version",
        "sampler",
        "adapter_stack"
      ],
      "properties": {
        "candidate_id": { "type": "string" },
        "base_model_sha256": { "type": "string" },
        "tokenizer_sha256": { "type": "string" },
        "layout_sha256": { "type": "string" },
        "quantization": { "enum": ["f32", "q8_0", "q4_0", "other"] },
        "artifact_bytes": { "type": "integer", "minimum": 1 },
        "runtime_version": { "type": "string" },
        "sampler": {
          "type": "object",
          "required": ["temperature", "top_k", "top_p", "seed", "max_output_tokens"],
          "properties": {
            "temperature": { "type": "number" },
            "top_k": { "type": "integer" },
            "top_p": { "type": "number" },
            "seed": { "type": "integer" },
            "max_output_tokens": { "type": "integer" }
          }
        },
        "adapter_stack": {
          "type": "array",
          "items": {
            "type": "object",
            "required": ["adapter_id", "family", "sha256", "bytes"],
            "properties": {
              "adapter_id": { "type": "string" },
              "family": { "enum": ["ADP1", "ASP1", "ALR1", "other"] },
              "sha256": { "type": "string" },
              "bytes": { "type": "integer" }
            }
          }
        },
        "delivery_receipt_id": { "type": "string" }
      }
    },
    "environment": {
      "type": "object",
      "required": [
        "browser_name",
        "browser_version",
        "os_name",
        "os_version",
        "device_class",
        "device_tier",
        "wasm_simd",
        "webgpu",
        "device_memory_gb_bucket"
      ],
      "properties": {
        "browser_name": { "type": "string" },
        "browser_version": { "type": "string" },
        "os_name": { "type": "string" },
        "os_version": { "type": "string" },
        "device_class": { "enum": ["desktop", "laptop", "tablet", "phone"] },
        "device_tier": { "enum": ["L", "M", "H"] },
        "wasm_simd": { "type": "boolean" },
        "webgpu": { "type": "boolean" },
        "device_memory_gb_bucket": { "type": "number" },
        "storage_persisted": { "type": "boolean" }
      }
    },
    "raw_gate_contract": {
      "type": "object",
      "required": [
        "single_model_call_only",
        "allow_retry",
        "allow_decomposition",
        "allow_constrained_decoding_substitute",
        "allow_post_repair",
        "allow_ui_transformation",
        "tool_calls_allowed",
        "network_policy"
      ],
      "properties": {
        "single_model_call_only": { "const": true },
        "allow_retry": { "const": false },
        "allow_decomposition": { "const": false },
        "allow_constrained_decoding_substitute": { "const": false },
        "allow_post_repair": { "const": false },
        "allow_ui_transformation": { "const": false },
        "tool_calls_allowed": { "const": false },
        "network_policy": {
          "type": "object",
          "required": ["prompt_egress_allowed", "allowed_fetch_kinds"],
          "properties": {
            "prompt_egress_allowed": { "const": false },
            "allowed_fetch_kinds": {
              "type": "array",
              "items": { "enum": ["none", "manifest", "model_piece", "adapter_piece", "shell_asset"] }
            }
          }
        }
      }
    },
    "threshold_profiles": {
      "type": "array",
      "items": {
        "type": "object",
        "required": ["profile_id", "metric_weights", "hard_rules", "min_composite_score"],
        "properties": {
          "profile_id": { "type": "string" },
          "metric_weights": {
            "type": "object",
            "properties": {
              "instruction_adherence": { "type": "number" },
              "factual_consistency": { "type": "number" },
              "source_grounding": { "type": "number" },
              "semantic_completeness": { "type": "number" },
              "format_validity": { "type": "number" },
              "usefulness": { "type": "number" },
              "safety": { "type": "number" }
            }
          },
          "hard_rules": { "type": "array", "items": { "type": "string" } },
          "min_composite_score": { "type": "number" }
        }
      }
    },
    "cases": {
      "type": "array",
      "items": {
        "type": "object",
        "required": [
          "case_id",
          "split",
          "category",
          "template_family_id",
          "semantic_cluster_id",
          "conversation",
          "expected_contract",
          "threshold_profile_id"
        ],
        "properties": {
          "case_id": { "type": "string" },
          "split": { "enum": ["public_dev", "shadow_holdout", "sealed_release", "human_review"] },
          "category": { "type": "string" },
          "template_family_id": { "type": "string" },
          "semantic_cluster_id": { "type": "string" },
          "conversation": {
            "type": "array",
            "items": {
              "type": "object",
              "required": ["role", "content"],
              "properties": {
                "role": { "enum": ["system", "user", "assistant_context"] },
                "content": { "type": "string" }
              }
            }
          },
          "context_bundle_id": { "type": "string" },
          "expected_contract": {
            "type": "object",
            "properties": {
              "must_ask_clarifying_question": { "type": "boolean" },
              "must_refuse": { "type": "boolean" },
              "must_comply": { "type": "boolean" },
              "required_slots": { "type": "array", "items": { "type": "string" } },
              "json_schema_id": { "type": "string" },
              "max_sentences": { "type": "integer" },
              "exact_keywords": { "type": "array", "items": { "type": "string" } }
            }
          },
          "scoring_mode": { "enum": ["programmatic", "hybrid", "human_required"] }
        }
      }
    },
    "runs": {
      "type": "array",
      "items": {
        "type": "object",
        "required": [
          "case_id",
          "raw_output_utf8",
          "raw_output_sha256",
          "stop_reason",
          "latency_ms",
          "prompt_tokens",
          "completion_tokens",
          "tokens_per_second",
          "peak_memory_bytes",
          "network_receipt"
        ],
        "properties": {
          "case_id": { "type": "string" },
          "raw_output_utf8": { "type": "string" },
          "raw_output_sha256": { "type": "string" },
          "stop_reason": { "enum": ["eos", "max_tokens", "error", "abort"] },
          "latency_ms": { "type": "integer" },
          "prompt_tokens": { "type": "integer" },
          "completion_tokens": { "type": "integer" },
          "tokens_per_second": { "type": "number" },
          "peak_memory_bytes": { "type": "integer" },
          "network_receipt": {
            "type": "object",
            "required": ["outbound_request_count", "forbidden_request_count", "prompt_body_egress_detected"],
            "properties": {
              "outbound_request_count": { "type": "integer" },
              "forbidden_request_count": { "type": "integer" },
              "prompt_body_egress_detected": { "type": "boolean" }
            }
          },
          "metric_values": {
            "type": "object",
            "properties": {
              "instruction_adherence": { "type": "number" },
              "factual_consistency": { "type": "number" },
              "source_grounding": { "type": "number" },
              "semantic_completeness": { "type": "number" },
              "format_validity": { "type": "number" },
              "repetition": { "type": "number" },
              "truncation": { "type": "number" },
              "usefulness": { "type": "number" },
              "safety": { "type": "number" }
            }
          },
          "composite_score": { "type": "number" },
          "pass": { "type": "boolean" }
        }
      }
    },
    "aggregate": {
      "type": "object",
      "required": [
        "shadow_score",
        "sealed_score",
        "human_review_score",
        "load_success_rate",
        "promotion_decision"
      ],
      "properties": {
        "shadow_score": { "type": "number" },
        "sealed_score": { "type": "number" },
        "human_review_score": { "type": "number" },
        "load_success_rate": { "type": "number" },
        "promotion_decision": { "enum": ["promote", "hold", "reject"] }
      }
    }
  }
}

The catalog below defines 128 cases: four public-development cases, two generated shadow-holdout cases, one sealed release case, and one human-review case for each of the 16 required categories. The split discipline is intentional: no template family or semantic cluster may cross a split boundary. The shadow and sealed cases should be re-lexicalized each release cycle to keep them contamination-resistant.

Metrics thresholds and review rules

The metric system should combine deterministic validators, lightweight model-assisted judges, and human review, in that order. IFEval and IFBench support using directly verifiable rules whenever possible; summmary-faithfulness research shows that semantic similarity alone is not enough for factual consistency; and work on LLM-as-a-judge shows why expert tasks still need humans in the loop. For TinyRustLM, that implies a preference ordering of programmatic validation first, retrieval/QA-style factuality checks second, LLM judges only as assistants, and human review whenever domain specificity, safety, or ambiguity rises above what automation can reliably score.

The exact metric definitions should be these:

The threshold profiles below give exact per-case rules. Every case inherits the profile shown in the catalog above, so case-level thresholds are exact even when defined by profile. TRUNC=1 means “no truncation failure,” and SAFE=1 means “correct safe behavior with no harmful fulfillment.” Composite score is the weighted mean of applicable normalized metrics after hard rules are checked.

Promotion should use hard gates first, aggregates second. Public development is non-promotional. It is only a readiness check for whether the candidate is worth spending shadow and sealed budget on.

The anti-cheating checks should be explicit and non-waivable:

Human review should follow these exact rules:

Production delivery and promotion gates

The delivery design should preserve the same honesty as the evaluation design. TinyRustLM already presents a receipt-centered, local-only UI, and MiniModel already defines checksum-bound manifests, chunk hashes, chunk-list hashes, Merkle roots, peer-offer proofs, browser handoff, and metadata-only ledgers. That stack is a strong base for progressive, resumable, verifiable local delivery, but the delivery UX must still clearly distinguish a tiny runtime smoke artifact from a useful chat model.

The smoke artifact should be the only automatic asset. Publicly, TinyRustLM already has explicit smoke artifacts and even a 17,160,000-byte deterministic q8 artifact documented as runtime smoke rather than assistant quality. Use that pattern directly: load the smoke artifact or smoke microbench automatically if desired, but label it Runtime Test Only — Not a Chat Model in the UI and receipts. A smoke artifact is allowed to verify parsing, ABI stability, piece reassembly, tokenizer parity, and load success; it is not allowed to look like a hidden fallback assistant.

The useful default model should load only after informed consent. Chrome’s Prompt API is a good UX reference here: it exposes availability, requires user activation to trigger download, and surfaces download progress. Borrow that UX pattern, not the trust model. Before any TinyRustLM utility model download begins, the user should be shown the exact artifact ID, signed checksum, quantization, byte size, estimated peak memory, expected cold-start time on the current device tier, whether persistence will be requested, and how to remove the model later. Consent should be granular: “Download once,” “Download and keep offline,” or “Not now.”

Progressive download and resume should use piecewise transfer rather than opaque whole-file fetches. MiniModel’s manifest and peer-piece concepts already support chunk hashes and Merkle roots. On the web side, resumability should use HTTP range requests where available, piece verification should use SHA-256 via Web Crypto, and pieces should be stored in IndexedDB with a service worker controlling shell caching and offline restart. If persistent storage is available and granted, store model pieces and receipts there; if not, explain that storage remains best-effort and may be evicted by the browser.

The integrity receipt should be first-class and user-visible. TinyRustLM’s public UI already anticipates receipts with origin, bytes, checksums, actual fetches, route hashes, phases, and lineage/gates. The production receipt should record at minimum: shell version, candidate ID, base model checksum, adapter stack checksums, manifest checksum, chunk-list checksum, Merkle root, source lane, peer URL or external source URL, byte count, first-seen timestamp, final-verified timestamp, browser/device fingerprint bucket, persistence state, and a statement that prompt bodies were never transmitted. That same receipt should power rollback and audit.

The “no hidden server inference” rule should be enforced by both policy and telemetry. TinyRustLM already states local browser inference and no external analytics in its runtime docs. Production should add a signed network witness to each session receipt listing only manifest, shell-asset, model-piece, and adapter-piece fetches, with prompt-body egress fixed at false. If the product ever supports optional remote features in another mode, they must live behind a separate mode switch with separate receipts and separate evaluation; they may not silently cohabit the local-only promise.

The footprint and latency promotion gates should be concrete:

Device capability tiers should be assigned conservatively from local evidence, not marketing claims. Use a deterministic smoke benchmark, approximate device memory, storage estimate, and a short local generation benchmark to place the device into Tier L, M, or H. Tier L should be eligible for smoke and Feather only; Tier M for Feather and Balanced; Tier H for all three. If the benchmark contradicts the nominal device tier—for example, memory pressure or repeated load failures—downgrade automatically and record the downgrade in the receipt ledger.

Rollout tiers and the definition of good quality

Because the current audited public runtime still advertises a 33.5 MB model budget and only smoke artifacts are publicly verified, the honest immediate ship state is: smoke only. The three utility tiers below are the targets for the first branch that proves larger verified artifacts, keeps the local-only and receipt guarantees, and passes the raw gates. If a branch remains within the currently published public envelope, it should not be marketed as a useful default chat assistant.

The three production model tiers should be these:

Rollout and rollback should be slot-based and receipt-driven:

The requirement-by-requirement definition of good quality with reasonable local footprint should be the following:

The practical conclusion is straightforward. Do not currently ship a publicly claimed TinyRustLM default assistant model on the basis of the audited public evidence. Ship the smoke artifact, harden the receipt-and-piece pipeline, and then use the raw-output promotion system above to admit the first Feather local model only when it demonstrably clears shadow, sealed, human-review, footprint, latency, and browser-reliability gates. That is the narrow path that stays honest about both quality and local footprint.