Exact Speculative and Assisted Decoding for TinyRustLM

Full report

On this page

Executive recommendation and evidence boundaries

Recommendation: conditional go for a narrowly scoped experiment; no product adoption yet. TinyRustLM should first test exact, linear speculative decoding with one genuinely compatible tiny draft and the qualified champion, plus a model-free prompt-lookup baseline. Shipping should occur only if the implementation preserves the champion’s declared distribution and wins preregistered end-to-end latency, memory, energy, and reliability gates. Published speedups are evidence that the technique can work, not estimates for TinyRustLM. The uploaded specification’s distinctions among artifact bytes, logical composition bytes, incremental bytes, physical deduplicated bytes, and peak resident bytes should be treated as mandatory accounting categories.

The original speculative-sampling and speculative-decoding papers prove that a cheaper proposal model can accelerate generation without changing the target model’s output distribution when proposals are accepted probabilistically and rejections use the correct residual distribution. Their measured speedups arose on particular models, devices, kernels, and workloads and therefore cannot be transferred to TinyRustLM without local measurement.

Evidence labels used below

Immediate decision

Minimum adoption conditions

TinyRustLM should adopt exact speculation only when all of the following are simultaneously true:

1. The target can score a proposed sequence in one prefill-like operation and return the target distributions for every proposed position plus the position after the proposal.
2. The implementation has a transactional target and draft cache abstraction with logical rollback.
3. The draft and champion compatibility manifest passes strict identity and tokenizer checks.
4. Greedy output is token-identical to the champion baseline under the same declared decoding identity.
5. Stochastic tests support distributional equivalence to the champion after accounting for the implementation’s declared finite-precision arithmetic.
6. End-to-end gains include both model loads, both prefills, both KV caches, synchronization, rejection work, and memory pressure.
7. The implementation can fail before output or fail visibly after output without silently restarting and duplicating answer text.
8. An unsuccessful implementation can be deleted cleanly by removing the speculative orchestrator and optional target-verification API while leaving the champion baseline unchanged.

Likely outcome by topology. As an engineering inference, the strongest candidate is a native or native-companion configuration in which the champion remains resident and a truly tiny draft is resident on otherwise underused CPU capacity. A browser-only configuration may still win when the champion’s q_len > 1 kernels efficiently amortize weight access, but it is more likely to be constrained by duplicate weights, duplicate KV, WebGPU synchronization, and transient loading memory. Prompt lookup may therefore be the best browser-local acceleration even when a neural draft wins natively. Public GPU inference frameworks emphasize that speculative decoding is most beneficial at low batch sizes and depends on efficient collective verification.

No assertion in this report implies access to TinyRustLM source code, .slm files, models, routing logic, test cases, private prompts, outputs, or machines.

Exactness taxonomy and compatibility contracts

The central classification rule is not “small versus large.” It is whether the proposal mechanism and verifier preserve the champion’s declared token distribution after the complete decoding transformation: temperature, penalties, token suppression, top-\(k\), top-\(p\), grammar masks, stop rules, and any other versioned processor. A proposal source may be weak, deterministic, neural, retrieval-based, or architecturally embedded; exactness comes from verification and correction, not from the proposal’s name.

The exact independent-draft algorithm is established by the original speculative-decoding and speculative-sampling work. Hugging Face currently documents assisted decoding, prompt lookup, self-speculative early exit, multi-token prediction, cross-tokenizer assistance, and an explicitly approximate static-ensemble mode. Medusa, LayerSkip, EAGLE, lookahead decoding, and MTP provide alternative proposal architectures, but each carries different training and verifier requirements.

Tokenizer compatibility contract

Hugging Face’s Universal Assisted Generation translates context between tokenizers by decoding, re-encoding, identifying a common suffix, and realigning caches. Its later token-level-intersection work addresses heterogeneous-vocabulary sampling. These are material advances, but they should not be treated as evidence that an arbitrary pair of tokenizers is exact under TinyRustLM’s champion distribution. In particular, an ID-level residual distribution \([q-p]_+\) is undefined until both distributions have been expressed on a common sample space.

Engineering recommendation: the initial compatibility manifest should require byte-identical tokenizer artifacts and explicit equality of:

• vocabulary size, token strings, token IDs, added-token order, merges or unigram scores, normalization, pre-tokenization, byte fallback, unknown-token behavior, and special-token map;
• chat-template bytes and version;
• rendered system, developer, and user prompt token IDs;
• BOS, EOS, padding, stop-token, and stop-string policies;
• context limit, sliding-window behavior, truncation side, RoPE parameters and scaling, position-ID generation, and cache origin;
• model weights, quantization metadata, adapters in ordered composition, logits processors, grammar identity, and sampling policy.

A small output-vocabulary draft can still be exact if it consumes the champion’s tokenization and defines \(p(x)=0\) for omitted output tokens; the residual sampler then retains the champion’s missing mass. Remapping IDs or changing input segmentation is the dangerous part.

Model and prompt identity. The target and draft need not share weights, architecture, context length, or probability quality, but they must be evaluated on semantically and positionally aligned prefixes. A shorter draft context can be used only under a versioned draft truncation rule whose proposal probabilities are computed from that truncated prefix while target verification uses the complete champion prefix. Exactness can survive a poor proposal distribution because the draft is merely a proposal, but acceptance may collapse. By contrast, silently applying different chat templates, system prompts, adapter orders, BOS rules, or position IDs can cause both low acceptance and invalid cache alignment.

Different RoPE scaling policies do not mathematically invalidate a proposal model by themselves: the draft may use any proposal distribution. They do invalidate any attempt to share KV, hidden states, positions, or cached prompt transformations between models. Shared prompt prefill must therefore mean “run the same declared prompt independently through each model,” not “copy one model’s KV into the other,” unless a separately proved cache transform exists.

Current implementation landscape. As of retrieval on August 1, 2026:

• Hugging Face Transformers documents assisted decoding for greedy and sampled generation, prompt lookup, self-speculative early exit, MTP, and cross-tokenizer assistance; documented restrictions and behavior vary by release.
• TensorRT-LLM 1.3.0 release-candidate documentation covers draft/target, n-gram, EAGLE 3, MTP, suffix-automaton enhancement, parallel drafts, and user-provided drafters. Its relaxed MTP mode is explicitly not strict token-match acceptance.
• vLLM’s current documentation covers multiple speculative techniques and warns that mathematical losslessness does not guarantee bitwise log-probability stability across batching and kernels.
• llama.cpp publicly exposes local draft/target speculative decoding in its server tooling, making it a relevant native reference, but TinyRustLM should pin and audit a specific commit rather than treating changing master documentation as normative.
• WebLLM and Transformers.js publicly provide browser-local inference through WebGPU, WASM, and ONNX-related stacks. In the reviewed public documentation, no implementation-grade exact speculative-sampling API was identified; that negative search result is not proof of absence.

Mathematical reconstruction and normative algorithms

Let \(h\) be the current authoritative target-token prefix. Define:

• \(q_i(x)\): the champion’s complete processed next-token distribution at speculative position \(i\), conditioned on \(h,y_1,\ldots,y_{i-1}\);
• \(p_i(x)\): the draft’s actual proposal distribution at that same logical position;
• \(y_i\sim p_i\): the proposed token;
• \(k\): maximum proposal length for this round.

“Processed” means after every declared temperature, mask, grammar, repetition penalty, top-\(k\), top-\(p\), suppression, and other policy transformation. The champion distribution \(q_i\), not the draft, is authoritative.

Exact stochastic step

For each proposal \(y_i\), accept it with

\[ a_i(y_i)=\min\left(1,\frac{q_i(y_i)}{p_i(y_i)}\right). \]

If the first rejection occurs at position \(j\), sample the replacement token from

\[ r_j(x)= \frac{[q_j(x)-p_j(x)]_+} {\sum_v [q_j(v)-p_j(v)]_+}, \qquad [z]_+=\max(z,0). \]

Stop the speculative round after that replacement. If every proposal is accepted, sample one additional token from \(q_{k+1}\). This is the algorithm reconstructed in the original papers.

Correctness argument

For one position, let

\[ A=\sum_x\min(p(x),q(x)). \]

The accepted path contributes probability \(\min(p(x),q(x))\) to output token \(x\). Rejection occurs with probability

\[ 1-A=\sum_x[q(x)-p(x)]_+. \]

The corrected path contributes

\[ (1-A)\frac{[q(x)-p(x)]_+}{1-A}=[q(x)-p(x)]_+. \]

Therefore,

\[ \Pr[\text{output}=x] =\min(p(x),q(x))+[q(x)-p(x)]_+ =q(x). \]

Conditioning this identity on every accepted prefix and applying induction gives the champion’s autoregressive joint distribution. The expected single-position acceptance is

\[ A=1-\operatorname{TV}(p,q), \]

so acceptance measures distributional overlap, not simply model size.

Normative greedy pseudocode

function greedy_assisted_round(state, requested_k):
    require state.identity_manifest_is_valid
    require state.target_cache.is_committed
    require state.draft_cache.is_committed

    remaining = state.max_new_tokens - state.emitted_token_count
    if remaining == 0:
        return FINISHED_MAX_LENGTH

    k = min(requested_k, remaining)

    if k == 0:
        token = target_baseline_argmax_one(state)
        commit_and_emit(token)
        return token

    tx_draft = draft_cache.begin_transaction()
    proposals = []

    for i in 0 .. k-1:
        draft_logits = draft.forward_pending_and_get_logits(tx_draft)
        draft_dist = apply_declared_draft_processors(draft_logits, state, i)

        if draft_dist.has_nan_or_no_valid_token:
            tx_draft.rollback()
            token = target_baseline_argmax_one(state)
            commit_and_emit(token)
            return token

        y = argmax_with_declared_tie_break(draft_dist)
        proposals.push(y)
        tx_draft.append_provisionally(y)

        if y is an active stop token:
            break

    tx_target = target_cache.begin_transaction()
    result = target.verify_linear(tx_target, proposals)
    # result.logits[i] predicts proposals[i]
    # result.logits[len(proposals)] predicts the extra token

    accepted = 0

    for i in 0 .. len(proposals)-1:
        q = apply_champion_processors(result.logits[i], state, i)

        if q.has_nan_or_no_valid_token:
            abort_round_without_emitting(tx_target, tx_draft)
            return FATAL_TARGET_NUMERIC_OR_MASK_ERROR

        target_token = argmax_with_declared_tie_break(q)

        if proposals[i] != target_token:
            tx_target.commit_proposal_prefix(accepted)
            tx_target.rollback_proposal_tail()
            tx_draft.rollback_to_authoritative_length(accepted)
            state.set_pending_authoritative_token(target_token)
            emit(proposals[0 .. accepted])
            emit(target_token)
            return accepted + 1 tokens

        accepted += 1

        if proposals[i] is an active stop token:
            tx_target.commit_proposal_prefix(accepted)
            tx_target.rollback_proposal_tail()
            tx_draft.commit_prefix(accepted)
            emit(proposals[0 .. accepted])
            mark_terminal_state()
            return accepted tokens

        if accepted == remaining:
            tx_target.commit_proposal_prefix(accepted)
            tx_target.rollback_proposal_tail()
            tx_draft.commit_prefix(accepted)
            emit(proposals)
            mark_finished_max_length()
            return accepted tokens

    q_extra = apply_champion_processors(
        result.logits[len(proposals)],
        state after proposals,
        len(proposals)
    )

    if q_extra.has_nan_or_no_valid_token:
        abort_round_without_emitting(tx_target, tx_draft)
        return FATAL_TARGET_NUMERIC_OR_MASK_ERROR

    z = argmax_with_declared_tie_break(q_extra)
    tx_target.commit_all_proposals()
    tx_draft.commit_all_proposals()
    state.set_pending_authoritative_token(z)
    emit(proposals)
    emit(z)
    return len(proposals) + 1 tokens

Normative stochastic pseudocode

function exact_speculative_round(state, requested_k, rng_contract):
    require state.identity_manifest_is_valid
    require state.target_cache.is_committed
    require state.draft_cache.is_committed

    remaining = state.max_new_tokens - state.emitted_token_count
    if remaining == 0:
        return FINISHED_MAX_LENGTH

    k = min(requested_k, remaining)

    if k == 0:
        z = target_baseline_sample_one(
            state,
            rng_contract.target_baseline(state.round_id)
        )
        commit_and_emit(z)
        return z

    tx_draft = draft_cache.begin_transaction()
    proposals = []
    proposal_log_probs = []

    for i in 0 .. k-1:
        draft_logits = draft.forward_pending_and_get_logits(tx_draft)
        p = apply_declared_draft_processors(draft_logits, state, i)

        if p.has_nan_or_no_valid_token:
            tx_draft.rollback()
            z = target_baseline_sample_one(
                state,
                rng_contract.target_baseline(state.round_id)
            )
            commit_and_emit(z)
            return z

        y = sample(
            p,
            rng_contract.draft_sample(state.round_id, i)
        )

        require log(p[y]) is finite
        proposals.push(y)
        proposal_log_probs.push(retain_or_checkpoint(p))
        tx_draft.append_provisionally(y)

        if y is an active stop token:
            break

    tx_target = target_cache.begin_transaction()
    result = target.verify_linear(tx_target, proposals)

    accepted = 0

    for i in 0 .. len(proposals)-1:
        q = apply_champion_processors(result.logits[i], state, i)

        if q.has_nan_or_no_valid_token:
            abort_round_without_emitting(tx_target, tx_draft)
            return FATAL_TARGET_NUMERIC_OR_MASK_ERROR

        p = recover_exact_proposal_distribution(proposal_log_probs[i])

        log_ratio = log(q[proposals[i]]) - log(p[proposals[i]])
        accept_probability = min(1, exp_clamped(log_ratio))

        u = uniform_open_closed(
            rng_contract.accept_uniform(state.round_id, i)
        )

        if u <= accept_probability:
            accepted += 1

            if proposals[i] is an active stop token:
                tx_target.commit_proposal_prefix(accepted)
                tx_target.rollback_proposal_tail()
                tx_draft.commit_prefix(accepted)
                emit(proposals[0 .. accepted])
                mark_terminal_state()
                return accepted tokens

            if accepted == remaining:
                tx_target.commit_proposal_prefix(accepted)
                tx_target.rollback_proposal_tail()
                tx_draft.commit_prefix(accepted)
                emit(proposals[0 .. accepted])
                mark_finished_max_length()
                return accepted tokens

            continue

        residual[x] = max(q[x] - p[x], 0) for every target token x
        residual_mass = stable_sum(residual)

        if residual_mass is not finite or residual_mass <= numeric_floor:
            abort_round_without_emitting(tx_target, tx_draft)
            return FATAL_RESIDUAL_INVARIANT_ERROR

        residual = residual / residual_mass
        z = sample(
            residual,
            rng_contract.correction_sample(state.round_id, i)
        )

        tx_target.commit_proposal_prefix(accepted)
        tx_target.rollback_proposal_tail()
        tx_draft.rollback_to_authoritative_length(accepted)
        state.set_pending_authoritative_token(z)

        emit(proposals[0 .. accepted])
        emit(z)
        return accepted + 1 tokens

    q_extra = apply_champion_processors(
        result.logits[len(proposals)],
        state after proposals,
        len(proposals)
    )

    if q_extra.has_nan_or_no_valid_token:
        abort_round_without_emitting(tx_target, tx_draft)
        return FATAL_TARGET_NUMERIC_OR_MASK_ERROR

    z = sample(
        q_extra,
        rng_contract.target_extra(state.round_id)
    )

    tx_target.commit_all_proposals()
    tx_draft.commit_all_proposals()
    state.set_pending_authoritative_token(z)

    emit(proposals)
    emit(z)
    return len(proposals) + 1 tokens

These algorithms follow the acceptance and correction construction in the foundational work, but add explicit cache, identity, stop, cancellation, and failure semantics needed for an implementation.

Normative edge-case behavior

RNG contract. A counter-based interface should derive independent draws for draft_sample, accept_uniform, correction_sample, target_extra, and target_baseline. This prevents changes in proposal length from shifting unrelated random draws and makes fault reproduction practical. The exact output distribution does not require the same draft and target random streams, but replay does require a stable, versioned mapping.

Finite-precision obligation. Mathematical exactness does not imply bitwise equality between a target’s q_len=1 and q_len=k kernels. Different matrix shapes, reductions, quantized deblocking, and device scheduling may perturb logits. A greedy argmax can flip near a tie; a stochastic probability vector can differ even when all token rankings remain unchanged. TinyRustLM must therefore define whether the champion’s declared distribution is the baseline single-token kernel or a broader numerically tolerated implementation family. The strongest policy is to treat the champion-alone kernel as authoritative and require batched verification to pass logit and output conformance tests against it. vLLM’s documentation similarly distinguishes algorithmic losslessness from floating-point and batching stability.

Verification, cache, residency, and browser architecture

Minimum target API

verify_linear(
    target_identity,
    committed_cache_handle,
    pending_authoritative_token?,
    proposed_target_ids[0..m],
    position_state,
    decoding_state_snapshots
) -> {
    logits_for_proposals[0..m-1],
    logits_after_all_proposals,
    provisional_cache_handle,
    numerical_metadata,
    cancellation_epoch
}

The API’s positional contract must be explicit:

• logits_for_proposals[0] predicts proposed_target_ids[0] from the committed prefix.
• logits_for_proposals[i] predicts proposal \(i\) after proposals \(0\ldots i-1\).
• logits_after_all_proposals predicts the extra token after every proposed token.
• The target forward pass must use ordinary causal masking, not permit any proposal token to attend to a future proposal.
• The result is provisional until the orchestrator commits an accepted prefix.
• A request identity, sequence number, and cancellation epoch must accompany every worker or native-companion message.

This k+1-distribution interface is the key performance prerequisite identified by the original algorithms: the target scores the proposal block in parallel, turning repeated decode operations into a prefill-like verification pass.

Cache transaction model

A cache transaction should expose:

checkpoint = cache.begin(committed_length)
cache.append_provisional(tokens)
cache.commit_prefix(token_count)
cache.rollback_to(logical_length)
cache.finalize()

The logical state should distinguish:

1. Committed KV length: tokens whose K/V tensors are materialized and authoritative.
2. Pending authoritative token: a sampled or selected token already emitted but not yet consumed by the next model forward.
3. Provisional verification tail: target K/V for proposed tokens that may be accepted or rolled back.
4. Terminal marker: EOS, stop, cancellation, or max-length state.
5. Content identity: model, tokenizer, template, adapters, positional policy, prompt, cache layout, and quantization identity.

Suppose proposals \(y_1,\ldots,y_k\) are verified and rejection occurs at \(j\). The target may retain K/V only for \(y_1,\ldots,y_{j-1}\). The replacement token is usually “pending”: it has been sampled from the target-derived residual but has not yet passed through the target transformer. The next target call first consumes that token. The draft must similarly roll back rejected proposals and ingest the authoritative replacement before proposing again.

With paged KV, rollback can be logical: pages containing rejected tokens may remain allocated for reuse, but their token ranges must become unreachable. Page references must not be returned to a pool until all in-flight device work and message epochs that could address them have completed. TensorRT-LLM’s public implementation documentation illustrates request-level rewind and paged-cache handling as a concrete reference, though TinyRustLM must independently validate its own layout.

Separate-cache rule. A target and independent draft require separate KV caches. Even if both consume the same token IDs, their layers, hidden widths, head counts, head dimensions, quantization, RoPE, and cache layouts generally differ. Prompt computation may be scheduled together, but cache tensors are not interchangeable.

For a model with \(L\) layers, \(H_{\mathrm{KV}}\) KV heads, head dimension \(D\), sequence length \(S\), and \(b\) bytes per KV element, an approximate batch-one KV payload is

\[ M_{\mathrm{KV}} = 2SLH_{\mathrm{KV}}Db, \]

before page tables, alignment, allocator fragmentation, metadata, and provisional speculative tails. TinyRustLM’s peak memory must include \(M_{\mathrm{KV,target}}+M_{\mathrm{KV,draft}}\).

Logit and residual scratch. Returning all full-vocabulary distributions requires approximately

\[ M_{\mathrm{target\ logits}}=(k+1)Vb_\ell. \]

Retaining all draft distributions costs roughly \(kVb_\ell\). Exact rejection correction needs the full \(p_j\) and \(q_j\) at the rejected position. TinyRustLM can trade memory for recomputation by storing only draft checkpoints and recomputing \(p_j\) after rejection, but that recomputation belongs in the latency and energy model. Sparse top-\(k\) representations are exact only when omitted proposal probabilities are known to be zero and the residual calculation still covers all target-supported tokens.

Suggested architecture visualization — place here

sequenceDiagram
    participant UI as Browser UI
    participant O as Speculation Orchestrator
    participant D as Draft Worker
    participant T as Champion Worker or Native Companion

    UI->>O: Generate(request identity, seed, policy)
    O->>D: Synchronize authoritative prefix and pending token
    D-->>O: Proposals y[1..k], proposal probabilities, draft checkpoint
    O->>T: Verify prefix + y[1..k], transaction epoch
    T-->>O: q[1..k+1], provisional target KV

    loop In proposal order
        O->>O: Accept with min(1, q(y)/p(y))
    end

    alt First rejection at j
        O->>O: Sample residual [q_j - p_j]+
        O->>T: Commit y[1..j-1], rollback tail
        O->>D: Roll back tail, queue correction token
        O-->>UI: Emit accepted prefix + correction
    else All accepted
        O->>O: Sample target extra token
        O->>T: Commit all proposals
        O->>D: Commit proposals, queue extra token
        O-->>UI: Emit proposals + extra token
    else Cancellation or stale epoch
        O->>T: Roll back provisional state
        O->>D: Roll back provisional state
        O-->>UI: Cancelled or visible error
    end

Kernel implications

• Quantized target kernels: verification changes the target’s query length from one to a small \(k\). This may convert weight-dominated matrix-vector work into a better-amortized small matrix multiplication, but only if the q4 kernel is optimized for these shapes.
• WASM SIMD: standardized 128-bit SIMD can accelerate packed low-bit unpacking and dot products, but it does not eliminate the extra draft weight stream or cache traffic.
• WebGPU: use fixed, preallocated maximum-\(k\) buffers and query actual adapter limits at runtime. Avoid returning \((k+1)V\) logits to JavaScript when acceptance and residual computation can occur on-device; CPU readback and synchronization can erase verification gains. WebGPU remained on the W3C Candidate Recommendation track in 2026, so implementation and limit variability must remain part of the compatibility matrix.
• Native CPU: the principal opportunity is amortizing a full champion-weight traversal across several token positions. The main risks are a draft that also saturates memory bandwidth and a target kernel whose \(q_len=k\) path performs worse than repeated decode.
• CPU draft plus GPU champion: proposal IDs and selected probabilities are small transfers, but the GPU cannot verify until drafting completes. This topology helps only when the CPU draft is fast and does not contend with GPU command preparation or unified memory.
• Tree methods: Medusa- or EAGLE-style branches require tree attention masks, node-to-parent position maps, branch-aware logit extraction, and more complex cache commit. Linear verification should precede them.

Residency topology comparison

Required byte reporting

Peak resident accounting must be timeline-based. Loading a draft while the champion and its old scratch arena remain resident may create a larger peak than steady-state decoding.

Cost model, scheduling, grammar, and draft selection

Let:

• \(L_t,L_d\): target and draft load costs;
• \(P_t(S),P_d(S)\): prompt-prefill costs for prompt length \(S\);
• \(d_i\): cost to generate proposal \(i\);
• \(V_t(k)\): target cost to verify \(k\) proposals and produce the extra distribution;
• \(C_{\mathrm{sync}}(k)\): worker/device/IPC synchronization and transfer;
• \(C_{\mathrm{rollback}}(j,k)\): logical and physical rollback after rejection at \(j\);
• \(C_{\mathrm{residual}}(j)\): probability recovery, positive-part subtraction, normalization, and sampling;
• \(a_i\): conditional acceptance probability at proposal position \(i\);
• \(S_i=\Pr(\text{first }i\text{ proposals accepted})\), with \(S_0=1\).

No independence assumption is needed for \(S_i\). Empirically,

\[ S_i=\prod_{j=1}^{i}a_j \]

when \(a_j\) is defined conditionally on survival to position \(j\).

Ignoring terminal truncation, every round emits one authoritative token, plus one token for every accepted proposal. Thus

\[ \mathbb{E}[G(k)] = 1+\sum_{i=1}^{k} S_i = \sum_{i=0}^{k}S_i. \]

Under a constant conditional acceptance \(\alpha\),

\[ \mathbb{E}[G(k)] = \frac{1-\alpha^{k+1}}{1-\alpha}. \]

This is the standard speculative-decoding throughput structure; TinyRustLM should estimate the position-dependent \(S_i\) rather than assume a constant rate.

An implementation-grade expected round cost is

\[ \begin{aligned} \mathbb{E}[C_{\mathrm{round}}(k)] =& \sum_{i=1}^{k} d_i +V_t(k) +C_{\mathrm{sync}}(k) \\ &+ \sum_{j=1}^{k} S_{j-1}(1-a_j) \left[ C_{\mathrm{rollback}}(j,k) +C_{\mathrm{residual}}(j) \right] \\ &+ S_kC_{\mathrm{full\ accept}} +C_{\mathrm{cache\ append}} +C_{\mathrm{telemetry}}. \end{aligned} \]

The steady-state per-token cost is

\[ \bar C_{\mathrm{spec}}(k) = \frac{\mathbb{E}[C_{\mathrm{round}}(k)]} {\mathbb{E}[G(k)]}. \]

If \(C_t(1)\) is the measured champion-alone cost per generated token, the necessary steady-state break-even condition is

\[ \bar C_{\mathrm{spec}}(k)<C_t(1). \]

For a session expected to generate \(N\) tokens, define fixed incremental cost

\[ F=L_d+P_d(S)+C_{\mathrm{draft\ initialization}} +C_{\mathrm{extra\ allocation}}. \]

A rough minimum generation length is then

\[ N_{\min}

\frac{F} {C_t(1)-\bar C_{\mathrm{spec}}(k)} \]

when the denominator is positive. If it is nonpositive, no amount of session length amortizes the draft under that configuration.

The full end-to-end comparison should measure

\[ T_{\mathrm{baseline}} = L_t+P_t(S)+T_{\mathrm{decode,target}}, \]

\[ T_{\mathrm{spec}} = L_t+L_d+P_t(S)+P_d(S) +T_{\mathrm{draft}} +T_{\mathrm{verify}} +T_{\mathrm{reject}} +T_{\mathrm{sync}} +T_{\mathrm{rollback}} +T_{\mathrm{finalize}}. \]

A reported “tokens per second” that begins after both models and prompts are warm is therefore only one of several views.

Adaptive speculation length

Hugging Face changed its default assisted-generation scheduling toward confidence-based dynamic lookahead in Transformers 4.45 and reported gains over static schedules on its tested model pairs and hardware. Those results support testing the idea but do not determine TinyRustLM’s policy.

The scheduler must be content-identity-bound and versioned. It may use local counters such as recent position-wise acceptance, draft entropy, remaining context, and measured device timings. It should not upload private prompts or outputs, retain hidden user-history features, or optimize against sealed answer quality. An online policy update that changes behavior during a preregistered evaluation should invalidate the run.

Grammar and deterministic opportunities

A grammar mask replaces the unmasked champion distribution \(q\) with

\[ q_G(x)= \frac{q(x)\mathbf 1[x\in A_G]} {\sum_v q(v)\mathbf 1[v\in A_G]}, \]

where \(A_G\) is the valid-token set in the current grammar state. Exact speculation is then exact relative to \(q_G\), provided the grammar state and masking order are identical in baseline and speculative execution.

Target probability verification can be skipped only when the declared target decoder has exactly one valid next target token. Even then, the forced token must eventually pass through the target transformer to create KV for subsequent prediction. TinyRustLM may batch a sequence of grammar-forced tokens as a prefill-like append and may omit unnecessary LM-head projection at intermediate forced positions if the target API supports it. It may not skip target state advancement.

Prompt copies, whitespace predictions, closing braces, indentation, and syntax continuations are not automatically forced. Unless the validated grammar’s target-token support is a singleton, they remain proposals and require target verification. Public guided-decoding systems support JSON Schema, regular expressions, and EBNF-like constraints, demonstrating the operational value of grammar masks, but TinyRustLM must qualify its own parser and token-mask generation.

Draft training and selection

LayerSkip specifically combines layer dropout and early-exit training to support self-speculative execution, while MTP and Medusa add future-token heads. EAGLE drafts at a feature level. These are not drop-in properties of an arbitrary champion checkpoint.

Contamination boundary. Draft training may use appropriately licensed public data, nonsealed synthetic prompts, and champion probability traces generated under an authorized process. It must not use sealed product evaluation answers, hidden human-preference labels from the final test, private user prompts without explicit consent, or test-set-derived routing rules. Provenance receipts should bind dataset versions, filters, champion identity, training code, and draft artifact hash.

Preregistered evaluation, metrics, and promotion gates

Comparison arms

Every paired run must bind the same champion artifact, tokenizer, template, rendered prompt IDs, adapters, decoding profile, grammar, stop policy, seed family, context limit, hardware, browser version, worker topology, and thermal/power condition. Speculative policy and draft identity are the intended independent variables.

Workload strata

The preregistered corpus should include casual chat, ambiguous requests, source conflict, current-preference questions, prompts requiring distrust of model memory, summaries, rewrites, structured JSON, extraction, code, repetitive text, multilingual text, long context, and adversarial low-acceptance inputs. Each stratum should report acceptance and quality separately; a high aggregate rate dominated by repetitive copying is insufficient evidence for conversational deployment.

Long-context tests should cross page boundaries, draft context limits, target context limits, sliding-window thresholds, and browser-memory pressure points. Multilingual tests should include combining marks, composed and decomposed Unicode, emoji sequences, right-to-left text, CJK segmentation, byte fallback, and scripts in which tokenizer normalization differs.

Metrics

Exactness tests

For deterministic champion decoding, the acceptance criterion is zero token mismatches against champion-alone output across all retained matched-identity cases. The comparison must include EOS and stop positions, not only visible decoded text.

For stochastic exactness:

1. Build synthetic vocabularies of two to six tokens with rational or exactly representable probabilities.
2. Enumerate all short target and draft distributions, including disjoint supports, equal distributions, tiny overlap, zero target probability, zero proposal probability, and masked tokens.
3. Compare exact sequence probabilities by exhaustive state enumeration.
4. Run Monte Carlo tests against target probabilities with preregistered confidence intervals and correction for multiple comparisons.
5. Test the distribution of first rejection positions and corrected tokens.
6. Test transformed distributions after temperature, top-\(k\), top-\(p\), penalties, and grammar masks.
7. Compare q_len=1 and q_len=k target logits and resulting distributions on the real kernels.
8. Maintain deterministic golden vectors for the reference speculative interpreter and its RNG domains.

Deterministic replay is useful for debugging, but ordinary champion-alone and speculative runs need not produce the same sampled answer because they use random draws differently. Distributional equivalence is the mathematical criterion.

Suggested promotion gates

These values are engineering recommendations to preregister before seeing final results, not claimed industry standards.

The performance gate should be applied separately to browser-only, browser/native, and native-only classes. It is acceptable for the scheduler to disable speculation on known low-acceptance strata, but that disabling policy must itself be frozen before the final evaluation.

Suggested result visualizations

Place these immediately after the preregistered results table:

• A survival curve showing \(S_i\) by proposal position and workload stratum.
• A latency decomposition waterfall with load, both prefills, draft, verification, correction, synchronization, and output handling.
• A break-even plot with session output length on the horizontal axis and total latency ratio on the vertical axis.
• A scatter plot of peak resident bytes versus median completion latency for every topology.
• An acceptance-versus-draft-entropy calibration plot.
• A heat map of method by workload stratum, with separate panels for performance and quality.
• An empirical cumulative distribution of per-request speedup so that regressions are not hidden by a mean.
• A Pareto frontier of latency, energy, and peak resident memory.

Fault tolerance, privacy, implementation backlog, and deletion rules

Fault-injection matrix

No-duplicate-output rule. Fallback to champion-alone execution may be silent only before the answer has emitted its first token. Once output has been streamed, an unrecoverable speculative failure must terminate that answer visibly. Automatically restarting from the prompt could repeat or contradict already displayed text and obscure the failure.

Privacy-safe receipt

A local receipt may contain:

receipt_version
request_nonce
champion_artifact_hash
draft_artifact_hash or "none"
tokenizer_hash
chat_template_hash
adapter_chain_hash
sampling_policy_hash
grammar_hash
position_policy_hash
scheduler_policy_hash
hardware_class
browser/runtime version
seed commitment or local replay seed
target-call count
draft-call count
proposal count
accepted-count histogram
rejection-position histogram
cold and warm timing aggregates
peak resident byte categories
bounded numeric errors
failure and fallback code

It should exclude prompts, outputs when private, model bytes, decoded token strings, local filesystem paths, credentials, peer identifiers, hidden user history, raw KV, and unrestricted device fingerprints. A receipt intended for sharing can hash or bucket hardware information more coarsely than a local diagnostic receipt.

Implementation components

Zero-third-party-crate Rust assessment. A same-tokenizer, linear greedy implementation is feasible in a minimal Rust core. Exact stochastic sampling is also feasible but requires careful implementations of stable log-softmax, categorical sampling, residual arithmetic, and RNG. The following are poor initial fits for a zero-third-party-crate core:

• cross-tokenizer Unicode normalization and retokenization bridges;
• complete JSON Schema, regex, or EBNF grammar compilation;
• GPU portability layers and shader toolchains;
• tree-attention construction for Medusa or EAGLE;
• statistical test frameworks;
• arbitrary model-format parsing and compression;
• cryptographic content hashing if TinyRustLM does not already contain a qualified implementation.

A clean architecture can keep the exact orchestration core dependency-free while generating tokenizer manifests, grammar automata, and test fixtures in audited build tools. Runtime use of an external grammar or GPU component must still be explicitly qualified; “zero crates” must not motivate an improvised, under-tested Unicode or cryptographic implementation.

TDD and adoption sequence

Clean deletion criteria

Delete rather than retain the speculative path when any of these remains true after the preregistered optimization budget:

• the target lacks an efficient batched-verification path;
• greedy mismatches persist under matched identities;
• stochastic distribution tests fail;
• draft loading and prefill do not amortize over expected sessions;
• peak memory violates the product’s supported hardware envelope;
• the draft raises energy per completed token without an accepted compensating product benefit;
• gains occur only on repetitive benchmarks and not on designated conversational strata;
• the scheduler must inspect private content in an unapproved way;
• cache rollback or cancellation remains unreliable;
• cross-tokenizer complexity exceeds its demonstrated incremental benefit;
• the feature cannot be cleanly disabled without branching the champion decoder throughout the codebase.

Unknowns and annotated primary-source bibliography

Facts requiring authorized local verification

The following determine adoption and cannot be resolved from public sources:

• champion and candidate draft parameter counts, artifact encodings, quantization layouts, tokenizer hashes, vocabulary IDs, templates, adapters, and licenses;
• whether the target runtime can compute position-aligned \(k+1\) logits in one call;
• numerical differences between TinyRustLM’s single-token and multi-token q4 kernels;
• draft and target KV layouts, page sizes, rollback costs, and pending-token semantics;
• actual browser and native loading peaks, allocator fragmentation, GPU allocation limits, and OOM behavior;
• prompt-prefill cost for both models and whether either prefill can overlap loading;
• acceptance \(S_i\) by the required workload strata;
• whether the draft is bandwidth-bound on the same CPU resources needed by the champion;
• WebGPU command-submission, synchronization, device-loss, and CPU-readback costs;
• WASM SIMD availability and fallback behavior across supported browsers;
• native-companion IPC latency, authentication, cancellation, and privacy boundaries;
• model artifact deduplication at file, tensor, and content-block levels;
• actual session-length distribution needed to amortize draft load and prefill;
• energy measurement availability;
• exact champion stop-string, grammar, top-\(p\), repetition-penalty, and processor order;
• whether a self-speculative, MTP, Medusa, or EAGLE-compatible champion artifact exists;
• whether continuation after EOS requires reusable final KV or always creates a new templated prompt;
• whether private evaluation permits storing token IDs or only aggregate receipts.

Annotated primary and official sources Retrieval date for every web source below: August 1, 2026.

Final adoption position. TinyRustLM should implement a removable research branch containing a scalar reference algorithm, transactional cache interface, same-tokenizer greedy mode, and exact stochastic mode. It should compare that branch against champion-alone and prompt lookup before investing in cross-tokenizer assistance, trained heads, self-speculation, or trees. The feature should be promoted only when local evidence shows that a clean implementation preserves the qualified champion and wins the full product cost function—not merely a warm tokens-per-second benchmark.