Exact Speculative And Assisted Decoding For A Tiny Draft And Qualified Champion

Full report

On this page

1. Executive Adoption Recommendation and Assumptions

The integration of speculative decoding into a browser-local, Rust-native inference engine presents a highly complex but potentially transformative architectural evolution. The fundamental objective is to leverage a small, computationally inexpensive "draft" model to predict a sequence of tokens that a larger "champion" model subsequently verifies in parallel. Based on a rigorous analysis of probabilistic sampling algorithms, browser memory constraints, and WebAssembly (WASM) / WebGPU execution models, the adoption of exact speculative decoding is conditionally recommended for TinyRustLM. Adoption must be strictly contingent upon the empirical validation of break-even cost formulas under preregistered, end-to-end performance gates. In browser environments, language model inference is predominantly bound by memory bandwidth rather than raw compute capability1. While speculative decoding successfully amortizes the memory bandwidth cost of the champion model across multiple tokens in a single forward pass, it introduces substantial new overheads. These include the draft model's memory residency, the duplication or complex management of Key-Value (KV) caches, and the synchronous latency of draft generation. This recommendation requires treating exact speculative decoding as a purely lossless acceleration technique. The champion's declared decoding distribution and raw outputs are authoritative; exact methods must guarantee mathematical equivalence to standard autoregressive generation3. Approximate methods, while offering higher theoretical throughput, violate this guarantee and must be explicitly budgeted separately under stringent semantic regression parameters. If a clean, zero-third-party-crate Rust implementation fails to outpace the standard champion baseline in target browser topologies without exceeding peak resident memory constraints, the speculative path must be deleted entirely without legacy support. This analysis operates on the assumption that TinyRustLM functions within the constraints of modern web standards, specifically relying on WebGPU for hardware-accelerated compute and WASM SIMD for CPU-bound tasks1. It is further assumed that artifact bytes remain strictly local or move directly between consenting peers, necessitating a rigid privacy boundary and decentralized cryptographic receipt generation.

2. Exact Versus Approximate Method Taxonomy

A foundational requirement for evaluating acceleration methods is distinguishing between algorithms that preserve the exact output distribution of the champion model and those that approximate it. Exact methods guarantee that the sequence generated under speculation has a probability strictly equal to the probability of that sequence being generated by the champion model running autoregressively. Independent draft speculative sampling represents the canonical exact method, where a standalone draft model generates candidate tokens, and the champion model verifies them in parallel3. Rejection sampling is utilized to mathematically correct differences between the draft and champion distributions. Prompt n-gram lookup serves as another exact, albeit parameter-free, drafting technique. It assumes the next tokens will copy a sequence already present in the prompt or recent context, generating candidates via simple string matching7. Verification remains exact because the champion evaluates the copied tokens conventionally. Universal speculative decoding methods, such as Token-Level Intersection (TLI) and String-Level Exact Match (SLEM), extend exactness to models with heterogeneous tokenizers8. TLI restricts the draft to a shared vocabulary space, while SLEM matches decoded text, preserving exactness through text normalization and re-tokenization9. Approximate methods trade exact distributional equivalence for higher acceptance rates or lower computational overhead. In the context of TinyRustLM, these must be treated as distinct configurations subject to semantic regression budgets. Early-exit drafts and self-speculative decoding bypass the final layers of the champion model to generate a draft11. While avoiding the need to load a separate draft model, skipping layers alters internal representations; unless verified with rigorous rejection sampling, this approach is approximate. Multi-token prediction heads, such as the Medusa architecture, require the champion to be trained with multiple output heads to predict [Figure omitted from source export] tokens simultaneously12. This modifies the champion's weights and alters the core generation process. Feature-level autoregression architectures like EAGLE and EAGLE-3 base their drafting on top-layer feature prediction rather than token prediction, requiring specialized draft architectures and training-time testing, often deviating from strict distributional exactness unless heavily constrained13.

Hidden best-of-N answer generation—running multiple distinct completions and utilizing a reward model or heuristic to select the preferred output—is categorically excluded from this taxonomy, as it alters the fundamental generation objective and violates the product constraints of TinyRustLM.

3. Mathematical Reconstruction and Correctness Argument

The theoretical foundation of exact speculative decoding rests on rejection sampling, proving that the output distribution remains indistinguishable from the champion model4. To reconstruct this, let the target (champion) model's probability distribution for the next token be denoted as [Figure omitted from source export] and the draft model's distribution be [Figure omitted from source export]. The drafting phase begins with the draft model autoregressively generating a sequence of [Figure omitted from source export] tokens [Figure omitted from source export] by sampling from its distribution [Figure omitted from source export]. During verification, the champion model evaluates the entire drafted sequence in a single parallel forward pass, producing target probabilities [Figure omitted from source export] for each respective position. Token-wise acceptance relies on a random variable [Figure omitted from source export] drawn from a deterministic RNG source for each token [Figure omitted from source export]. The token is accepted if the drawn value satisfies the condition [Figure omitted from source export]. Equivalently, the token is accepted with a probability of [Figure omitted from source export]5. If token [Figure omitted from source export] is rejected, all subsequent drafted tokens [Figure omitted from source export] are immediately discarded. To ensure the final output strictly adheres to the champion's declared distribution, a replacement token must be sampled from an adjusted rejection correction distribution, defined as: [Figure omitted from source export] If all [Figure omitted from source export] proposals are accepted, the algorithm capitalizes on the champion's parallel forward pass by sampling an extra target token directly from [Figure omitted from source export]. Stop-token handling is straightforward but critical: if the draft proposes an EOS (End of Sequence) token and it is accepted by the champion, the generation halts precisely at that point, ensuring exact semantic termination. The proof obligation that the output follows the champion's declared distribution is absolute4. The total probability of emitting a specific token [Figure omitted from source export] encompasses two disjoint cases: the probability that it was drafted and accepted, plus the probability that it was rejected and subsequently resampled from the residual correction distribution. The probability of Case 1 (acceptance) is [Figure omitted from source export]. The probability of Case 2 (rejection and resampling) relies on the total probability of any rejection occurring, which is [Figure omitted from source export]. The probability of sampling [Figure omitted from source export] from the residual distribution [Figure omitted from source export] is therefore [Figure omitted from source export]. Combining these cases, the total probability of emitting token [Figure omitted from source export] is [Figure omitted from source export], which algebraically simplifies exactly to [Figure omitted from source export]4. This mathematically guarantees that regardless of how poor the draft model [Figure omitted from source export] is, the final output distribution is exactly [Figure omitted from source export]. A poor draft model simply yields a high rejection rate, degenerating throughput without altering semantic fidelity.

4. Normative Greedy and Stochastic Pseudocode

The following normative pseudocode defines the exact implementation for a zero-third-party-crate Rust core. It covers batch size one, speculation lengths of zero and [Figure omitted from source export], full acceptance, partial rejection, handling of zero-probability draft tokens (numerical underflow), all-masked logits, and strict RNG consumption for deterministic replay.

Rust /// Core Speculative Decoding Step for TinyRustLM /// Assumes f32 precision for probabilities to mitigate bf16/f16 underflow. fn speculative\_decode\_step( champion: &mut Model, draft: &mut Model, prompt\_tokens: &\[u32\], rng: &mut Rng, k: usize, temperature: f32, max\_tokens: usize, ) \-\> Vec\<u32\> { // Handle Speculation Length 0 (Standard Autoregressive) if k \== 0 { let logits \= champion.forward\_single(\&prompt\_tokens); let probs \= apply\_temperature\_and\_softmax(logits, temperature); return vec\!\[sample\_from\_probs(\&probs, temperature, rng)\]; }

let mut proposals \= Vec::with\_capacity(k); let mut draft\_probs \= Vec::with\_capacity(k); let mut current\_input \= prompt\_tokens.to\_vec();

// 1\. Drafting Phase for \_ in 0..k { if current\_input.len() \>= max\_tokens { break; }

let logits \= draft.forward\_single(\&current\_input); let probs \= apply\_temperature\_and\_softmax(logits, temperature); let token \= sample\_from\_probs(\&probs, temperature, rng);

proposals.push(token); draft\_probs.push(probs); current\_input.push(token);

// EOS inside a proposal immediately halts further drafting if is\_eos(token) { break; } }

let actual\_k \= proposals.len(); if actual\_k \== 0 { return vec\!\[\]; }

let mut accepted\_tokens \= Vec::new();

// 2\. Parallel Verification Phase // Champion evaluates the last prompt token \+ all drafted tokens in one batched pass let champion\_logits\_batch \= champion.forward\_batch(\&current\_input);

// 3\. Acceptance / Rejection Phase for i in 0..actual\_k { let p\_logits \= \&champion\_logits\_batch\[i\]; let p\_probs \= apply\_temperature\_and\_softmax(p\_logits, temperature);

let target\_prob \= p\_probs\[proposals\[i\] as usize\]; let draft\_prob \= draft\_probs\[i\]\[proposals\[i\] as usize\];

// Handle zero-probability draft tokens and numerical underflow safely let acceptance\_ratio \= if draft\_prob \<= f32::EPSILON { 1.0 // If drafted despite 0 prob, force target evaluation } else { (target\_prob / draft\_prob).min(1.0) };

// Distinguish Greedy vs Stochastic let accept \= if temperature \== 0.0 { // Greedy: Accept only if the proposed token is exactly the target's argmax let is\_argmax \= target\_prob \== p\_probs.iter().cloned().fold(f32::NAN, f32::max); is\_argmax && target\_prob \> 0.0 } else { // Stochastic: Consume RNG deterministically rng.gen::\<f32\>() \<= acceptance\_ratio };

if accept { accepted\_tokens.push(proposals\[i\]); if is\_eos(proposals\[i\]) { // Rollback caches for unverified/discarded tokens beyond EOS champion.rollback\_kv(actual\_k \- (i \+ 1)); draft.rollback\_kv(actual\_k \- (i \+ 1)); return accepted\_tokens; } } else { // 4\. Correction Phase (First or Later Rejection) let mut residual \= vec\!\[0.0; p\_probs.len()\]; let mut sum \= 0.0; for j in 0..p\_probs.len() { residual\[j\] \= (p\_probs\[j\] \- draft\_probs\[i\]\[j\]).max(0.0); sum \+= residual\[j\]; }

// Handle catastrophic precision loss or all-masked logits let corrected\_token \= if sum \> f32::EPSILON { for j in 0..residual.len() { residual\[j\] /= sum; } sample\_from\_probs(\&residual, temperature, rng) } else { // Fallback directly to target distribution sample\_from\_probs(\&p\_probs, temperature, rng) };

accepted\_tokens.push(corrected\_token);

// Rollback state: champion rejects tokens from i+1 to k, draft rolls back identically champion.rollback\_kv(actual\_k \- i); draft.rollback\_kv(actual\_k \- i); return accepted\_tokens; } }

// 5\. Bonus Token Phase (All proposals accepted) if current\_input.len() \< max\_tokens { let final\_logits \= \&champion\_logits\_batch\[actual\_k\]; let final\_probs \= apply\_temperature\_and\_softmax(final\_logits, temperature); let bonus\_token \= sample\_from\_probs(\&final\_probs, temperature, rng); accepted\_tokens.push(bonus\_token); }

accepted\_tokens }

5. Tokenizer, Template, Model, and Positional Compatibility

Model and prompt artifact compatibility dictates the viability of any speculative execution. The draft and champion models represent distinct content identities; modifying any sub-component invalidates the generation state unless a mathematically proven transform exists. Draft and target tokenizer compatibility presents a significant engineering hurdle. The simplest scenario involves an exact shared vocabulary and equivalent token IDs, allowing seamless integer matching during the verification phase. If both models utilize the same tokenizer algorithm but have different added tokens or differing ID mappings (byte-equivalent but ID-different tokenizers), a retokenization bridge—an exact 1:1 mapping table—must be injected between the draft output and the target input. When tokenizers are fundamentally incompatible, exactness can still be preserved via Universal Speculative Decoding methods8. Token-Level Intersection (TLI) is a method that dynamically constrains the draft model to sample only from the intersection of the two vocabularies9. The draft's probability mass is renormalized, ensuring that the target model always receives valid tokens. While this preserves exactness with minimal overhead, it restricts the draft model's expressive capacity. Conversely, String-Level Exact Match (SLEM) allows the draft model to sample freely from its full vocabulary9. The draft tokens are decoded into strings and subsequently re-tokenized by the target model's tokenizer. This universal byte-level verification yields higher acceptance rates but introduces substantial complexity: the system must perform dynamic string decoding, manage incompatible normalization schemes, and execute complex rollback logic when a sequence only partially aligns at the sub-word level. For a zero-third-party-crate Rust core, TLI is recommended as it requires significantly less additional target work while preserving mathematical exactness. Beyond tokenization, both artifacts must share bound chat templates and system prompts. If a draft model utilizes an incompatible chat template, its generated logits will drift immediately upon encountering user/assistant boundaries. Positional policies and context limits also heavily influence compatibility. If the champion model utilizes a Rotary Position Embedding (RoPE) scaled to 128,000 tokens while the draft model is capped at 4,096, drafting will fail catastrophically as the sequence extends beyond the draft's capacity. Positional embeddings must align, and the engine must track sampling masks carefully to ensure that BOS/EOS rules do not trigger prematurely in the draft architecture.

6. Target Batch-Verification API and Kernel Implications

The core architectural constraint of browser-based execution is memory bandwidth, particularly across the CPU-to-GPU bus and within the GPU's VRAM1. Standard autoregressive decoding utilizes a single-token target API. Speculative decoding necessitates a fundamental redesign to support batched verification. The target API must shift to evaluating [Figure omitted from source export] proposed tokens simultaneously. Repeated single-token calls to the champion model are strictly forbidden, as they invoke the memory bandwidth penalty of reading the full parameter weights [Figure omitted from source export] times, nullifying the premise of speculative decoding. The batched verification API must accept an input array of size [Figure omitted from source export] and return all logits for the [Figure omitted from source export] proposed tokens, plus the final bonus token.

In WebGPU, dispatching a compute shader incurs latency overhead relative to native execution. Batching [Figure omitted from source export] tokens requires only a single WebGPU command buffer submission, effectively mimicking prefill-like execution2. Causal masking must be meticulously applied during this batched pass; the attention mechanism must ensure that token [Figure omitted from source export] can only attend to previous tokens and the prompt, not to [Figure omitted from source export]. The logits buffer size presents a memory management challenge in WASM. A champion model outputting [Figure omitted from source export] logit vectors for a vocabulary of 128,000 in f32 requires a contiguous buffer of several megabytes (e.g., [Figure omitted from source export]). This must be pre-allocated in WebGPU storage buffers and mapped to WASM memory without zero-initialization overhead. Furthermore, weight-only quantization (e.g., Q4 kernel shapes) traditionally relies on matrix-vector (GEMV) operations. For batched verification, this becomes a thin matrix-matrix (GEMM) operation. WebGPU compute shaders must be explicitly tuned for narrow matrix shapes to maximize thread group occupancy and prevent under-utilization of GPU compute units1.

7. Draft/Target KV Transaction and Rollback Model

Key-Value (KV) cache state management dictates peak memory residency and determines the efficiency of the rollback phase. Speculative decoding requires entirely separate KV caches for the draft and champion models, as they possess different embedding dimensions, layer counts, and attention heads. The transaction model follows a strict append, commit, and rollback cycle. Both models first execute a shared prompt prefill, writing to their respective isolated caches. During the drafting phase, the draft model performs a proposal append [Figure omitted from source export] times, advancing its KV cache pointer. The target model then performs a batched verification, appending the [Figure omitted from source export] proposals to its own KV cache. If [Figure omitted from source export] tokens are accepted (where [Figure omitted from source export]), an accepted-prefix commit occurs, and the logical pointers for valid sequence length are permanently advanced by [Figure omitted from source export]. If [Figure omitted from source export], a rejected-tail rollback is triggered. Crucially, because the KV cache is implemented as an append-only ring buffer or via paged granularity (PagedAttention), physical memory zeroing is unnecessary. The rollback is an [Figure omitted from source export] logical operation: the system simply decrements the sequence length pointers of both the draft and champion caches by [Figure omitted from source export]. Context truncation and cancellation during verification must trigger this same [Figure omitted from source export] pointer reset. Peak memory calculations must explicitly count duplicate KV memory and scratch spaces. If a 1B parameter draft model is used alongside a 7B champion, the draft model's weights represent only \~14% of the champion's size. However, its KV cache may still consume significant VRAM if the context window is large. TinyRustLM must monitor physical deduplicated bytes and logical composition bytes via atomic counters to ensure browser limits (typically 2GB \- 4GB per WebGPU context) are not exceeded6. Exact state after EOS requires freezing the KV cache pointers immediately to prevent out-of-bounds memory accesses during the bonus token generation phase.

8. Loading, Residency, Browser, CPU, and WebGPU Topologies

In a browser context, the memory boundary between JavaScript, WASM linear memory, and WebGPU VRAM is rigid. Passing tensors across these boundaries destroys performance1.

Streaming the champion weights on-demand while the draft is resident is mathematically incompatible with speculative decoding. The [Figure omitted from source export]\-token verification phase requires a full forward pass over the champion's weights. If these weights must be streamed from the Origin Private File System (OPFS) for every verification step, the latency will drastically exceed standard generation times, nullifying any speedup.

9. Cost and Break-Even Formulas

Evaluating the efficacy of speculative decoding requires moving beyond naive "tokens per second" metrics to derive strict mathematical break-even formulas based on wall-clock execution time16. Let the variables be defined as follows:

• [Figure omitted from source export] \= Time to generate one draft token (draft prefill and decode cost).
• [Figure omitted from source export] \= Time for the champion to verify [Figure omitted from source export] tokens in a single batch.
• [Figure omitted from source export] \= Time for the champion to generate one token autoregressively.
• [Figure omitted from source export] \= Average token acceptance probability by position.
• [Figure omitted from source export] \= Speculation length.
• [Figure omitted from source export] \= Fixed costs per cycle, including memory synchronization, cache rollback, and rejection correction distribution calculation.

The expected number of accepted tokens per verification step, [Figure omitted from source export], is modeled as the expected value of a truncated geometric distribution. Taking into account the bonus token generated upon full acceptance, the formula is: [Figure omitted from source export] The expected wall-clock time per speculative step is: [Figure omitted from source export] The fundamental break-even condition dictates that speculative decoding is only beneficial if the time spent per accepted token is strictly less than the time spent generating a token autoregressively with the champion: [Figure omitted from source export] Expanding this yields the exact requirement for performance improvement: [Figure omitted from source export] If this inequality does not hold for a given prompt or hardware topology, exact speculative decoding becomes a net negative to the product, wasting energy, triggering unnecessary model-switch costs, and degrading latency. The system must continuously evaluate this formula at runtime.

10. Adaptive Scheduling and Grammar Interaction

A fixed speculation length [Figure omitted from source export] is highly suboptimal18. If the acceptance rate [Figure omitted from source export] drops due to context ambiguity, generating [Figure omitted from source export] tokens wastes [Figure omitted from source export] compute time and increases [Figure omitted from source export] without improving [Figure omitted from source export]. Adaptive scheduling allows the system to dynamically modulate [Figure omitted from source export]. Heuristic dynamic schedules are the simplest: increase [Figure omitted from source export] by 1 (up to a hardware-defined limit) if all tokens are accepted, and decrease [Figure omitted from source export] by 1 if the first drafted token is rejected. Confidence-based schedules (such as DISCO or SpecDec++) offer a more sophisticated approach. In these architectures, a learned predictor or the draft model's own logit entropy is evaluated at each step. If the draft model's confidence falls below a specific threshold, it voluntarily halts generation early, returning a dynamically sized sequence [Figure omitted from source export] to the champion for verification18. This schedule must be tightly bound to a versioned policy to prevent feedback loops derived from private prompts or hidden answer quality. Interactions with grammar and deterministic tokens present unique acceleration opportunities20. When utilizing a structured generation engine (e.g., XGrammar for JSON schema enforcement), specific tokens (such as {", ":) are deterministically forced by the output grammar. For these tokens, the neural draft proposal mechanism can be bypassed entirely. The grammar engine drafts the forced tokens at near-zero latency. However, target verification remains absolutely mandatory for these grammar-forced tokens. The champion model must process them in a batched forward pass to update its internal KV cache state accurately. Skipping target verification for grammar tokens would corrupt the champion's context, leading to catastrophic semantic failure in subsequent non-forced generation steps.

11. Draft Training, Selection, and Contamination Boundaries

A small specialist is not automatically a valid speculative draft. Generic small instruct models frequently exhibit fundamentally different vocabulary distributions and alignment characteristics compared to a target champion, leading to abysmal acceptance rates11. Draft models must undergo draft-specialized training to align their probability manifolds with the champion. Knowledge distillation is the standard approach, where the draft model is trained directly on the raw logits (minimizing Kullback-Leibler divergence) of the champion23. Techniques like AdaSPEC employ selective token filtering during this distillation process, allowing the draft to focus its limited parameter capacity on easily predictable tokens while deferring ambiguous tokens to the champion, thereby optimizing the acceptance rate24. Contamination boundaries are paramount. Draft training must possess independent provenance and strict contamination controls. Draft models must never be trained or distilled on sealed product evaluation answers or private user history. TinyRustLM must demand cryptographic provenance for any imported draft model, verifying its training corpus against a known-clean manifest before permitting it to operate alongside the champion. Online adaptation (fine-tuning the draft on user prompts locally) is permitted only if the weights remain cryptographically locked to the local device.

12. Preregistered Workload, Metrics, Quality, and Exactness Matrix

To prevent confirmation bias, evaluation must be conducted against preregistered workload strata. A single repetitive benchmark is explicitly insufficient as conversational evidence. The workload strata must include: casual chat, ambiguity handling, source conflict resolution, untrusted memory parsing, summaries, rewrites, strict JSON extraction, repetitive text, multilingual text, long context processing, and adversarial low-acceptance inputs (where [Figure omitted from source export] intentionally approaches 0). Metrics must extend significantly beyond tokens per second. The preregistered matrix must track:

1. Accepted Tokens per Target Pass: Target [Figure omitted from source export].

2. First-Token Latency (TTFT): Must not regress by more than the cost of one single draft forward pass.

3. Acceptance by Position and Task: Track [Figure omitted from source export] decay as [Figure omitted from source export] increases.

4. Energy and Peak Memory: Track peak VRAM usage and model/cache byte ratios.

5. Target and Draft Calls: Track the ratio of compute utilized.

Exactness and quality gates are immutable. For greedy decoding (Temperature \= 0.0), exact token identity is required; the output byte array must match the champion baseline bit-for-bit. For stochastic exact sampling, the system must use deterministic RNG replay across both algorithms to ensure sequence exactness, supplemented by distributional divergence testing over 10,000 samples to verify that the rejection correction formula is perfectly implemented. Approximate modes require sealed human and semantic evaluation, adhering to an explicit, pre-defined regression budget (e.g., [Figure omitted from source export] degradation on MMLU).

13. Fault Injection, Privacy-Safe Receipts, and Fallback Behavior

Browser execution environments are inherently unstable. WebWorkers die, WebGPU contexts are lost due to OS-level preemption, and memory allocation can fail silently1. The implementation must survive a rigorous fault-injection matrix.

Fail-visible fallback without duplicate answer text is critical. If a fault occurs during the parallel verification phase, the system must seamlessly downgrade to the champion's standard autoregressive loop. It must recover its state from the last committed accepted-prefix pointer, ensuring the user interface never displays duplicated or corrupted text. Evidence and privacy must be maintained via privacy-safe receipts. Telemetry receipts may include artifact hashes, tokenizer hashes, static policy definitions, random seeds, acceptance counts, wall-clock timings, peak memory, and bounded mathematical errors. Receipts must cryptographically exclude all private prompts, outputs (where private), raw model bytes, local filesystem paths, credentials, and hidden user history, ensuring zero-trust peer-to-peer integrity.

14. TDD Backlog, Promotion Thresholds, and Clean Deletion Criteria

The implementation within the zero-third-party-crate Rust core must proceed via a rigorous Test-Driven Development (TDD) plan.

1. Synthetic Probability Model: Implement the rejection sampling and correction math. Use exhaustive arrays simulating [Figure omitted from source export] and [Figure omitted from source export] to verify the algorithm produces exactly [Figure omitted from source export] over 1,000,000 Monte Carlo simulations.

2. Exhaustive Tiny-Vocabulary Tests: Utilize a toy tokenizer (vocab size \= 100). Implement KV cache pointer updates, testing exact sequence rollback upon injected rejections.

3. Deterministic Golden Vectors: Implement standard draft/champion models and replay known prompts at Temperature \= 0.0 to ensure bit-for-bit output equality against a Python baseline.

4. Browser Execution: Deploy to WASM/WebGPU, testing real tokenizer implementations (TLI/SLEM bridges) and verifying asynchronous WebWorker message passing.

5. Sealed Quality Evaluation: Run the full preregistered matrix.

Promotion Thresholds: The speculative decoding pipeline will be merged into the primary release branch only if [Figure omitted from source export] and end-to-end wall-clock latency improves by [Figure omitted from source export] over the standard champion on target hardware, without breaching peak VRAM limits.Clean Deletion Stop Rules: If the implementation fails to meet the promotion thresholds within a predefined 8-week evaluation window, or if the break-even formula remains consistently inverted under real-world usage, the entire speculative code path, specific cache abstractions, and associated scheduler logic must be cleanly deleted from the repository. No unsuccessful assisted-decoding path will be retained as legacy code.

15. Unknowns Requiring Private Models and Local Execution

Because access to private TinyRustLM code, proprietary .slm files, and routing logic is restricted, several engineering variables remain strictly unknown and require local, authorized verification:

• The exact memory layout of the internal Model struct, specifically how RoPE states, attention masks, and KV cache pointers are exposed to the Rust borrow checker. Safe, zero-copy rollbacks require mutable access to these indices without triggering memory reallocation.
• The proprietary routing logic. It is unknown how TinyRustLM dynamically decides whether a small task specialist operates as an independent responder for a query or is demoted to serve as a speculative draft for a larger champion.
• The statistical distribution of target devices. The precise ratio of clients utilizing WASM SIMD CPU fallbacks versus those with full WebGPU hardware acceleration will fundamentally alter the [Figure omitted from source export] verification cost and dictate whether speculative decoding is viable in production.

16. Annotated Primary-Source Bibliography

• \[cite: 3\] Leviathan, Y., Kalman, M., & Matias, Y. (2022). Fast Inference from Transformers via Speculative Decoding. (arXiv:2211.17192). The foundational paper introducing speculative sampling. Reconstructs the exact mathematical proof that rejection sampling maintains the target distribution identically.
• \[cite: 12, 13\] Cai, T., et al. (2024). Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads / Li, Y., et al. (2025). EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test. (arXiv:2503.01840). Key research on approximate and feature-level drafting mechanisms utilizing tree attention and multi-token prediction heads.
• \[cite: 4, 5\] Chen, C., et al. (2023). Accelerating Large Language Model Decoding with Speculative Sampling. (arXiv:2302.01318). DeepMind's independent derivation of speculative decoding, reinforcing the exactness of the output distribution via rigorous mathematical proofs.
• \[cite: 8, 9, 10\] Timor, N., et al. (2025). Accelerating LLM Inference with Lossless Speculative Decoding Algorithms for Heterogeneous Vocabularies. (arXiv:2502.05202). Defines the Token-Level Intersection (TLI) and String-Level Exact Match (SLEM) algorithms, enabling draft models with differing tokenizers to preserve exact distributions.
• \[cite: 18, 19\] Mamou, J., et al. (2024). Dynamic Speculation Lookahead Accelerates Speculative Decoding (DISCO) / Huang, K., et al. (2024). SpecDec++: Boosting Speculative Decoding via Adaptive Candidate Lengths. (arXiv:2405.19715). Analyzes adaptive speculation length scheduling based on draft model confidence and Markov Decision Processes.
• \[cite: 23, 24\] Hu, Y., et al. (2025). AdaSPEC: Selective Knowledge Distillation for Efficient Speculative Decoders. (arXiv:2510.19779). Details knowledge distillation mechanics for aligning draft models with champions to maximize acceptance rates.
• \[cite: 22, 25\] Williams, M., et al. (2026). Speculative Decoding with a Speculative Vocabulary (SpecVocab). (arXiv:2602.13836). Details methods for reducing output distribution bottlenecks by dynamically predicting vocabulary subsets.
• \[cite: 20, 21\] WebLLM / XGrammar (2025). CMU-CS-25-112. Documentation detailing WASM/WebGPU compilation stacks and constrained grammar generation via zero-overhead mask generation, vital for browser-native Rust deployments.

Works cited

1. Combining WebAssembly with WebGL High-Performance Graphics Processing, https://dev.to/tianyaschool/combining-webassembly-with-webgl-high-performance-graphics-processing-322

2. WebGPU: Next-Generation Graphics and Compute | MinhVo, https://minhvo.is-a.dev/blogs/webgpu-next-generation-graphics-and-compute

3. \[2211.17192\] Fast Inference from Transformers via Speculative Decoding \- ar5iv \- arXiv, https://ar5iv.labs.arxiv.org/html/2211.17192

4. Speculative Decoding in Production: How a 1B Draft Model Cuts 70B Latency by 3-5, https://aistackinsights.ai/blog/speculative-decoding-production-llm-inference-guide

5. Speculative Decoding: Getting K Tokens for the Price of One | Salman Quazi, https://www.salmanq.com/blog/speculative-decoding/

6. Run AI Models in the Browser with WebGPU & WASM \- Mad Devs, https://maddevs.io/writeups/running-ai-models-locally-in-the-browser/

7. llama-cpp-pydist \- PyPI, https://pypi.org/project/llama-cpp-pydist/

8. \[2502.05202\] Accelerating LLM Inference with Lossless Speculative Decoding Algorithms for Heterogeneous Vocabularies \- arXiv, https://arxiv.org/abs/2502.05202

9. \[Feature Request\] SLEM (String-Level Exact Match) speculative decoding for heterogeneous vocabularies \#45020 \- GitHub, https://github.com/vllm-project/vllm/issues/45020

10. \[Feature\]: Universal Speculative Decoding for Heterogeneous Vocabularies (TLI / Token-Level Intersection) · Issue \#38173 · vllm-project/vllm \- GitHub, https://github.com/vllm-project/vllm/issues/38173

11. 𝚂𝚙𝚎𝚌𝙵𝚘𝚛𝚐𝚎: A Flexible and Efficient Open-Source Training Framework for Speculative Decoding \- arXiv, https://arxiv.org/html/2603.18567v1

12. Medusa: Simple Framework for Accelerating LLM Generation with Multiple Decoding Heads, https://www.youtube.com/watch?v=JmYFunlTeVI

13. \[2503.01840\] EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test \- arXiv, https://arxiv.org/abs/2503.01840

14. Speculative Speculative Decoding \- arXiv, https://arxiv.org/html/2603.03251v2

15. Edge LLM Browser Agents with WebGPU/WASM: On ... \- surfs.dev, https://surfs.dev/resources/edge-llm-browser-agents-webgpu-wasm-planning-streaming-tools-privacy

16. TokenTiming: A Dynamic Alignment Method for Universal Speculative Decoding Model Pairs \- ACL Anthology, https://aclanthology.org/2026.acl-long.1983.pdf

17. Domino: Decoupling Causal Modeling from Autoregressive Drafting in Speculative Decoding \- arXiv, https://arxiv.org/html/2605.29707v1

18. \[2405.04304\] Dynamic Speculation Lookahead Accelerates Speculative Decoding of Large Language Models \- arXiv, https://arxiv.org/abs/2405.04304

19. \[2405.19715\] SpecDec++: Boosting Speculative Decoding via Adaptive Candidate Lengths, https://arxiv.org/abs/2405.19715

20. GitHub \- alvinreal/awesome-opensource-ai: Curated list of the best truly open-source AI projects, models, tools, and infrastructure. Daily updated., https://github.com/alvinreal/awesome-opensource-ai

21. Democratizing On-Device LLM Inference with Machine Learning Compilers and Web Technologies \- Carnegie Mellon University, http://reports-archive.adm.cs.cmu.edu/anon/2025/CMU-CS-25-112.pdf

22. \[2602.13836\] Speculative Decoding with a Speculative Vocabulary \- arXiv, https://arxiv.org/abs/2602.13836

23. Temperature-Centric Investigation of Speculative Decoding with Knowledge Distillation \- ACL Anthology, https://aclanthology.org/2024.findings-emnlp.767.pdf

24. Paper page \- AdaSPEC: Selective Knowledge Distillation for Efficient Speculative Decoders, https://huggingface.co/papers/2510.19779