Localization of Numerical Divergence in Prerelease TinyRustLM Inference

Full report

On this page

Executive Decision and Falsification

The recommended approach to isolate numerical divergence in the prerelease TinyRustLM engine is the construction of a strictly deterministic, simulated-scalar Python reference pipeline. This pipeline must exactly mirror the project's row-wise symmetric int8 quantization scheme and its explicit left-to-right F32 accumulation contracts. By enforcing strict environmental determinism within the canonical PyTorch 2.7.0 environment, this reference will act as the absolute ground truth. The subsequent audit must sequentially evaluate scalar Rust, optimized native Rust, and WebAssembly (WASM) inferences against this ground truth using a hierarchical, teacher-forced tracing methodology stored in the SafeTensors format.

The strongest reason this recommendation could fail is if the historical 54-second model loading failure stems from an environment-level exception—such as memory exhaustion, file-system mapping limits in the WASM sandbox, or structural binary incompatibility in the proprietary model.slm2 file—rather than a true arithmetic divergence during inference. If the execution framework crashes before the first token's forward pass completes, a numerical tracing harness cannot collect the necessary intermediate activations to perform differential debugging, rendering numerical analysis moot until the runtime environment is stabilized.

The smallest experiment that could falsify this recommendation is to execute a single-token forward pass of a minimal, synthetically generated dummy payload (e.g., a one-layer, one-head, one-hidden-dimension configuration) through the Python reference, scalar Rust, and WASM pipelines. If this minimal trace crashes during the load phase or the initial execution step, it confirms a structural or environmental defect rather than an arithmetic divergence. This would invalidate the immediate need for deep numerical profiling, pivoting the engineering focus toward memory management and structural binary parsing.

Epistemological Ledger

To ensure rigorous differential debugging without access to the proprietary codebase, the following analysis partitions the project state into distinct epistemic categories. This separation prevents the conflation of verified architectural truths with local implementation artifacts.

Project-Supplied Facts

The current implementation binds exactly six artifacts: model.slm2, tokenizer.tokenizer2, template.template2, sampling.sampling2, prompt.prompt2, and composition.acg2. The active model is a six-member Qwen3 composition, with the model.slm2 payload occupying exactly 598,308,416 bytes. A previous source-framework trace failed at approximately 54 seconds during the model load phase; consequently, reconstructed and scalar comparison stages never executed. The canonical environment is strictly defined as Python 3.12.10, PyTorch 2.7.0, Transformers 4.51.0, NumPy 2.2.5, Safetensors 0.5.3, Tokenizers 0.21.1, and Accelerate 1.14.0. The target weight profile relies on row-wise symmetric int8 matrices with finite F32 scales, alongside F32 normalization weights and admitted biases. The scalar execution contract demands explicit F32 operations and left-to-right accumulation without Fused Multiply-Add (FMA) instructions. Canonical source paths are strictly E:\\Source\\Rust\\TinyRustLM.com, and model payloads reside in D:\\LLMs\\TinyRustLM. MiniModel.org is explicitly authorized to supply live initial server seeds, overriding any legacy project prose prohibiting external network sourcing for initial default models. The rejected Qwen3-0.6B revision c1899de289a04d12100db370d81485cdf75e47ca remains available exclusively for runtime diagnosis.

Externally Verified Facts

The Qwen2.5/Qwen3 0.5B base architecture utilizes 24 hidden layers, a hidden dimension of 896, an intermediate (MLP) dimension of 4864, and tied word embeddings1. The model employs Grouped Query Attention (GQA) with 14 query heads and 2 key/value heads, yielding a head dimension of 641. A critical architectural deviation in Qwen3 models is the introduction of independent Root Mean Square Normalization (RMSNorm) applied exclusively to the queries (q\_norm) and keys (k\_norm) prior to Rotary Position Embeddings (RoPE)4. The MLP utilizes a SwiGLU activation, requiring the element-wise multiplication of a SiLU-activated gate projection and an up projection7.

Within the execution environment, PyTorch utilizes non-deterministic algorithms and TensorFloat-32 (TF32) on Ampere-class GPUs by default, which reduces FP32 mantissa precision to 10 bits9. Reproducibility requires explicitly disabling TF32 and enabling deterministic algorithms11. Furthermore, WebAssembly strictly enforces IEEE-754 32-bit and 64-bit floating-point arithmetic; divergence only legally occurs regarding NaN payloads and via the "relaxed SIMD" proposal13. Finally, subnormal floating-point handling differs across hardware architectures, where enabling Flush-to-Zero (FTZ) or Denormals-Are-Zero (DAZ) breaks IEEE-754 strictness16.

Hypotheses

The 54-second crash during the model.slm2 loading phase is hypothesized to result from an out-of-memory exception, a 32-bit WASM memory addressing limitation, or an invalid binary offset in the custom .slm2 format parsing logic, rather than an arithmetic failure. Regarding tokenization, Rust Byte Pair Encoding (BPE) tokenizers are hypothesized to fail or diverge on invalid UTF-8 byte fallbacks compared to Python reference tokenizers19. During inference, native Rust optimizations likely utilize AVX-512 VNNI instructions (e.g., vpdpbusd), which compute fused 8-bit dot products and accumulate into 32-bit integers, bypassing the rounding errors inherent in the specified scalar left-to-right F32 accumulation and thus generating localized numerical drift21.

Recommendations

The engineering team must abandon attempts to validate inference solely through structural inspection. Mathematical correctness cannot be established by successfully loading a binary into memory structs. The diagnostic process must focus on deep execution tracing. Furthermore, legacy compatibility paths for the retired SLM1 format should be discarded immediately to reduce technical debt and clarify the active codebase.

Locally Unverified Conditions

The analysis proceeds without verifying the exact byte-level struct packing of the proprietary .slm2 and .acg2 formats. The underlying hardware architecture (x86\_64 versus ARM64) executing the WebAssembly environment remains unknown, which dictates the severity of relaxed SIMD divergence13. Additionally, the presence of hardware-accelerated VNNI or AMX instruction sets on the local development machine at E:\\Source\\Rust is unverified.

Specification of Execution Reference Arms

To rigorously isolate quantization artifacts from core engine implementation bugs, the differential debugging program must evaluate execution across five distinct reference arms. Comparing a quantized inference engine directly against an unquantized FP16 PyTorch model is a widespread anti-pattern. Quantization inherently alters the underlying mathematics through representation limits and asymmetric scaling. A bug-free quantized engine must diverge from an FP16 baseline; interpreting this divergence as a bug leads to perpetual, unresolvable optimization cycles.

Arm A: Original Source Model (FP16/BF16)

This arm represents the unmodified Hugging Face canonical representation of the Qwen3 model, executing in PyTorch using bfloat16 or float16 precision. Its purpose is to establish the theoretical upper bound of the model's linguistic capability and to capture the reference logits prior to weight compression. The expected parity with the deployed TinyRustLM engine is statistical similarity only. Exact numerical parity is mathematically impossible and must not be a product acceptance gate.

Arm B: Emulated Quantized Source-Framework (The Ground Truth)

Arm B is a custom PyTorch implementation executing the independently reconstructed row-wise symmetric int8 weights. This arm serves as the absolute, bit-exact ground truth for all subsequent engineering targets in TinyRustLM. To ensure validity, this execution must enforce strict determinism by setting torch.use\_deterministic\_algorithms(True) and torch.backends.cuda.matmul.allow\_tf32 \= False9. Furthermore, standard PyTorch torch.matmul functions cannot be used, as cuBLAS utilizes hierarchical block accumulation that alters floating-point rounding24. The harness must utilize a custom PyTorch extension or highly controlled tensor looping to emulate the strict left-to-right sequential accumulation required by the TinyRustLM scalar contract.

Arm C: Scalar Rust

This arm is the native Rust implementation executing standard scalar f32 multiplication and addition in a strict, unoptimized left-to-right loop. The expected parity between Arm C and Arm B is exact agreement (0 Units in the Last Place, or ULP). Because both arms execute identical arithmetic operations in an identical sequence over identical quantized weights, any divergence strictly indicates an indexing bug, a shape broadcasting error, a tensor layout mismatch, or an improperly handled subnormal floating-point value.

Arm D: Optimized Native Rust

This represents the production Rust target utilizing hardware intrinsics, specifically AVX-512 vpdpbusd for INT8 Vector Neural Network Instructions (VNNI) dot products, or their NEON equivalents21. This arm will inherently diverge from both Arm B and Arm C. Vectorized dot-product instructions compute fused multiplications and maintain higher internal precision (or utilize intermediate 32-bit integer accumulators) before ultimately casting the sum to f3222. The acceptance criteria for Arm D is approximate agreement within a defined absolute error bound (e.g., [Figure omitted from source export]) relative to Arm C.

Arm E: Browser WebAssembly (WASM)

Arm E is the final deployment target executing via a browser's WASM engine. WebAssembly strictly enforces IEEE-754 semantics26. If the WASM target disables the "relaxed SIMD" proposal, the arithmetic must be perfectly deterministic13. Under these conditions, it must achieve exact 0 ULP parity with Arm C. If WebAssembly SIMD is enabled, the execution will exhibit bounded, hardware-dependent deviation, and should be evaluated against the tolerances established for Arm D.

Reference-Model Loading as an Isolated Experiment

The historical trace failure at roughly 54 seconds during model loading suggests a critical bottleneck in memory allocation or file I/O, independent of the neural network's arithmetic logic. Evaluating the inference engine requires separating environment failure from runtime disagreement.

The canonical environment is bound to Python 3.12.10 and PyTorch 2.7.0. The diagnostic harness must first verify this execution space before allocating the model tensors. A 54-second delay strongly implies synchronous I/O blocking or severe memory swapping. The model.slm2 file, measuring 598,308,416 bytes, must not be eagerly loaded into resident memory. Instead, the implementation should employ virtual memory mapping (mmap). Memory mapping is particularly critical in WebAssembly environments, which impose strict linear memory addressing limits. Attempting to allocate a contiguous 598 MB buffer in a 32-bit WASM sandbox often triggers an out-of-bounds trap15.

Furthermore, to isolate the loading sequence, the execution must be confined to a single thread. Race conditions during the initialization of the simulated scalar accumulation logic can cause silent memory corruption. By setting torch.set\_num\_threads(1) and explicitly disabling CPU subnormal flushing (FTZ/DAZ)17, the environment is stabilized, ensuring that if a crash occurs, it is definitively a parsing or memory mapping error within the .slm2 format, rather than a transient thread starvation or hardware state issue. The offline local revision loading must utilize the rejected c1899de289a04d12100db370d81485cdf75e47ca revision as the diagnostic payload, confirming that the import mechanism properly isolates the artifact identity without conflating it with subsequent product qualification data.

Architectural Defect Classes and Implementation Nuances

When auditing the TinyRustLM implementation against the Qwen3 0.5B architecture, specific topological nuances must be meticulously verified. The following details rank the most plausible defect classes without assuming which currently exists locally.

Tied Embeddings and Tensor Orientation

The Qwen3 0.5B model utilizes tied word embeddings, a configuration explicitly marked by tie\_word\_embeddings: true in the public configuration2. In this architecture, the input embedding matrix, possessing the shape \[151936, 896\], is physically shared with the final language modeling head (lm\_head). A frequent implementation defect in low-level languages is allocating two separate physical buffers in memory. If the inference state machine mutates one buffer without mirroring the changes, or fails to properly transpose the memory stride when executing the final logit projection, the output distribution will catastrophically degrade. The physical tensor count inside the .slm2 file must be verified against the logical tensor count expected by the execution graph.

Grouped-Query Attention (GQA) and KV-Cache Layout

The attention mechanism employs Grouped-Query Attention with 14 query heads and only 2 key/value heads2. Consequently, the concatenated QKV projection matrix is asymmetric. During the scaled dot-product attention calculation, the Key and Value tensors must be logically repeated or physically broadcasted to match the 14 Query heads. A highly prevalent defect class involves misaligned memory access during this broadcasting. Native optimized Rust implementations generally favor a \[batch, head, seq, head\_dim\] memory layout to ensure contiguous memory access during the dot products, whereas standard model checkpoints often serialize weights in a \[batch, seq, head, head\_dim\] layout. Writing to the KV cache with transposed strides will result in random, high-magnitude activation values.

The Q/K Normalization Anomaly

A distinctive feature of the Qwen3 architecture, contrasting sharply with models like LLaMA, is the application of Root Mean Square Normalization (RMSNorm) to the queries and keys immediately after the linear projection but strictly before the Rotary Position Embeddings (RoPE)4. This normalization is applied exclusively over the head dimension, which is exactly 64 for this model configuration ([Figure omitted from source export]). Failing to apply q\_norm and k\_norm causes the attention softmax to become overly peaked, destroying the model's predictive entropy.

RoPE Position Indexing: Prefill versus Decode

Rotary Position Embeddings inject spatial structure using trigonometric absolute position indices. The implementation transitions between prefill and decode stages fundamentally alter this calculation28. During the prefill stage, a prompt of length [Figure omitted from source export] computes RoPE for positions [Figure omitted from source export] to [Figure omitted from source export] simultaneously using dense matrix operations. During the autoregressive decode stage, the newly generated token must compute RoPE precisely for position [Figure omitted from source export]. A pervasive defect class is off-by-one errors in decode position indexing. If the state machine miscalculates the offset, the attention mechanism misaligns the query's spatial rotation relative to the cached keys, resulting in immediate, unrecoverable autoregressive divergence.

Accumulation Order and IEEE-754 Strictness

The fundamental arithmetic of floating-point addition dictates that it is commutative but not associative. In strict IEEE-754 compliance, [Figure omitted from source export] is not necessarily equal to [Figure omitted from source export]. The project's scalar contract mandates explicit left-to-right accumulation. Standard pairwise summation, often utilized in BLAS libraries to bound numerical error to [Figure omitted from source export], will diverge from a linear loop, which accumulates error bounded to [Figure omitted from source export]24. The differential program must explicitly enforce a sequential loop in the Python reference to match the Rust execution behavior, preventing false positive divergence flags.

Error Metrics and Tolerance Setting Procedures

Applying a single, universal epsilon (e.g., [Figure omitted from source export]) across an entire transformer architecture is inadequate for differential debugging. Accumulation errors scale non-linearly with sequence length and the dimension of the hidden states. Neural network tensors frequently contain values clustered near zero where relative error calculations explode, alongside large outlier magnitudes where absolute error dominates. Therefore, a multi-faceted metric approach is required.

Metric Derivations

1. Maximum Absolute Error (MaxAE): [Figure omitted from source export]. This metric is essential for identifying catastrophic failures, such as NaN or Infinity values, or massive architectural routing bugs where a completely incorrect memory block is accessed.

2. Mean Relative Error (MRE): [Figure omitted from source export]. This metric evaluates precision decay over deep layers while ignoring near-zero values where floating-point noise is irrelevant.

3. Cosine Similarity: [Figure omitted from source export]. This is the most robust metric for intermediate high-dimensional neural activations. Vectors that maintain high cosine similarity (e.g., [Figure omitted from source export]) preserve their directional geometry within the latent space. This indicates that the model's semantic understanding remains completely intact, even if scalar magnitudes drift slightly due to discrepancies between FMA and non-FMA instructions.

4. Top-Token Margin: This metric evaluates the specific logit values of the top 5 predicted tokens in both execution arms. Divergent autoregressive paths only occur if the absolute ranking of the top token flips; minor fluctuations in the underlying logit values are harmless if the argmax remains stable.

Tolerance Table Justification

The following tolerance thresholds are proposals based on standard quantization error propagation arithmetic29. All tolerances assume a baseline comparison against Arm B (Emulated Quantized).

 

Layer-by-Layer Capture Plan and Trace Format

Differential debugging requires tapping the execution graph at deterministic intervals. Dumping every activation for every token across 24 layers of a 0.5B model generates highly redundant gigabytes of data. The capture plan utilizes a hierarchical bounded trace format.

Bounded Trace Format Strategy

To avoid floating-point rounding errors caused by JSON text serialization, all tensor captures must be serialized using the SafeTensors binary format. SafeTensors employs a well-defined binary structure: an initial 8-byte header specifies the length of a UTF-8 encoded JSON metadata section, followed directly by the raw, byte-aligned tensor buffers31. This format supports zero-copy memory mapping, making it optimal for rapid trace comparison.

When designing the comparison algorithm, full tensor slices (e.g., shape \[1, seq\_len, 896\]) are strictly necessary for the early layers (Layers 0 through 2\) during integration testing. However, for deep layers and ongoing regression testing, generating full slices is prohibitive. In these cases, cryptographic hashes of the raw byte arrays (e.g., BLAKE3) or targeted statistical summaries (L2 norm, Min, Max, NaN-count) are sufficient to prove ongoing mathematical stability.

Sequential Interception Nodes

The differential harness must capture the exact following states for a given layer\_idx. These nodes verify model-specific details against the exact public configuration of the Qwen3 architecture.

1. input\_ids: The exact integer tokens fed to the model embedding layer.

2. embed\_out: The output of the tied token embedding matrix27.

3. pre\_norm: The output of the RMSNorm applied prior to the attention mechanism.

4. qkv\_proj: The raw, pre-normalization outputs of the Query, Key, and Value linear projections.

5. qk\_norm: The specific outputs of the Qwen3 q\_norm and k\_norm normalizers4.

6. rope\_out: The Query and Key tensors immediately post-Rotary Position Embedding.

7. attn\_scores: The raw [Figure omitted from source export] matrix before causal masking and softmax.

8. attn\_mask: The causal lower-triangular boolean mask32.

9. attn\_probs: The post-softmax attention probability distributions.

10. attn\_out: The output of the Value matrix mixing and subsequent o\_proj.

11. post\_norm: The output of the RMSNorm applied prior to the MLP.

12. mlp\_gate & mlp\_up: The separated branches of the SwiGLU activation8.

13. mlp\_silu: The SiLU-activated gate branch.

14. mlp\_out: The combined output following the down\_proj.

15. final\_norm: The output of the terminal RMSNorm prior to the language head.

16. logits: The final vocabulary projection (shape \[1, seq\_len, 151936\])33.

Minimal Deterministic Diagnostic Traces

To isolate cache, index, and state-machine bugs, the differential harness must execute a matrix of specific edge-case traces. Free-running autoregressive generation allows minute errors to compound rapidly; therefore, all diagnostic traces must utilize teacher-forced inputs. This technique supplies the known sequence to the engine step-by-step, regardless of the model's actual token prediction, ensuring that layer [Figure omitted from source export] in both arms receives the exact same input state.

Prioritized Experiment Matrix

 

Required Decision Artifacts

Divergence-Localization Decision Tree

When the automated harness detects a divergence, engineers must execute the following state machine to systematically isolate the error without requiring access to the private Rust source code:

1. State 1: Verify Weight Extraction. Are the raw weights extracted from the Python reference bit-exact with the weights loaded by the Rust engine?

• No: Stop. The .slm2 file parser is misaligned, the endianness is flipped, or the byte offsets mapped in composition.acg2 are incorrect.
• Yes: Proceed to State 2\.

2. State 2: Verify Initial Embedding. Does Arm C (Scalar) match Arm B (Python) at the embed\_out stage?

• No: Stop. The token-to-row lookup logic is flawed, or the tied-embedding memory reference is corrupted.
• Yes: Proceed to State 3\.

3. State 3: Verify Linear Projections. Do the qkv\_proj outputs match exactly (0 ULP)?

• No: Stop. The left-to-right accumulation contract is violated. Inspect the inner matrix-multiplication loop for implicit FMA usage or pairwise summation optimizations.
• Yes: Proceed to State 4\.

4. State 4: Verify Attention Normalization. Does the attn\_probs (Softmax) stage diverge?

• No: Proceed to State 5\.
• Yes: Stop. Check if the unique q\_norm and k\_norm operations were correctly applied to the head dimension. Check for DAZ/FTZ denormal flushing hardware flags16.

5. State 5: Verify Autoregressive State. Does divergence only manifest when Sequence Length [Figure omitted from source export]?

• No: The bug is a general arithmetic defect present in prefill.
• Yes: Stop. This is a severe autoregressive KV cache misalignment. Verify that slot\_mapping and rotary\_position\_ids are incrementing correctly during the decode phase28.

Executable-Style Pseudocode Harness

The following schema defines the execution harness, bridging Python reference data to Rust structs for evaluation without exposing private code.

 

 

 

Python

\# differential\_harness.py import torch import safetensors.torch from qwen\_sim import Qwen3RowInt8Simulated

\# 1\. Enforce strict numerical determinism torch.use\_deterministic\_algorithms(True) torch.backends.cuda.matmul.allow\_tf32 \= False torch.manual\_seed(42) torch.set\_num\_threads(1)

def generate\_differential\_trace(model\_path, output\_path, tokens):     \# Initialize the simulated model from the proprietary format     model \= Qwen3RowInt8Simulated.from\_slm2(model\_path)     activations \= {}          \# Forward hook factory to capture exact tensor state     def hook\_fn(name):         def hook(module, input, output):             \# Capture as FP32, detached, forced to CPU to avoid device drift             activations\[name\] \= output.detach().cpu().to(torch.float32)         return hook

    \# Register hooks (Layer 0 subset utilized for minimal tracing)     model.layers\[0\].q\_proj.register\_forward\_hook(hook\_fn("L0\_q\_proj"))     model.layers\[0\].k\_norm.register\_forward\_hook(hook\_fn("L0\_k\_norm"))     model.layers\[0\].attn.softmax.register\_forward\_hook(hook\_fn("L0\_attn\_probs"))

    \# Execute trace under teacher-forced conditions     with torch.no\_grad():         logits \= model(torch.tensor(\[tokens\], dtype=torch.long))          activations\["final\_logits"\] \= logits.detach().cpu().to(torch.float32)

    \# Serialize to Bounded Trace Format     metadata \= {         "model\_architecture": "qwen3-0.5b",          "execution\_arm": "B\_emulated\_quantized",         "seq\_len": str(len(tokens))     }     safetensors.torch.save\_file(activations, output\_path, metadata=metadata)

\# Generate the baseline trace for the KV Transition hypothesis generate\_differential\_trace("D:\\\\LLMs\\\\TinyRustLM\\\\model.slm2",                              "E:\\\\Source\\\\Rust\\\\TinyRustLM.com\\\\traces\\\\trace\_L0\_kv.safetensors",                              \[151643, 856\])

Implementation Sequence and Project Directives

To rectify the current state of TinyRustLM and proceed to a stable prerelease, the engineering team must execute the following sequence.

Implementation Sequence

1. Halt Advanced Feature Work: Immediately cease development on custom peers, IP settings, and Hugging Face local-file imports. These are advanced features that distract from core inference stability.

2. Implement Network Override: Configure the Windows .NET companion and browser client to automatically fetch a known-working 0.5B default model payload from MiniModel.org upon initialization. This satisfies the non-negotiable user requirement to "open it and have a useful conversation" without complex setup.

3. Trace Generation: Execute the diagnostic matrix within the Python reference environment to generate the golden .safetensors trace files.

4. Integration Automation: Construct a Rust test suite that loads the .safetensors files, injects the identical input\_ids into the Rust scalar pipeline, and asserts the intermediate tensors against the Tolerance Table.

5. Iterative Resolution: Utilize the Divergence-Localization Decision Tree to walk backward from the final logits until the first layer of numerical divergence is pinpointed and patched.

Ship/No-Ship Criteria

• Ship: The MiniModel.org network download initializes successfully without user intervention. Arm C (Scalar Rust) produces exactly the top-1 logit predictions as Arm B (Python Reference) across a 512-token context window. Furthermore, the browser WASM target opens successfully and processes a prompt without triggering Out-of-Memory exceptions or WASM linear memory traps.
• No-Ship: Arm C diverges from Arm B by [Figure omitted from source export] Absolute Error in the early attention layers (prior to Layer 2). The application attempts to load the 598 MB payload synchronously into WASM linear memory, freezing the browser thread. No kernels may be optimized, no derivatives trained, and no numerical agreement claimed until Arm C achieves exact top-1 logit parity with Arm B.

Unresolved Local Measurements and Blind Spots

The primary blind spot in this analysis is the lack of visibility into the exact WebAssembly target compiler flags used in the E:\\Source\\Rust repository. If the Rust toolchain is configured to emit relaxed SIMD instructions, the WASM execution will silently drift from strict IEEE-754 compliance. An independent cross-check must be established by compiling a separate, minimal Rust binary using standard cargo build (without TinyRustLM's specific optimization flags) to execute the matrix multiplication, ensuring the divergence is an artifact of the codebase and not the local compiler environment.

What to Stop Doing

The engineering team must stop attempting to validate inference through structural inspection alone. A binary loading successfully into structs does not validate mathematical correctness. Second, stop maintaining historical and obsolete readers. If the custom .slm2 parser requires modifications to fix KV cache logic, the layout must be overwritten coherently; do not clutter the prerelease branch with legacy compatibility paths. Finally, stop using standard PyTorch FP16 inference for the baseline. Using full-precision targets to debug an INT8 implementation will result in perpetual ghost-chasing due to acceptable quantization noise.

Experiment-Lesson Template

To preserve closed evidence while authorizing new experiments after a verified fix, document all findings utilizing the following compact schema. This ensures lessons are retained compactly within the Git history.

• Question: Does the Qwen3 Q/K normalization occur before or after the linear projection?
• Exact Inputs: Single teacher-forced token ID \[151643\].
• Method: Intercept q\_proj output in Arm B. Compare directly with q\_norm output in Arm C.
• Result: Arm C absolute error diverges significantly to [Figure omitted from source export].
• Uncertainty: Could initially be caused by a scaling discrepancy in the RMSNorm epsilon ([Figure omitted from source export]).
• Decision: Qwen3 applies RMSNorm strictly after projection. The Rust implementation applied it before. Code modified to correct the topological order.
• Reusable Lesson: Qwen architectural topologies differ substantially from LLaMA in their attention pre-processing sequence. Assumptions regarding standard topological order cause catastrophic divergence.
• Evidence Identity: Commit c1899de289a04d12100db370d81485cdf75e47ca, Trace file: trace\_L0\_qnorm.safetensors.

Works cited

1. Qwen/Qwen2.5-0.5B-Instruct \- Hugging Face, https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct

2. config.json · Qwen/Qwen2.5-0.5B at, https://huggingface.co/Qwen/Qwen2.5-0.5B/blob/060db6499f32faf8b98477b0a26969ef7d8b9987/config.json

3. gated\_attention/configuration\_qwen3.py at main \- GitHub, https://github.com/qiuzh20/gated\_attention/blob/main/configuration\_qwen3.py

4. Text Capability Loss in Vision-Language Adaptation \- arXiv, https://arxiv.org/html/2609.00746v1

5. Qwen3 support please?? · Issue \#1902 · OpenNMT/CTranslate2, https://github.com/OpenNMT/CTranslate2/issues/1902

6. modeling\_qwen3\_moe.py \- huggingface/transformers \- GitHub, https://github.com/huggingface/transformers/blob/main/src/transformers/models/qwen3\_moe/modeling\_qwen3\_moe.py

7. Helion SwiGLU MLP Example, https://helionlang.com/examples/swiglu.html

8. SwiGLU in modern LLMs \- Sebastian Raschka, https://sebastianraschka.com/faq/docs/swiglu-modern-llms.html

9. PyTorch and TensorFloat32 \- NVIDIA CUDA, https://dev-discuss.pytorch.org/t/pytorch-and-tensorfloat32/504

10. CUDA semantics — PyTorch 2.14 documentation, https://docs.pytorch.org/docs/stable/notes/cuda.html

11. Give Me FP32 or Give Me Death?Challenges and Solutions ... \- arXiv, https://arxiv.org/html/2506.09501v1

12. Solving Reproducibility Challenges in Deep Learning and LLMs, https://www.ingonyama.com/oldblogs/solving-reproducibility-challenges-in-deep-learning-and-llms-our-journey

13. WebAssembly Runs Identically Everywhere, Except Where It Leaks, https://scrapfly.dev/posts/wasm-cpu-architecture-leak/

14. WebAssembly Specification, https://webassembly.github.io/spec/versions/core/WebAssembly-3.0-draft.pdf

15. rotifer-spec/rotifer-ir-specification.md at main \- GitHub, https://github.com/rotifer-protocol/rotifer-spec/blob/main/rotifer-ir-specification.md

16. 36821 – Flush denormals to Zero Flag, https://gcc.gnu.org/bugzilla/show\_bug.cgi?id=36821

17. CUDA Pro Tip: Flush Denormals with Confidence \- NVIDIA Developer, https://developer.nvidia.com/blog/cuda-pro-tip-flush-denormals-confidence/

18. Best-practices | Avoiding subnormal floating-point calculations, https://co-design.pop-coe.eu/best-practices/avoid-subnormals.html

19. Effectively Avoid Problems When Consuming Legacy Character, https://colinsblog.net/2023-10-21-nonstandard-string-encodings-rust/

20. bpe-tokenizer \- crates.io: Rust Package Registry, https://crates.io/crates/bpe-tokenizer

21. Efficient way for using int8 AVX512-VNNI instruction, especially, https://stackoverflow.com/questions/79265574/efficient-way-for-using-int8-avx512-vnni-instruction-especially-about-loading-t

22. How to implement an efficient \_mm256\_madd\_epi8 dot-products of, https://stackoverflow.com/questions/51382276/how-to-implement-an-efficient-mm256-madd-epi8-dot-products-of-groups-of-four-i8

23. 1-Bit LLMs in Pure Rust \- Naz Quadri, https://nazquadri.com/blog/everyday-thoughts/oxibonsai-1-bit-pure-rust/

24. Pairwise summation \- Wikipedia, https://en.wikipedia.org/wiki/Pairwise\_summation

25. GitHub \- xigh/herbert-rs: Local LLM inference engine written from, https://github.com/xigh/herbert-rs

26. Types — WebAssembly 1.1, https://webassembly.github.io/reference-types/core/syntax/types.html

27. Qwen2 \- Transformers \- Hugging Face, https://huggingface.co/docs/transformers/v4.40.0/model\_doc/qwen2

28. M-RoPE (Multimodal Rotary Position Embedding) \- AWS Neuron SDK, https://awsdocs-neuron.readthedocs-hosted.com/en/latest/vllm-neuron/docs/design/multimodal/mrope.html

29. Quantization Error Propagation: Revisiting Layer-Wise Post-Training, https://arxiv.org/html/2504.09629v3

30. Accurate Compensation for Quantized Large Language Models \- arXiv, https://arxiv.org/pdf/2507.11017

31. Gradient-Based Model Fingerprinting for LLM Similarity Detection, https://arxiv.org/html/2506.01631v2

32. Multi-Head Attention Fundamentals | rasbt/LLMs-from-scratch, https://deepwiki.com/rasbt/LLMs-from-scratch/4.1-multi-head-attention-fundamentals

33. Matrix-Driven Identification and Reconstruction of LLM Weight, https://arxiv.org/html/2508.06309v3

34. config.json · Qwen/Qwen3-8B at main \- Hugging Face, https://huggingface.co/Qwen/Qwen3-8B/blob/main/config.json