Exact Tokenizer Parity for GGUF LLaMA Family Models in Pure C

Full report

On this page

Executive summary

If the goal is to replace llama.cpp tokenization in a pure C# NuGet package, the critical design decision is to treat GGUF tokenizer metadata as a full execution contract, not as a hint. The load-bearing fields are the token list itself, optional per-token scores and types, special-token IDs, BPE merges where applicable, plus the llama.cpp-specific tokenizer extras such as tokenizer.ggml.pre, tokenizer.ggml.add_bos_token, tokenizer.ggml.add_eos_token, tokenizer.ggml.add_space_prefix, tokenizer.ggml.remove_extra_whitespaces, and tokenizer.ggml.precompiled_charsmap. The GGUF core docs standardize the base tokenizer arrays and tokenizer.huggingface.json, while current llama.cpp extends that metadata surface in its loader and architecture-key registry. Exact parity therefore requires implementing both the standardized GGUF fields and the llama.cpp extensions that real GGUF files depend on today.

A second non-obvious point is that “SentencePiece” is not one thing in practice. In current llama.cpp, LLAMA_VOCAB_TYPE_SPM is the LLaMA tokenizer path, described in the public header as a tokenizer “based on byte-level BPE with byte fallback,” while LLAMA_VOCAB_TYPE_UGM is the Unigram path used for T5-like models. SentencePiece’s own model proto supports both BPE and UNIGRAM, and also stores byte_fallback, normalization settings, special IDs, piece scores, and piece types. So the C# replacement should not implement a single generic “SentencePiece tokenizer”; it should implement at least three concrete engines: SentencePiece-BPE for LLaMA-style tokenizers, SentencePiece-Unigram for UGM/T5-like tokenizers, and GPT-2 byte-level BPE for gpt2-class GGUFs.

The fastest route to parity is a layered architecture. First, implement a strict GGUF tokenizer descriptor loader and validator. Second, implement the three tokenization engines with deterministic encode and decode semantics. Third, add a llama.cpp oracle harness that exercises llama_tokenize, llama_detokenize, llama_vocab_get_text, llama_vocab_get_score, llama_vocab_get_attr, and special-token getters against the same GGUF. Fourth, lock behavior with golden JSON fixtures and a corpus that stresses whitespace, Unicode normalization, special tokens, combining marks, byte fallback, and decode quirks. The public llama.cpp API already exposes essentially everything you need for an oracle.

The largest current risk to correctness is assuming that tokenizer.huggingface.json can replace the GGUF arrays outright. It cannot. The model-facing token IDs are defined by the GGUF token array order, and the GGUF docs require scores and token types, when present, to align one-to-one with that array. The embedded Hugging Face JSON is valuable as a compatibility and reconstruction aid for the normalizer, pre-tokenizer, decoder, and added-token behavior, but it should be treated as a secondary pipeline description that must be validated against the GGUF array-derived vocabulary order before it is trusted as the runtime source of token IDs. That recommendation is partly an inference, but it is strongly supported by the GGUF spec, Hugging Face’s “full pipeline configuration” model, and the known history of old GGUF tokenizer mismatches in llama.cpp.

Ground truth contracts in GGUF and llama.cpp

GGUF’s tokenizer section gives the baseline contract: tokenizer.ggml.model, tokenizer.ggml.tokens, optional tokenizer.ggml.scores, optional tokenizer.ggml.token_type, optional tokenizer.ggml.merges, optional tokenizer.ggml.added_tokens, and the standard special-token IDs such as BOS, EOS, UNK, SEP, and PAD. The spec explicitly states that scores and token_type, when present, must be the same length and indexing as tokens. It also allows embedding the entire Hugging Face tokenizer.json as tokenizer.huggingface.json.

llama.cpp extends this contract materially. Its architecture key registry contains tokenizer metadata keys for tokenizer.ggml.pre, tokenizer.ggml.add_bos_token, tokenizer.ggml.add_eos_token, tokenizer.ggml.add_sep_token, tokenizer.ggml.add_space_prefix, tokenizer.ggml.remove_extra_whitespaces, tokenizer.ggml.precompiled_charsmap, tokenizer.huggingface.json, tokenizer.chat_template, normalizer flags, FIM token IDs, and more. Real issue logs from current models show these fields appearing in production GGUFs. For a C# replacement that aims to match llama.cpp, these fields are not optional niceties; they are part of the de facto tokenizer ABI.

A practical way to think about precedence is this: the GGUF arrays define the on-disk vocabulary identity, the llama.cpp extra keys define runtime behavior, and the embedded Hugging Face tokenizer JSON describes the original tokenization pipeline. If all three are present, the safest approach is to treat GGUF token IDs as authoritative, use llama.cpp-style extra keys to reproduce runtime behavior, and use tokenizer.huggingface.json to fill in behavior that GGUF alone cannot express exactly, especially for added-token flags or unusual decoder/pre-tokenizer pipelines. That last sentence is an inference, but it is the most defensible parity strategy from the available sources.

GGUF metadata fields versus concrete runtime usage

One subtle but important point is that the core GGUF docs do not mention tokenizer.ggml.pre, but the current llama.cpp loader and converter ecosystem clearly depend on it for many modern BPE-family models. That means a “GGUF-only” tokenizer implementation that ignores llama.cpp’s extra tokenizer keys will still fail parity against real-world GGUFs.

Algorithms required for exact parity

Tokenizer family feature matrix

SentencePiece-BPE for LLaMA style models

For LLaMA-family parity, the safest mental model is SentencePiece packaging, BPE segmentation, byte fallback, and special whitespace semantics. The Hugging Face LLaMA docs describe the tokenizer as a byte-pair encoding model based on SentencePiece, using ByteFallback and no normalization, and llama.cpp’s public header describes its SPM vocabulary type as the LLaMA tokenizer based on byte-level BPE with byte fallback. SentencePiece’s own proto supports model_type = BPE and byte_fallback = true.

In current llama.cpp, raw input is first split into fragments so that special tokens can be handled distinctly. The implementation builds a cache of special tokens out of control, user-defined, and unknown-type tokens, sorts them longest-first by text length, and then partitions the input accordingly. When parse_special == false, control and unknown tokens are ignored as special-token candidates, but user-defined tokens are still pre-tokenized first. That behavior is easy to miss, and omitting it will create parity drift on models with user-defined sentinels or template tokens.

For the SPM path specifically, llama.cpp prefixes a literal space onto a raw fragment when add_space_prefix is enabled and the previous fragment was special or the fragment is first, then escapes whitespace to the SentencePiece metaspace symbol before tokenization. The current code also documents the historic empty-string behavior: encoding "" with special tokens enabled returns a BOS-only sequence, while encoding without special tokens returns an empty sequence. BOS and EOS insertion then follow the model-level flags rather than a generic library default.

The internal SPM tokenizer path in llama.cpp is implemented as a merge procedure over UTF-8 character symbols: split input into UTF-8 code-point-sized symbols, seed a priority queue with all adjacent bigrams, repeatedly merge the highest-scoring bigram, then resegment the merged output; if a final symbol does not correspond to a token and cannot be decomposed through the reverse-merge map, it falls back to byte emission. That is not the same as naively replaying Hugging Face fast-tokenizer behavior; for exact parity, you should mirror the on-disk token texts, scores, reverse-merge handling, and byte-token path.

SentencePiece-Unigram for UGM models

The Unigram path is significantly different and should live in its own engine. SentencePiece’s proto stores piece scores, piece types, normalizer settings, and special IDs directly in the model, and llama.cpp’s UGM path is explicitly documented as the T5-style Unigram tokenizer.

The current llama.cpp implementation states that the UGM session is based on SentencePiece’s optimized Viterbi algorithm. It first normalizes the input, then performs dynamic programming over a trie of candidate pieces. It uses double score accumulation “to make tokenization results exactly the same as in the HF tokenizer using SentencePiece,” assigns user-defined tokens a score of 0.0 to make them more likely to be selected, and scores unknown-token fallback as min_score - 10. During backtracking it merges consecutive unknown pieces into a single UNK token. Those are parity-critical details; if you store scores in float everywhere or if you emit one UNK per code point, you will diverge.

Normalization also matters more than most ports expect. SentencePiece’s NormalizerSpec includes precompiled_charsmap, add_dummy_prefix, remove_extra_whitespaces, and escape_whitespaces, while TrainerSpec also includes treat_whitespace_as_suffix. llama.cpp’s UGM normalizer logic uses those semantics directly when constructing normalized text, including whether space is prepended or appended and whether repeated spaces are merged. Exact parity therefore requires an implementation of the charsmap transform, not just basic .NET Unicode normalization.

GPT-2 byte-level BPE

For gpt2-class GGUF tokenizers, the irreducible contract is the OpenAI byte-to-unicode mapping, ranked merge list, and pre-tokenizer regex. OpenAI’s original encoder.py defines bytes_to_unicode(), the canonical GPT-2 regex, rank-ordered BPE merges, and byte-level decode back to UTF-8. Hugging Face’s GPT-2 tokenizer docs and implementation preserve the same basic behavior and note that leading spaces alter tokenization unless add_prefix_space=True is used.

In llama.cpp, however, BPE parity is not just “GPT-2 default regex forever.” The project now records a pre-tokenizer identity into GGUF via tokenizer.ggml.pre, and the update script describes this as a hash-derived identifier intended to let llama.cpp implement the same pre-tokenizer as the upstream model. The architecture key registry includes LLM_KV_TOKENIZER_PRE, and issue history shows real GGUFs carrying values like phi2, codeqwen, dbrx, gpt-4o, and deepseek-r1-qwen. A C# replacement that only implements the OpenAI default regex will match some GPT-2-derived models and silently fail many newer ones.

That leads to a concrete implementation requirement: represent tokenizer.ggml.pre as an explicit registry of pre-tokenizer families. For each family, store at minimum the regex set, whether byte encoding is on or off, whether metaspace escaping is used, and any unusual added-token matching behavior. The current llama.cpp source includes multiple regex families and explicitly toggles whether BPE uses GPT-2 byte encoding or raw UTF-8 / SPM-style whitespace semantics.

Suggested tokenizer-flow diagram

flowchart TD
    A[Load GGUF metadata] --> B[Build TokenizerDescriptor]
    B --> C{tokenizer.huggingface.json present?}
    C -->|yes| D[Parse HF pipeline as secondary contract]
    C -->|no| E[Use GGUF arrays and llama.cpp extras only]
    D --> F[Validate HF vocab order against tokenizer.ggml.tokens]
    E --> G[Resolve tokenizer family]
    F --> G
    G --> H{llama / replit / ugm / gpt2}
    H -->|llama| I[SentencePiece-BPE engine]
    H -->|ugm| J[SentencePiece-Unigram engine]
    H -->|gpt2| K[GPT2 byte-level BPE engine]
    I --> L[Special-token partition]
    J --> L
    K --> L
    L --> M[Normalize / pre-tokenize]
    M --> N[Segment to token IDs]
    N --> O[Apply addBos/addEos/addSep]
    O --> P[Detokenize via exact decoder path]

Decoding, Unicode, and special-token edge cases

Decoding parity is where many almost-correct ports fail. In llama.cpp, token_to_piece and detokenize treat token families differently. For WPM, SPM, and UGM, normal tokens are decoded by unescaping SentencePiece whitespace; byte tokens decode to a single raw byte; and special or user-defined tokens can be rendered literally when requested. For BPE, normal tokens either use the same metaspace unescape path when escape_whitespaces is enabled or use ordinary byte-level decode when it is not. On top of that, llama_detokenize has a leading-space removal rule tied to add_space_prefix, and optionally strips BOS and EOS before rendering.

This matches two public-document quirks. SentencePiece documents the metaspace mechanism explicitly: whitespace is escaped to ▁ and detokenization is formed by joining pieces and replacing ▁ with a literal space. Hugging Face’s LLaMA docs also note a specific decode quirk: if the first token is the start of a word, decoding does not prepend the prefix space. That quirk maps closely to the leading-space behavior visible in llama.cpp’s detokenizer.

Unicode normalization needs particular discipline in C#. .NET gives you string.Normalize for NFC, NFD, NFKC, and NFKD, and Rune for Unicode-scalar-safe iteration; those are useful building blocks, but they are not sufficient on their own for SentencePiece parity because SentencePiece stores a compiled charsmap automaton and whitespace flags in the model itself. A good rule is: use .NET normalization only when the tokenizer contract explicitly asks for a standard normalization form; use a dedicated SentencePiece-charsmap implementation when precompiled_charsmap is present.

You should also model richer token attributes internally than the base GGUF docs currently standardize. llama.cpp exposes token attrs for UNKNOWN, UNUSED, NORMAL, CONTROL, USER_DEFINED, BYTE, plus NORMALIZED, LSTRIP, RSTRIP, and SINGLE_WORD. Even if a given GGUF only carries token_type today, your in-memory token descriptor should be able to represent the richer attribute surface so that tokenizer.huggingface.json and future GGUF revisions can populate it without redesign. That recommendation is forward-looking, but it follows directly from the public llama.cpp header and Hugging Face’s explicit added-token and pipeline-component model.

Security validation is not optional here. There is a published llama.cpp security advisory for a tokenizer overflow bug, and a recent SentencePiece crash disclosure related to malformed precompiled_charsmap parsing. A pure C# implementation should therefore reject malformed metadata aggressively: mismatched array lengths, out-of-range special IDs, invalid merge references, malformed UTF-8 in token text, and any charsmap blob whose trie walks or offsets exceed bounds. Exact parity is the goal, but exactly reproducing memory-unsafe behavior is not.

Parity harness and golden corpus design

The cleanest oracle is a tiny native test helper linked against llama.cpp, not the normal CLI. Use the public API to load the GGUF, obtain the llama_vocab, call llama_tokenize and llama_detokenize, and dump per-token text, score, attr, special IDs, and add-BOS/add-EOS flags as JSON. The public header exposes all of these. It also documents useful buffer-sizing semantics: llama_tokenize returns a negative required size if the buffer is too small, and llama_token_to_piece can do the same for decode buffers.

A robust fixture format should include three layers of truth. The first is model metadata truth: selected GGUF tokenizer keys, model file hash, and vocab hash. The second is behavioral truth: encode, decode, token_to_piece, and token-attribute dumps. The third is message-level truth: rendered chat template text and message token counts. Even if your initial scope is only encode/decode, storing the message layer from the start prevents spending weeks later rediscovering that “token count parity” fails because the template already contained BOS/EOS markers and your tokenizer added them again. Hugging Face’s chat-template docs explicitly warn that adding special tokens again after templating is often incorrect.

The golden corpus should be small enough to run per commit and rich enough to catch the real failure modes. At minimum, include: plain ASCII; leading-space versus no-leading-space variants; tabs and newlines; repeated spaces; punctuation and contractions; CJK without spaces; Arabic/Hebrew; combining-mark pairs in NFC and NFD/NFKC-sensitive forms; emoji and other astral characters; control-looking strings such as <s> and <|eot_id|> under both parse_special=false and parse_special=true; and text that forces byte fallback. SentencePiece’s own design docs and Hugging Face’s ByteLevel docs make clear why these cases matter.

Smoke-test corpus examples with published expected IDs

These are useful sanity fixtures, not sufficient acceptance fixtures. Real acceptance still needs model-specific GGUF corpora.

A useful fixture schema for your C# harness would look like this in spirit:

{
  "model": {
    "gguf_sha256": "…",
    "tokenizer_model": "llama",
    "tokenizer_pre": "default",
    "vocab_hash": "…"
  },
  "cases": [
    {
      "name": "leading-space",
      "text": " Hello world",
      "addSpecial": true,
      "parseSpecial": false,
      "expectedIds": [18435, 995],
      "expectedDecoded": " Hello world"
    }
  ],
  "vocabSpotChecks": [
    {
      "id": 1,
      "text": "<s>",
      "score": 0.0,
      "attr": ["CONTROL"]
    }
  ]
}

For seeds, the tokenizer layer itself should remain seed-free. If you also build end-to-end generation-parity tests, keep a seed field in the schema for consistency, but for tokenizer-only cases it should have no semantic effect. For generation parity, prefer greedy / temperature-zero paths and still persist the seed field so the fixture schema is stable even when stochastic samplers are exercised later. That is an engineering recommendation rather than a direct requirement from the cited sources.

Suggested parity-pipeline diagram

flowchart LR
    A[GGUF model] --> B[Native llama.cpp oracle]
    A --> C[Pure C# tokenizer]
    B --> D[JSON oracle dump]
    C --> E[JSON managed dump]
    D --> F[Comparator]
    E --> F
    F --> G{Match?}
    G -->|yes| H[Green parity report]
    G -->|no| I[Diff by case, token index, piece text, score, attr]
    I --> J[Golden corpus update or bug fix]

C# implementation and integration in the uploaded repo

From the uploaded source tree, the immediate repo-level blockers are clear. The abstraction surface is already good: you have ITokenizer, IInferenceRuntime, IInferenceSession, IModelAdapter, a GGUF reader, and a tokenizer factory seam. But the concrete tokenizer types in Uai.LlmRuntime.Tokenization still route through WhitespaceTokenizer, the GGUF factory currently returns stubs, and the embedded tokenizer.huggingface.json path is effectively a placeholder. In other words, the project already has the right seams, but not yet the parity engine behind them.

The first structural change I would make is to introduce a single immutable TokenizerDescriptor built from GGUF. It should carry: token texts as raw UTF-8 slices, scores, token kinds, richer attr flags, special-token IDs, add/remove flags, tokenizer.ggml.pre, merges, added-token metadata, charsmap bytes, and optionally the parsed HF tokenizer pipeline. That descriptor should be the only object allowed to instantiate a concrete tokenizer engine. This prevents “half GGUF, half ad hoc defaults” drift.

The second change is to keep the vocabulary in a memory-efficient representation. On .NET, memory-mapped files are a good fit for GGUF-scale data because they support random access over large files without copying the whole artifact into memory, and Span<T> / ReadOnlySpan<T> are ideal for zero-allocation slicing over mapped views or owned buffers. Because Span<T> is stack-only and cannot cross await boundaries, the tokenizer engines themselves should be synchronous, CPU-local components that the async orchestration layer calls from PrefillAsync or model-loading code, not from arbitrary async continuations that try to retain spans.

For Unicode handling, use Rune for scalar-safe traversal whenever the algorithm is defined over Unicode scalars rather than UTF-16 code units. This is especially important for golden corpus diagnostics and any UTF-8/UTF-16 boundary code. However, for exact compatibility with OpenAI GPT-2 byte-level BPE or SentencePiece byte fallback, keep a separate byte-oriented path that never round-trips through UTF-16 unless the algorithm explicitly requires it. Rune helps you stay correct on the Unicode side; it should not tempt you into erasing the byte-level nature of GPT-2-style tokenization.

A practical engine split for the repo would look like this:

• SentencePieceBpeTokenizerEngine
• SentencePieceUnigramTokenizerEngine
• Gpt2BpeTokenizerEngine
• SpecialTokenPartitioner
• Detokenizer
• HuggingFaceTokenizerJsonParser
• TokenizerParityOracle
• TokenizerGoldenCorpus

Wire those behind the existing ITokenizer interface, and make CountTokens(IEnumerable<LlmMessage>) depend on an explicit chat-template rendering stage followed by exact tokenization. Otherwise you will get correct text encode/decode and still have wrong message budgeting. Hugging Face’s chat-template docs are explicit that the template often already contains what would otherwise be “special tokens,” so templating and tokenization cannot be treated as independent defaults.

There is one repo-specific bug worth fixing very early: the uploaded LlamaModelConfig.FromGguf appears to read "tokenizer.ggml.tokens" as though it were a scalar metadata value. In GGUF, that field is an array of strings and its length is the vocab size. Fixing that early will prevent the config layer from silently disagreeing with the tokenizer layer.

Prioritized PRs and minimal acceptance tests

Recommended PR sequence

Minimal acceptance tests that should gate merge

The practical finish line is not “the tokenizer returns plausible IDs.” The finish line is this: for every supported GGUF tokenizer family, your C# runtime and llama.cpp produce the same IDs for the same text, the same text for the same IDs, the same handling of BOS/EOS/special parsing, and the same token metadata for representative vocab entries. Once that is true, replacing llama.cpp tokenization becomes a tractable engineering problem instead of a permanent source of off-by-one and mystery-regression bugs.