Producing One Greedy-Decoded Token Locally in C

Full report

On this page

Executive summary

For a local, offline, reproducible C# path that emits exactly one greedily decoded token, the most practical choice today is ONNX Runtime GenAI. It already exposes a C# API, loads a local model folder, includes tokenizer support, implements the generation loop, and handles logits processing, sampling/search, and KV-cache management. The project’s official support matrix lists C#, Windows/Linux/macOS, and multiple execution providers including CPU, CUDA, DirectML, OpenVINO, and others. The NuGet package is Microsoft.ML.OnnxRuntimeGenAI 0.14.1, and its package metadata shows compatibility with .NET 6+.

If your immediate goal is simply to prove the pipeline “prompt → encode → one forward decode step → argmax token → decoded token text” with no external API calls, an ONNX Runtime GenAI console app is the shortest route. If your deeper goal is to replace a llama.cpp dependency with a NuGet-centric C# runtime and inject low-level teleodynamic-style control, then ONNX Runtime GenAI is best treated as the near-term demonstration platform, not necessarily the end-state. Your uploaded design notes already point to the right long-range sequence: exact tokenizer parity, GGUF correctness, tensor binding, scalar forward pass, typed KV cache, deterministic greedy decode, then quantized CPU kernels and governed low-level control.

The key engineering distinction is this: if you only need a working local C# one-token decode today, use ONNX Runtime GenAI. If you need a true llama.cpp replacement with low-level inference control, you should build a managed runtime around local tokenizer parity + GGUF reading + your own forward pass/kernels, because LLamaSharp is still explicitly based on llama.cpp and ships backend packages that wrap native backends rather than replacing them.

Local approaches compared

The three most relevant local approaches for this problem are shown below. The first is the best fit for “working now in C# without API calls.” The second is the best fit for “I want lower-level control without llama.cpp.” The third is the best fit for “I need GGUF today and accept llama.cpp under the hood.”

If you specifically want a Hugging Face local-format route in .NET, the realistic options are either exporting to ONNX and using ONNX Runtime, or using a general tensor backend like TorchSharp. TorchSharp is explicitly documented as a .NET wrapper over the library that powers PyTorch, not as a turnkey Hugging Face causal-LM runtime; that means direct local Hugging Face checkpoint execution is possible in principle, but typically requires substantial model-specific loading and graph code. For your specific “one greedy token, low-level control, no llama.cpp” goal, that makes TorchSharp a research platform rather than the fastest production path.

Greedy decoding and tokenizer handling

Greedy decoding is mathematically simple: for the final-step logits vector \(z \in \mathbb{R}^{V}\) over vocabulary size \(V\), the selected token is

token* = argmax_i z_i

Because softmax is monotonic in each coordinate, the same token also maximizes the conditional probability:

token* = argmax_i softmax(z)_i

So greedy decoding for exactly one token means: encode the prompt, run one next-token forward step, read the final-step logits, take the argmax once, decode that token ID, and stop. Hugging Face’s generation guide states the same rule directly: greedy search chooses the next token with the highest probability at each step.

Tokenizer handling matters because “token string” is not always the same thing as a standalone human-readable string fragment. In byte-level BPE, the tokenizer can remap raw bytes to visible characters, which is why GPT-2-style tokenizers can represent arbitrary byte sequences without an unknown token. In SentencePiece/Llama-style tokenization, whitespace is preserved explicitly—typically through the ▁ metasymbol—so decoding must respect the tokenizer’s own decoder rules. That is why, in practice, you should use the model/tokenizer’s own stream decoder or full decoder rather than trying to interpret token IDs yourself.

Microsoft.ML.Tokenizers is useful here because it already ships tokenizer abstractions and implementations for BPE, Tiktoken, Llama, and Phi2, which makes it a good building block for a custom C# decode stack even if you eventually replace the model runtime underneath.

flowchart LR
    A[Load model or tokenizer from disk] --> B[Encode prompt to token IDs]
    B --> C[Run one next-token decode step]
    C --> D[Read final-step logits]
    D --> E[argmax over vocabulary]
    E --> F[Selected token ID]
    F --> G[Decode token piece with tokenizer decoder]
    G --> H[Print token ID and token text]

That flow is the same whether you use ONNX Runtime GenAI, a manual ONNX session, or your own future managed GGUF runtime. The only difference is who owns tokenization, logits extraction, and the generation loop.

Recommended minimal implementation with ONNX Runtime GenAI

Why this is the recommended path

ONNX Runtime GenAI is the cleanest answer to your stated requirements because it already provides the pieces you need in one C#-consumable package: local model load, tokenizer, generator, search options, next-token API, and token stream decode. The official repository describes it as implementing the generative AI loop for ONNX models, including pre/post-processing, inference with ONNX Runtime, logits processing, search and sampling, KV cache management, and grammar specification. The C# examples also show loading a model folder and using Generator, Tokenizer, and TokenizerStream locally.

Assumptions and packages

This report assumes Windows 10+, x64, and .NET 6+ as requested. The code below targets .NET 8 for simplicity, but the package metadata indicates compatibility with .NET 6 and newer.

Use this package:

A model folder is required, not just a single .onnx file. The official C# example README runs the executable with -m {path to model folder}, and the sample helper constructs new Config(path) for a “model folder containing GenAI config.”

Project file

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net8.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.ML.OnnxRuntimeGenAI" Version="0.14.1" />
  </ItemGroup>
</Project>

The package version and framework compatibility above come from the NuGet package metadata.

Minimal one-token greedy decode program

The code below performs exactly one next-token decode step. It is local-only, reads a model folder from disk, encodes the prompt, performs one greedy step by disabling sampling, then prints the token ID and the decoded token text. The important design choice is that it calls GenerateNextToken() once and only once. That makes the result “exactly one decoded token,” independent of longer generation settings. This usage matches the official generation loop shape shown in the ONNX Runtime GenAI examples.

using Microsoft.ML.OnnxRuntimeGenAI;

public static class Program
{
    public static int Main(string[] args)
    {
        try
        {
            if (args.Length < 1)
            {
                Console.Error.WriteLine("Usage: OneTokenOrtGenAI <modelFolder> [prompt]");
                return 2;
            }

            string modelFolder = args[0];
            string prompt = args.Length >= 2
                ? args[1]
                : "The capital of France is";

            if (!Directory.Exists(modelFolder))
            {
                Console.Error.WriteLine($"Model folder not found: {modelFolder}");
                return 2;
            }

            // Load the local model folder.
            using var config = new Config(modelFolder);

            // For reproducibility, clear provider overrides and let the default CPU path run.
            // If you want DirectML or CUDA later, configure that explicitly.
            config.ClearProviders();

            using var model = new Model(config);
            using var tokenizer = new Tokenizer(model);
            using var tokenizerStream = tokenizer.CreateStream();

            // Encode the prompt locally.
            var encodedPrompt = tokenizer.Encode(prompt);

            // Configure greedy decoding.
            var generatorParams = new GeneratorParams(model);
            generatorParams.SetSearchOption("do_sample", false);
            generatorParams.SetSearchOption("num_beams", 1);

            using var generator = new Generator(model, generatorParams);

            // Append the prompt and capture the token count before generation.
            generator.AppendTokenSequences(encodedPrompt);
            ulong promptTokenCount = generator.TokenCount();

            // Exactly one decoding step.
            generator.GenerateNextToken();

            int nextTokenId = generator.GetNextTokens()[0];
            string nextTokenText = tokenizerStream.Decode(nextTokenId);
            ulong totalTokenCount = generator.TokenCount();

            if (totalTokenCount != promptTokenCount + 1)
            {
                throw new InvalidOperationException(
                    $"Expected exactly one new token. Before={promptTokenCount}, After={totalTokenCount}.");
            }

            Console.WriteLine($"Prompt: {prompt}");
            Console.WriteLine($"PromptTokenCount: {promptTokenCount}");
            Console.WriteLine($"NextTokenId: {nextTokenId}");
            Console.WriteLine($"NextTokenText: [{EscapeVisible(nextTokenText)}]");
            Console.WriteLine($"TotalTokenCount: {totalTokenCount}");

            return 0;
        }
        catch (Exception ex)
        {
            Console.Error.WriteLine("Greedy one-token decode failed.");
            Console.Error.WriteLine(ex);
            return 1;
        }
    }

    /// <summary>
    /// Renders whitespace and control characters visibly so token text can be inspected safely.
    /// </summary>
    /// <param name="value">The decoded token text.</param>
    /// <returns>A console-safe representation of the token text.</returns>
    private static string EscapeVisible(string value)
    {
        return value
            .Replace("\\", "\\\\", StringComparison.Ordinal)
            .Replace("\r", "\\r", StringComparison.Ordinal)
            .Replace("\n", "\\n", StringComparison.Ordinal)
            .Replace("\t", "\\t", StringComparison.Ordinal)
            .Replace(" ", "␠", StringComparison.Ordinal);
    }
}

Build and run

A minimal offline build sequence on Windows looks like this:

dotnet new console -n OneTokenOrtGenAI
cd OneTokenOrtGenAI
dotnet add package Microsoft.ML.OnnxRuntimeGenAI --version 0.14.1
# Replace the generated .csproj and Program.cs with the versions above.
dotnet build -c Release
dotnet run -c Release -- "C:\Models\YourOnnxGenAIModelFolder" "The capital of France is"

This stays fully local at runtime as long as the model folder is already on disk and your code does not download anything. Official examples likewise run against a local model folder path.

What to expect

A real run will depend on the prompt and the exact ONNX model pack you place on disk, so the specific token ID and text are model-dependent. The structure of the output should look like this:

Prompt: The capital of France is
PromptTokenCount: 6
NextTokenId: 12345
NextTokenText: [␠Paris]
TotalTokenCount: 7

What matters for correctness is not a hard-coded token ID, but the invariants below. Those invariants are stable across models if the pipeline is implemented correctly.

How to verify correctness

The fastest verification checklist is:

1. Token count increases by exactly one after the single call to GenerateNextToken().
2. Repeated runs with the same prompt and the same local model files produce the same NextTokenId when do_sample=false.
3. The NextTokenText should be decoded via the tokenizer’s decoder, not by naïvely converting the integer token ID to text yourself.
4. If you instrument a longer loop, the first emitted token should match the first token from that longer greedy generation under the same settings.

A simple xUnit-style verification pattern is:

// Illustrative test skeleton.
Assert.Equal(promptTokenCount + 1, totalTokenCount);
Assert.False(string.IsNullOrEmpty(nextTokenText));

The first assertion checks that your session really produced one new token. The second checks that the decode path is wired correctly. For tokenizer families that preserve whitespace or byte-level structure, seeing visible leading-space markers in your console-safe rendering is normal.

Deterministic tokenizer-only harness

If you want a tiny, deterministic proof of the encode/argmax/decode mechanics without carrying a full model folder, use a local tokenizer file and feed a synthetic logits vector into an explicit ArgMax. This is not a neural forward pass, but it proves the exact mechanics of tokenization + greedy token selection + decoded token text, which is often the cleanest first unit test when building your own runtime. Microsoft.ML.Tokenizers is well suited here because it exposes tokenizer abstractions and includes a Llama tokenizer implementation.

Package

Program

This program loads a local Llama tokenizer.model, encodes a prompt, creates a deterministic synthetic logits vector whose argmax is the prompt’s final token ID, and then decodes that chosen token. Because the token ID is taken from the tokenizer’s own output, the test is tokenizer-self-consistent and reproducible. LlamaTokenizer.Create(stream) is shown in the package documentation, and EncodeToIds is the standard encode path.

using Microsoft.ML.Tokenizers;

public static class Program
{
    public static int Main(string[] args)
    {
        try
        {
            if (args.Length < 1)
            {
                Console.Error.WriteLine("Usage: OneTokenTokenizerHarness <tokenizer.model> [prompt]");
                return 2;
            }

            string tokenizerPath = args[0];
            string prompt = args.Length >= 2
                ? args[1]
                : "Hello, world!";

            if (!File.Exists(tokenizerPath))
            {
                Console.Error.WriteLine($"Tokenizer file not found: {tokenizerPath}");
                return 2;
            }

            using var stream = File.OpenRead(tokenizerPath);
            Tokenizer tokenizer = LlamaTokenizer.Create(stream);

            IReadOnlyList<int> promptIds = tokenizer.EncodeToIds(prompt);
            if (promptIds.Count == 0)
            {
                throw new InvalidOperationException("Tokenizer returned no prompt IDs.");
            }

            // Use the prompt's final token ID as the forced argmax so the test stays
            // self-consistent with the tokenizer on disk.
            int forcedTokenId = promptIds[^1];
            float[] logits = CreateSyntheticLogits(forcedTokenId);

            int greedyTokenId = ArgMax(logits);
            string greedyTokenText = tokenizer.Decode(new[] { greedyTokenId });

            Console.WriteLine($"Prompt: {prompt}");
            Console.WriteLine($"PromptIds: {string.Join(", ", promptIds)}");
            Console.WriteLine($"GreedyTokenId: {greedyTokenId}");
            Console.WriteLine($"GreedyTokenText: [{EscapeVisible(greedyTokenText)}]");

            return 0;
        }
        catch (Exception ex)
        {
            Console.Error.WriteLine("Tokenizer harness failed.");
            Console.Error.WriteLine(ex);
            return 1;
        }
    }

    /// <summary>
    /// Creates a deterministic logits vector whose argmax is the specified token ID.
    /// </summary>
    /// <param name="tokenId">The token ID that should win the greedy selection.</param>
    /// <returns>A logits vector with a unique maximum at <paramref name="tokenId"/>.</returns>
    private static float[] CreateSyntheticLogits(int tokenId)
    {
        if (tokenId < 0)
        {
            throw new ArgumentOutOfRangeException(nameof(tokenId));
        }

        var logits = new float[tokenId + 1];
        for (int i = 0; i < logits.Length; i++)
        {
            logits[i] = float.NegativeInfinity;
        }

        logits[tokenId] = 0.0f;
        return logits;
    }

    /// <summary>
    /// Returns the index of the maximum logit.
    /// </summary>
    /// <param name="logits">The logits vector to scan.</param>
    /// <returns>The index of the largest logit value.</returns>
    private static int ArgMax(ReadOnlySpan<float> logits)
    {
        if (logits.Length == 0)
        {
            throw new ArgumentException("Logits must not be empty.", nameof(logits));
        }

        int bestIndex = 0;
        float bestValue = logits[0];

        for (int i = 1; i < logits.Length; i++)
        {
            if (logits[i] > bestValue)
            {
                bestValue = logits[i];
                bestIndex = i;
            }
        }

        return bestIndex;
    }

    /// <summary>
    /// Makes whitespace and control characters visible in console output.
    /// </summary>
    /// <param name="value">The decoded token text.</param>
    /// <returns>A console-safe representation of the token text.</returns>
    private static string EscapeVisible(string value)
    {
        return value
            .Replace("\\", "\\\\", StringComparison.Ordinal)
            .Replace("\r", "\\r", StringComparison.Ordinal)
            .Replace("\n", "\\n", StringComparison.Ordinal)
            .Replace("\t", "\\t", StringComparison.Ordinal)
            .Replace(" ", "␠", StringComparison.Ordinal);
    }
}

What this harness proves

This harness proves four things cleanly:

• the tokenizer loads correctly from local disk,
• the prompt encodes to token IDs,
• your greedy selector chooses the maximum logit,
• the selected token ID decodes back through the tokenizer’s decoder.

That makes it an excellent first test before you wire in a real forward pass. It is especially useful if you are building a custom GGUF or ONNX execution path and want to isolate tokenizer correctness from model correctness. Your uploaded notes emphasize exactly this kind of staged parity work.

Roadmap to a C# llama.cpp replacement

What the current ecosystem implies

If the downstream program currently depends on a llama.cpp path, then LLamaSharp is not a replacement, only a C# wrapper over that ecosystem. Its own README says it is based on llama.cpp, and it exposes backend packages rather than a fully managed inference core. That makes it useful operationally, but it does not satisfy your stated architectural goal of replacing the llama.cpp dependency path and adding deeper control inside inference itself.

By contrast, the ONNX stack gets you out of the llama.cpp dependency tree quickly, but only if you are willing to standardize on ONNX model packs instead of GGUF. That is a valid product decision, but it is not the same thing as “a GGUF-native, pure-C# replacement for llama.cpp.” The latter requires a tokenizer-and-kernels project of your own. Your uploaded notes already frame that correctly: tokenizer parity first, then GGUF correctness, then the execution ladder up through deterministic one-token decode.

Practical staged plan

From where you are, the most effective sequence is:

Where to inject low-level control

If your actual differentiator is “teleodynamic-ish strategies at a very low level,” then the best control points are not at the prompt layer. They are at:

• tokenizer parity and normalization,
• logits post-processing before argmax/sampling,
• KV-cache admission/eviction,
• scheduler/batch admission,
• quantized kernel selection and fallback, and
• parity mode vs governed mode.

That last point matters. Your uploaded notes argue for separating deterministic parity mode from an adaptive governed mode. That is exactly right. If you try to inject adaptive control while also validating numerical parity for one-token greedy decode, you will make your own correctness work much harder. Build the single-token greedy oracle first, then add higher-level governance once the deterministic path is stable.

Security, privacy, and correctness checks

Local inference improves privacy primarily because the prompt, tokenizer assets, and weights stay on the local machine and the runtime can run fully on-device. ONNX Runtime GenAI explicitly describes itself as a way to run generative models on device, and its official examples are local-folder based rather than API based. That said, “local” is not the same thing as “safe by default”: you are still loading large untrusted files into a runtime that may contain native code.

For secure local use, the practical baseline is:

• pin exact NuGet versions,
• verify model file hashes before first use,
• store models in a read-only directory,
• disable any auto-download behavior in your own application,
• avoid logging raw prompts or decoded tokens if they may contain secrets,
• and do not assume a quantized model is semantically benign merely because it is “just weights.” A 2025 security paper showed practical attacks on GGUF quantization, demonstrating that malicious behaviors can survive in quantized artifacts.

If you stay in the GGUF/llama.cpp orbit at all, remember that LLamaSharp backend compatibility and model/backend version alignment are operationally important. The project documentation explicitly warns about native backend compatibility and model-file compatibility issues. That is another reason your eventual custom runtime should start with a small deterministic target—exactly one greedy token—before expanding to broader generation behavior.

Finally, correctness for this problem should be defined narrowly and rigorously:

• same prompt + same local files + greedy mode = same first token,
• one GenerateNextToken() call = one new token, and
• decoded token text must come from the tokenizer’s own decoder path because byte-level and SentencePiece tokenizers preserve whitespace and bytes in tokenizer-specific ways.

Your uploaded notes are directionally right: the shortest proof is one locally decoded greedy token; the real destination is a managed C# runtime whose deepest control surfaces are the tokenizer, logits, KV cache, and kernels—not the chat wrapper.