Binding All Required Tensors for a Model Runtime

Full report

On this page

Executive summary

Implementing step 3, “bind all required tensors,” is the point where a parsed model becomes executable. In practice, a good binder does four jobs: it enumerates the exact tensors the runtime needs, validates each tensor’s semantic contract (name, shape, dtype, layout, quantization, and device), chooses how each tensor’s memory is owned and placed, and then exposes stable handles that the execution engine can reuse without re-discovery on every inference. For most runtimes, weights and persistent state are bound once at model/session creation, while inputs, outputs, and dynamic state such as KV cache are rebound or refreshed per run. Ephemeral intermediates are usually not user-bound unless the runtime explicitly exposes activation arenas, output preallocation, or external state buffers.

For your specific context, the uploaded project notes already define step 3 as a LlamaTensorBinder / LlamaWeightMap stage for a pure C# llama.cpp replacement, with first-pass tensors such as token_embd.weight, per-block attention and FFN weights, output_norm.weight, and usually output.weight. That is the right decomposition. The missing rigor is to formalize tensor contracts from GGUF metadata and architecture hyperparameters, reject mismatches before execution starts, and treat tensor binding as a tested subsystem rather than an ad hoc lookup step.

When model details are unknown, the safest implementation strategy is to separate the problem into two layers. First, build a framework-neutral binding contract: TensorDesc { name, role, shape, dtype, device, byte_size, layout, quantization, storage_kind }. Second, map that contract to runtime-specific APIs: load_state_dict and .to(...) in PyTorch, TensorSpec / tf.device or TFLite set_tensor() in TensorFlow, Ort::Value with IoBinding in ONNX Runtime, setInputShape plus setTensorAddress in TensorRT, and memory-mapped tensor views plus an arena in a custom GGUF runtime.

The biggest implementation mistakes are predictable: binding by raw name without architectural validation, silently transposing shape conventions, passing host buffers where device buffers are required, rebinding hidden state with the wrong lifetime, assuming zero-copy where the API actually copies, and failing to account for quantized block formats. Official docs call out these failure modes directly: TFLite invalidates held NumPy views across allocate_tensors() and invoke(), ONNX Runtime may insert host-device copies unless I/O is explicitly bound, TensorRT requires correct host-vs-device placement and buffer alignment, PyTorch distinguishes parameters from buffers and reports missing/unexpected keys during weight loading, and GGML/GGUF exposes explicit tensor type, stride, contiguity, and row-size rules.

The practical recommendation is straightforward: treat binding as a schema-validation and memory-planning phase, not merely a dictionary lookup. If you do that, step 3 becomes the foundation for correctness, diagnostics, and performance, rather than a later source of parity bugs.

Scope and assumptions

The exact model family, framework version, and target accelerator are unspecified, so the report treats binding in two layers: a generic runtime design, and a concrete decoder-only transformer example aligned with the GGUF/LLaMA-style tensor names already present in your project notes. Where tensor shapes depend on architecture settings, the binder should derive them from model metadata such as vocab_size, hidden_size, intermediate_size, num_hidden_layers, num_attention_heads, num_key_value_heads, and head_dim, rather than hard-coding model-specific constants.

For GGUF-based runtimes specifically, the file format is designed to contain everything needed to load a model, with tensor metadata separate from tensor data, tensor offsets relative to the tensor data section, explicit alignment, and mmap compatibility. That strongly favors a design in which immutable weights are exposed as typed views over memory-mapped storage, while dynamic tensors such as inputs, outputs, scratch activations, and KV cache live in allocator-managed memory.

A useful implementation rule is to distinguish four tensor classes:

This split aligns with the documented API surfaces in PyTorch, TensorFlow/TFLite, ONNX Runtime, TensorRT, and GGML-style runtimes.

Required tensors and contracts

The most important conceptual point is that “all required tensors” does not mean every temporary activation in the graph. In most mainstream runtimes, the runtime allocates ephemeral intermediates itself. What you usually must bind are: model parameters and persistent buffers; user inputs; preallocated outputs if you want reuse or device-resident output; and any persistent state that the runtime exposes explicitly, such as KV cache or recurrent state. TensorRT makes this distinction explicit by saying an execution context needs device memory for internal activation tensors, while separately exposing input/output tensor addresses. TensorFlow C++ tensors and GGML contexts similarly separate tensor objects from allocator-managed storage.

Generic tensor contract

A production binder should validate at least these fields for every bindable tensor:

Official APIs surface each of these in some form: PyTorch distinguishes parameters and buffers and exposes device/dtype conversion; TensorFlow uses TensorSpec, Tensor, and tf.device; ONNX Runtime exposes type info, memory info, and Ort::Value; TensorRT exposes input shapes, tensor location, tensor addresses, and shape-inference I/O; GGML exposes type, dimensions, byte strides, contiguity, row-size helpers, and row-data validation.

Concrete LLaMA-style GGUF tensor set

Your uploaded notes already identify the right first-pass tensor list for a GGUF/LLaMA runtime. GGUF standardizes the relevant transformer tensor names, including token_embd, output_norm, output, and per-block names such as attn_q, attn_k, attn_v, attn_output, ffn_gate, ffn_up, and ffn_down. Hugging Face’s Llama config exposes the architecture dimensions needed to validate these tensors semantically.

These are the logical validation shapes. The raw stored order may differ by kernel convention or packed quantized layout, so the binder should validate semantic dimensions and compute bytes using runtime-specific row-size rules, not assume a single memory layout. GGUF/ggml also supports multiple quantized formats beyond plain F16/F32, including Q4_0, Q4_1, Q8_0, BF16, and newer quantized types, which means byte size is not always numel * sizeof(dtype).

For decoder-only inference, the usual run-time tensors are input_ids with shape [batch, seq], optional masks or positions of similar rank, logits with shape [batch, seq, vocab_size], and per-layer KV cache whose packed layout varies by runtime but is typically derived from num_key_value_heads, head_dim, current sequence length, and batch size. That cache should be modeled as a first-class bindable state tensor if you plan incremental decode or paging/offload. The exact cache layout is a runtime design choice, but its semantic dimensions come directly from the model config.

flowchart LR
    A[GGUF file or framework checkpoint] --> B[Parse metadata and tensor descriptors]
    B --> C[Validate required tensor registry]
    C --> D[Create tensor views or owned buffers]
    D --> E[Bind immutable weights]
    D --> F[Bind per-run inputs and outputs]
    D --> G[Bind persistent execution state]
    E --> H[Execution context]
    F --> H
    G --> H
    H --> I[Run]
    I --> J[Reuse state or release per-run buffers]

Lifetime and ownership

Lifetime bugs are some of the hardest binding bugs to diagnose, so the binder should make ownership explicit in the type system. PyTorch’s torch.frombuffer shares memory with the underlying CPU buffer and keeps that object alive through reference counting, but the returned tensor is not resizable and the behavior is undefined if the underlying buffer is not CPU-resident. TensorFlow C++ tensors expose refcounted TensorBuffers and require allocators to outlive the created tensor. TFLite’s tensor() gives a zero-copy NumPy view, but holding that view across allocate_tensors() or invoke() is explicitly unsafe. TensorRT requires that bound memory remain valid until the stream has reached the appropriate synchronization point.

In a custom GGUF runtime, the cleanest ownership model is usually: immutable weights are borrowed mmap views tied to the model object; scratch/intermediate memory is arena-owned by the execution context; outputs are either caller-owned or session-owned reusable buffers; and KV cache is session-owned with optional layer-wise device migration. GGML’s context and no_alloc/view facilities are a good primary-source model for this division.

Binding patterns across runtimes

There is no single universal “bind tensor” API. Instead, each runtime exposes a slightly different contract.

This table reflects official API surfaces rather than one implementation style.

PyTorch

PyTorch does not usually have a separate “binding phase” for inference. Parameters and persistent buffers are attached to the module, loaded with load_state_dict, and then placed on the desired device with .to(...). Non-parameter state belongs in registered buffers, not in ad hoc side containers. At a lower level, aliased tensor creation from external memory is possible, but then your binder owns the lifetime problem.

import torch

model = MyModel()
state = torch.load("model.pt", map_location="cpu", weights_only=True)
missing, unexpected = model.load_state_dict(state, strict=True)
assert not missing and not unexpected

model.eval()
model.to(device="cuda", dtype=torch.float16)

# External CPU buffer aliasing; shared-memory semantics apply.
token_buf = bytearray(batch * seq * 4)  # int32 example
input_ids_cpu = torch.frombuffer(token_buf, dtype=torch.int32).view(batch, seq)
input_ids = input_ids_cpu.to("cuda", non_blocking=True)

with torch.inference_mode():
    logits = model(input_ids).logits

load_state_dict is also a built-in diagnostic surface: it reports missing_keys and unexpected_keys, and it raises if a key maps onto a parameter or buffer registered as None. That makes it a useful model for your own custom binder diagnostics.

TensorFlow

For regular TensorFlow, the binding contract is usually expressed through TensorSpec and device placement through tf.device(...). For TFLite, binding is much more explicit: query get_input_details() and get_output_details(), resize dynamic inputs if necessary, call allocate_tensors(), then either copy with set_tensor() or write through the zero-copy callable returned by tensor(). TFLite also documents that intermediate tensors are not readable this way and that held NumPy views become invalid across allocation or invoke boundaries.

import tensorflow as tf
import numpy as np

@tf.function(input_signature=[tf.TensorSpec([None, None], tf.int32, name="input_ids")])
def run_model(input_ids):
    return model(input_ids)

with tf.device("/GPU:0"):
    logits = run_model(tf.constant(np_input_ids, dtype=tf.int32))

import tensorflow as tf

interpreter = tf.lite.Interpreter(model_path="model.tflite")
input_info = interpreter.get_input_details()[0]
output_info = interpreter.get_output_details()[0]

interpreter.resize_tensor_input(input_info["index"], [batch, seq], strict=False)
interpreter.allocate_tensors()

# Copying path:
interpreter.set_tensor(input_info["index"], input_ids_np)

# Zero-copy path:
input_view = interpreter.tensor(input_info["index"])
input_view()[:] = input_ids_np

interpreter.invoke()
output_view = interpreter.tensor(output_info["index"])
logits = output_view()  # obtain a fresh view; do not retain across future allocate/invoke calls

ONNX Runtime

ONNX Runtime is the clearest example of “binding matters for performance.” The official I/O Binding guide explains that, with non-CPU execution providers, inputs not already on the target device are copied during Run(), and outputs not preallocated on the target device are copied back to CPU at the end of Run(). IoBinding lets you bind device-resident input/output tensors or bind only an output OrtMemoryInfo so the session allocates the right device buffer for dynamic shapes. Ort::Value::CreateTensor supports both user-supplied memory and allocator-owned memory.

Ort::Env env{ORT_LOGGING_LEVEL_WARNING, "bind"};
Ort::Session session(env, "model.onnx", session_options);

Ort::MemoryInfo cuda_info("Cuda", OrtArenaAllocator, 0, OrtMemTypeDefault);
Ort::MemoryInfo cpu_info = Ort::MemoryInfo::CreateCpu(OrtArenaAllocator, OrtMemTypeDefault);

std::array<int64_t, 2> input_shape{batch, seq};
auto input = Ort::Value::CreateTensor<int64_t>(
    &cpu_info,
    input_ids.data(),
    input_ids.size(),
    input_shape.data(),
    input_shape.size());

Ort::IoBinding io(session);
io.BindInput("input_ids", input);

// Dynamic-shape output: let ORT allocate on device.
io.BindOutput("logits", &cuda_info);

session.Run(Ort::RunOptions{nullptr}, io);
io.SynchronizeOutputs();

If the output shape is known and reused, binding a reusable Ort::Value is usually better than rebinding OrtMemoryInfo every run. If you are targeting a GPU-heavy inference loop, this should be your default path.

TensorRT

TensorRT separates shape specification, host/device placement, and buffer address binding. The runtime tells you whether a tensor must live on host or device through getTensorLocation(). Dynamic input shapes must be provided with setInputShape(). Actual buffers are then bound with setTensorAddress or the input/output-specific variants, and inference is launched with enqueueV3(stream). TensorRT also documents that shape tensors are often host-resident, that bound pointers must satisfy alignment requirements, and that input/output memory must not be released too early.

auto* ctx = engine->createExecutionContext();
ctx->setInputShape("input_ids", Dims2{batch, seq});

if (engine->getTensorLocation("input_ids") == TensorLocation::kDEVICE) {
    // d_input and d_logits are device pointers with sufficient size and alignment.
    ctx->setInputTensorAddress("input_ids", d_input);
    ctx->setOutputTensorAddress("logits", d_logits);
} else {
    ctx->setInputTensorAddress("input_ids", h_input);
    ctx->setOutputTensorAddress("logits", h_logits);
}

bool ok = ctx->enqueueV3(stream);
if (!ok) throw std::runtime_error("TensorRT enqueueV3 failed");

// Do not release bound buffers before the required stream synchronization point.
cudaStreamSynchronize(stream);

A good TensorRT binder should additionally validate isShapeInferenceIO(name) and route those tensors separately, because shape-inference inputs are part of the binding contract even when they are not “normal” execution tensors.

Custom C++ runtime

For a custom GGUF runtime, the binding layer should be explicit and deterministic. A robust pattern is: parse GGUF tensor descriptors; map required standardized names to semantic slots; validate logical shapes against architecture metadata; create immutable memory-mapped tensor views for weights; allocate session-owned scratch buffers and persistent state buffers; and reject engine creation if any required tensor is missing or semantically inconsistent. That is precisely the design your uploaded notes are already pointing toward.

struct TensorDesc {
    std::string name;
    std::vector<int64_t> shape;
    DataType dtype;
    DeviceKind device;
    size_t byte_size;
    bool writable;
};

struct TensorView {
    TensorDesc desc;
    void* data;              // mmap view or owned allocation
    Ownership ownership;     // BorrowedMmap, ArenaOwned, CallerOwned
};

class LlamaTensorBinder {
public:
    void BindRequired(const GgufModel& gguf, const LlamaConfig& cfg) {
        BindWeight("token_embd.weight",  {cfg.vocab_size, cfg.hidden_size});
        BindWeight("output_norm.weight", {cfg.hidden_size});
        BindWeightIfPresentOrTied("output.weight", {cfg.hidden_size, cfg.vocab_size});

        for (int i = 0; i < cfg.num_hidden_layers; ++i) {
            BindWeight(fmt("blk.%d.attn_norm.weight", i), {cfg.hidden_size});
            BindWeight(fmt("blk.%d.attn_q.weight", i),    {cfg.hidden_size, cfg.hidden_size});
            BindWeight(fmt("blk.%d.attn_k.weight", i),    {cfg.hidden_size, cfg.num_key_value_heads * cfg.head_dim});
            BindWeight(fmt("blk.%d.attn_v.weight", i),    {cfg.hidden_size, cfg.num_key_value_heads * cfg.head_dim});
            BindWeight(fmt("blk.%d.attn_output.weight", i), {cfg.hidden_size, cfg.hidden_size});
            BindWeight(fmt("blk.%d.ffn_norm.weight", i),  {cfg.hidden_size});
            BindWeight(fmt("blk.%d.ffn_gate.weight", i),  {cfg.hidden_size, cfg.intermediate_size});
            BindWeight(fmt("blk.%d.ffn_up.weight", i),    {cfg.hidden_size, cfg.intermediate_size});
            BindWeight(fmt("blk.%d.ffn_down.weight", i),  {cfg.intermediate_size, cfg.hidden_size});
        }
    }
};

flowchart TD
    A[Read model signature and metadata] --> B[Build required tensor registry]
    B --> C[Resolve tensor by standardized name]
    C --> D{Found?}
    D -- no --> E[Fail fast with missing tensor diagnostic]
    D -- yes --> F[Validate role, rank, dims, dtype, quantization]
    F --> G{Placement decided?}
    G -- no --> H[Select CPU/GPU/EP memory target]
    G -- yes --> I[Create owning buffer or borrowed view]
    H --> I
    I --> J[Bind to runtime handle]
    J --> K[Warmup shape pass or test inference]
    K --> L{All outputs valid?}
    L -- no --> M[Emit mismatch diagnostics]
    L -- yes --> N[Cache bindings for reuse]

Failure modes and verification

A tensor binder should fail early and structurally. The best diagnostics name the missing or mismatched tensor, show the expected semantic contract, show the discovered raw metadata, and explain whether the issue is fatal, fallback-capable, or potentially auto-correctable. PyTorch’s load_state_dict is a good behavioral model here because it returns missing_keys and unexpected_keys, and because strict loading is the default correctness posture for model-state alignment.

The most common mismatch classes are:

Each item above is grounded in documented runtime behavior: strict state loading and buffers in PyTorch, TFLite’s copy-vs-view APIs and invalidated views, ONNX Runtime’s explicit I/O binding and synchronization, TensorRT’s shape/location/address rules, and GGML’s shape/stride/row-data helpers.

Automated tests should mirror those failure surfaces. The strongest minimum test suite is: a signature audit that enumerates required tensors and validates names; a shape/dtype audit that checks all resolved tensors against semantic architecture parameters; a missing/unexpected tensor test against intentionally broken checkpoints; a device-placement test that asserts runtime-specific host/device requirements; a dynamic-shape test for prefill and single-token decode; a lifetime test for reusable output/KV buffers; and an oracle-parity test that compares top logits or greedy next-token IDs against a trusted reference runtime for one known-good model. Your project notes already recommend exactly this style of parity-first development.

For debugging, log at two levels. At model load, emit a binding manifest listing tensor name, role, shape, dtype, placement, byte size, and ownership. At inference time, emit run-time binding summaries: input names, shapes, dtypes, devices, dynamic-shape decisions, and whether outputs were reused, newly allocated, or copied. This is especially important for ONNX Runtime and TensorRT, where an apparently correct inference result can still hide avoidable copies or wrong device placement.

Performance implications and implementation checklist

Binding strategy has first-order performance impact. ONNX Runtime’s documentation is explicit that failing to pre-place inputs and outputs on the execution device makes Run() include extra CPU-device copies, which can dominate latency and mislead users into blaming the runtime. Its device-tensor documentation further notes that fixed tensor addresses enable optimizations such as CUDA graph capture and better overlap of compute with transfers. TensorRT similarly benefits from stable tensor addresses and non-default streams, while warning that the default stream may introduce extra synchronization. PyTorch’s .to(..., non_blocking=True) can reduce host-side stalls when the transfer path supports it, and TFLite’s tensor() API avoids copies when used correctly.

For GGUF-style custom runtimes, the biggest performance wins typically come from: keeping large immutable weights memory-mapped or otherwise avoiding redundant copies; separating immutable weight storage from scratch arenas; using explicit row-size helpers for quantized tensor byte calculations; preallocating reusable outputs and KV cache; and avoiding per-token name lookups by compiling the binding manifest into direct pointers or indices once. GGML’s docs explicitly motivate preallocating a context buffer to avoid runtime allocation overhead and expose helpers such as ggml_row_size, ggml_nbytes, contiguity checks, and ggml_validate_row_data to make this safe.

A concise implementation checklist for step 3 is:

• Build a required tensor registry from architecture metadata, not from best-effort name guessing.
• Validate name, role, rank, semantic dimensions, dtype, byte size, and device before session creation.
• Separate immutable weights, per-run I/O, and persistent state in the API and memory planner.
• Prefer borrowed mmap views for GGUF weights and arena-owned buffers for scratch/intermediates.
• Make ownership and lifetime explicit in code and logs.
• Preallocate device-resident outputs and caches when latency matters.
• Treat dynamic-shape and shape-inference tensors as first-class bindings, not edge cases.
• Add parity tests and intentionally broken fixtures before expanding architecture support.
• Fail fast on missing/unexpected keys and print exact diagnostics.
• Cache the finalized binding manifest so steady-state inference does zero name resolution on the hot path. This is an implementation recommendation derived from the runtime APIs above.

Selected references

The following primary or official sources are the most useful anchors for implementing this step in practice:

• GGUF specification and standardized tensor names.
• GGML tensor types, shape/stride helpers, contiguity checks, row-size helpers, and validation helpers.
• Your uploaded project notes describing LlamaTensorBinder, first-pass required tensors, and parity-first sequencing.
• PyTorch Module.load_state_dict, register_buffer, Tensor.to, and torch.frombuffer.
• TensorFlow TensorSpec, tf.device, TensorFlow C++ Tensor, and TFLite Interpreter.
• ONNX Runtime Ort::Session, Ort::Value, Ort::IoBinding, I/O Binding guide, and device-tensor guide.
• TensorRT IExecutionContext tensor binding, tensor location, dynamic shape, temporary storage, and enqueueV3.
• Hugging Face Llama config and output shape documentation for semantic dimension checks.