Enterprise Architecture for a Fully Custom C Customer Engineering LLM Runtime

Full report

On this page

Executive summary

This report assumes .NET 7+, Linux containers for production deployment, and a customer-engineering runtime that must serve multiple enterprise tenants, enforce policy, expose a stable internal API, and integrate directly with provider-hosted LLMs while leaving room for an optional local/native inference path. That assumption is consistent with the attached references, which emphasize a direct-to-the-metal C# approach for local inference, and with the current provider and .NET documentation for direct HTTP, streaming, authentication, configuration, dependency injection, and observability.

The highest-confidence architecture is a Clean Architecture, SOLID, provider-neutral runtime core with a canonical inference contract at the center and pluggable model adapters around it. In practice, the runtime should treat provider-hosted models as the primary execution path for most enterprise workloads, because OpenAI, Azure OpenAI, Anthropic, and Gemini all expose materially different authentication methods, streaming semantics, rate-limit surfaces, and error behaviors; those differences are exactly what the adapter boundary should absorb. OpenAI supports Responses API creation, SSE streaming, structured outputs via JSON schema, and exposes error codes in the response object; Anthropic documents SSE streaming, retry-after, and rate-limit headers; Gemini documents REST generateContent, SSE streamGenerateContent, and project-level rate limits.

A secondary local-backend path is still strategically valuable for regulated, offline, cost-controlled, or ultra-low-latency workloads, but it should be isolated behind a separate backend boundary, and preferably even a separate worker process, until it is mature. The attached references consistently converge on the same local-inference fundamentals: GGUF ingestion, unmanaged/aligned memory, memory mapping, SIMD or hardware intrinsics, paged KV cache management, batching, and direct native interop. Those themes are consistent with official .NET guidance on NativeMemory, SIMD/intrinsics, P/Invoke/LibraryImport, Native AOT, and with primary sources around ONNX Runtime C#, ONNX opset/version semantics, and llama.cpp’s quantized inference model and OpenAI-compatible serving.

The enterprise recommendation is therefore:

1. Build the runtime core around canonical domain interfaces, not around any provider SDK.
2. Use built-in .NET DI, configuration, logging, and typed HTTP clients as foundational infrastructure.
3. Standardize on async, bounded, back-pressured request processing using channels and cooperative cancellation.
4. Implement policy, telemetry, rate limiting, retries, auth, and secrets as cross-cutting layers, not scattered in adapters.
5. Support provider-native caching and streaming, but keep them behind platform-neutral abstractions.
6. Treat local/native inference as a specialized backend, not as the default responsibility of the CE API tier.
7. Enforce automated testing, contract validation, and progressive delivery before model or runtime upgrades are allowed into production.

Architectural blueprint

The right shape is a four-ring Clean Architecture: Presentation, Application, Domain, and Infrastructure. The Domain layer owns canonical request/response types, policies, invariants, and interfaces. The Application layer owns orchestration use cases such as prompt rendering, token budgeting, tool routing, streaming, and retry decisions. Infrastructure owns provider adapters, tokenizers, native/local backends, caches, credentials, telemetry exporters, and persistence. Presentation exposes REST, gRPC, streaming, and health endpoints. This structure aligns well with .NET’s built-in DI container, options pattern, configuration providers, and structured logging.

flowchart LR
    subgraph ClientSurface[Client Surface]
        Web[Web App]
        CE[CE Tools]
        S2S[Service to Service]
    end

    subgraph Presentation[Presentation Layer]
        Api[REST API]
        Grpc[gRPC API]
        Stream[SSE/WebSocket Stream Endpoint]
        Health[Health/Readiness Endpoints]
    end

    subgraph Application[Application Layer]
        Orchestrator[Inference Orchestrator]
        PromptManager[Prompt Manager]
        Scheduler[Batching Scheduler]
        Policy[Policy & Guardrails]
        RateLimiter[Rate Limiter]
        Retry[Retry/Backoff Policy]
    end

    subgraph Domain[Domain Layer]
        Contracts[Canonical Contracts]
        Interfaces[Interfaces]
        Models[Model Manifest & Routing Policy]
    end

    subgraph Infrastructure[Infrastructure Layer]
        OpenAI[OpenAI Adapter]
        Azure[Azure OpenAI Adapter]
        Anthropic[Anthropic Adapter]
        Gemini[Gemini Adapter]
        Local[Local Backend Adapter]
        Tokenizer[Tokenizer Services]
        Cache[Cache Layers]
        Telemetry[Telemetry Pipeline]
        Secrets[Secrets & Credentials]
        Config[Typed Configuration]
    end

    subgraph External[External Systems]
        OAI[OpenAI API]
        AOAI[Azure OpenAI]
        Claude[Anthropic Claude API]
        GAI[Gemini API]
        Worker[Local Inference Worker]
        KV[Redis / Distributed Cache]
        Vault[Key Vault / Secret Manager]
        OTel[OTel Collector]
    end

    Web --> Api
    CE --> Api
    S2S --> Grpc

    Api --> Orchestrator
    Grpc --> Orchestrator
    Stream --> Orchestrator
    Health --> Scheduler

    Orchestrator --> PromptManager
    Orchestrator --> Scheduler
    Orchestrator --> Policy
    Orchestrator --> RateLimiter
    Orchestrator --> Retry
    Orchestrator --> Contracts
    Orchestrator --> Interfaces

    Interfaces --> OpenAI
    Interfaces --> Azure
    Interfaces --> Anthropic
    Interfaces --> Gemini
    Interfaces --> Local

    PromptManager --> Tokenizer
    Scheduler --> Cache
    Retry --> OpenAI
    Retry --> Azure
    Retry --> Anthropic
    Retry --> Gemini
    Retry --> Local

    OpenAI --> OAI
    Azure --> AOAI
    Anthropic --> Claude
    Gemini --> GAI
    Local --> Worker
    Cache --> KV
    Secrets --> Vault
    Telemetry --> OTel

The most important design decision is the canonical contract boundary. Your application should speak in terms of InferenceRequest, InferenceResponse, StreamingDelta, ToolInvocation, ModelDescriptor, UsageRecord, and ProviderError, not OpenAI-specific or Anthropic-specific DTOs. That is the only way to survive provider drift, model replacement, and dual-backend operation. OpenAI’s Responses API, Anthropic’s Messages API, and Gemini’s generation APIs are similar enough to normalize, but different enough that normalization must be deliberate.

A practical component partition is shown below.

The class structure below keeps dependencies flowing inward.

classDiagram
    direction LR

    class IInferenceOrchestrator {
      +ExecuteAsync(request, cancellationToken) Task~InferenceResponseDto~
      +StreamAsync(request, cancellationToken) IAsyncEnumerable~InferenceEventDto~
    }

    class IModelAdapter {
      +ProviderName string
      +SupportsStreaming bool
      +GenerateAsync(request, cancellationToken) Task~AdapterResponse~
      +StreamAsync(request, cancellationToken) IAsyncEnumerable~AdapterEvent~
    }

    class ITokenizer {
      +CountTokensAsync(request, cancellationToken) ValueTask~TokenCountResult~
      +TruncateAsync(request, budget, cancellationToken) ValueTask~InferenceRequestDto~
    }

    class IPromptManager {
      +RenderAsync(request, cancellationToken) ValueTask~PromptEnvelope~
      +CompactAsync(request, budget, cancellationToken) ValueTask~PromptEnvelope~
    }

    class IRequestScheduler {
      +EnqueueAsync(request, cancellationToken) Task~ScheduledRequest~
    }

    class IResponseCache {
      +TryGetAsync(key, cancellationToken) ValueTask~CacheHit~
      +SetAsync(key, response, ttl, cancellationToken) Task
    }

    class IRateLimitCoordinator {
      +AcquireAsync(key, weight, cancellationToken) ValueTask~Lease~
    }

    class InferenceOrchestrator
    class PromptManager
    class TokenizerFacade
    class ProviderAdapterRegistry
    class ResponseCache
    class RateLimitCoordinator

    IInferenceOrchestrator <|.. InferenceOrchestrator
    IPromptManager <|.. PromptManager
    ITokenizer <|.. TokenizerFacade
    IResponseCache <|.. ResponseCache
    IRateLimitCoordinator <|.. RateLimitCoordinator

    InferenceOrchestrator --> IPromptManager
    InferenceOrchestrator --> ITokenizer
    InferenceOrchestrator --> IRequestScheduler
    InferenceOrchestrator --> IResponseCache
    InferenceOrchestrator --> IRateLimitCoordinator
    InferenceOrchestrator --> IModelAdapter
    ProviderAdapterRegistry --> IModelAdapter

For dependency injection, use the .NET Generic Host, strongly typed options, and named or typed HttpClient registrations. Microsoft explicitly documents DI as a built-in framework capability, the options pattern as the preferred way to bind related hierarchical configuration, and IHttpClientFactory as the right foundation for DI-ready clients with centralized configuration, outgoing middleware, and logging.

A production adapter strategy looks like this:

OpenAI has an official .NET library, Azure OpenAI documents .NET package usage with Azure Identity, and Gemini’s official docs clearly show a REST-first path with provider-specific headers and SSE streaming endpoints. That combination strongly favors a runtime that owns its own adapter abstractions instead of treating any one SDK as the architectural center.

Runtime modules and canonical API contracts

A CE runtime should expose a small, durable surface area and hide provider churn behind it. The canonical request pipeline is: authenticate → authorize → resolve tenant/model policy → render prompt → count/truncate → acquire capacity → execute or stream via adapter → normalize usage/errors → emit telemetry → cache where allowed. That flow is justified by the provider/auth/rate limit/streaming differences in the official docs and by the enterprise need for consistent behavior across adapters.

The module boundaries should be explicit.

OpenAI’s Responses API supports hierarchical message roles, tool definitions with JSON schema, and structured outputs using json_schema with optional strict adherence. That makes it an excellent reference point for your canonical DTOs, even if you remain provider-neutral.

The API surface below is intentionally small and durable.

The error contract should be canonical even when upstreams differ.

The sequence below shows the preferred streaming path.

sequenceDiagram
    autonumber
    participant C as Client
    participant API as API Layer
    participant AUTH as Auth/Policy
    participant PM as Prompt Manager
    participant TOK as Tokenizer
    participant SCH as Scheduler
    participant RL as Rate Limiter
    participant ADP as Model Adapter
    participant UP as Provider or Local Backend
    participant TEL as Telemetry

    C->>API: POST /v1/responses?stream=true
    API->>AUTH: Validate identity and tenant policy
    AUTH-->>API: Authorized context
    API->>PM: Render prompt envelope
    PM->>TOK: Count / truncate / compact
    TOK-->>PM: Budgeted request
    API->>SCH: Enqueue scheduled request
    SCH->>RL: Acquire tenant and provider capacity
    RL-->>SCH: Lease granted
    SCH->>ADP: StreamAsync(canonical request)
    ADP->>UP: Provider-native streaming request
    UP-->>ADP: Delta / event stream
    ADP-->>API: Canonical streaming events
    API-->>C: SSE/WebSocket deltas
    ADP->>TEL: Usage, latency, trace, outcome
    API-->>C: Terminal event with usage and status

A representative C# contract set is below.

using System.Collections.Generic;
using System.ComponentModel.DataAnnotations;

namespace CeRuntime.Contracts;

/// <summary>
/// Represents a canonical inference request for any provider or local backend.
/// </summary>
public sealed class InferenceRequestDto
{
    [Display(Name = "Tenant")]
    public string Tenant { get; init; } = string.Empty;

    [Display(Name = "Model")]
    public string Model { get; init; } = string.Empty;

    [Display(Name = "Conversation")]
    public string? ConversationId { get; init; }

    [Display(Name = "Stream")]
    public bool Stream { get; init; }

    [Display(Name = "Max Output Tokens")]
    public int? MaxOutputTokens { get; init; }

    [Display(Name = "Temperature")]
    public double? Temperature { get; init; }

    [Display(Name = "Top P")]
    public double? TopP { get; init; }

    [Display(Name = "Messages")]
    public IReadOnlyList<MessageDto> Messages { get; init; } = [];

    [Display(Name = "Tools")]
    public IReadOnlyList<ToolDefinitionDto> Tools { get; init; } = [];

    [Display(Name = "Metadata")]
    public IReadOnlyDictionary<string, string> Metadata { get; init; } = new Dictionary<string, string>();
}

/// <summary>
/// Represents a canonical message in an inference request.
/// </summary>
public sealed class MessageDto
{
    [Display(Name = "Role")]
    public string Role { get; init; } = string.Empty;

    [Display(Name = "Content")]
    public string Content { get; init; } = string.Empty;
}

/// <summary>
/// Represents a tool definition exposed to a model.
/// </summary>
public sealed class ToolDefinitionDto
{
    [Display(Name = "Name")]
    public string Name { get; init; } = string.Empty;

    [Display(Name = "Description")]
    public string Description { get; init; } = string.Empty;

    [Display(Name = "Json Schema")]
    public string JsonSchema { get; init; } = string.Empty;
}

/// <summary>
/// Represents a normalized inference response.
/// </summary>
public sealed class InferenceResponseDto
{
    [Display(Name = "Id")]
    public string Id { get; init; } = string.Empty;

    [Display(Name = "Model")]
    public string Model { get; init; } = string.Empty;

    [Display(Name = "Output Text")]
    public string OutputText { get; init; } = string.Empty;

    [Display(Name = "Finish Reason")]
    public string FinishReason { get; init; } = string.Empty;

    [Display(Name = "Usage")]
    public UsageDto Usage { get; init; } = new();

    [Display(Name = "Provider")]
    public string Provider { get; init; } = string.Empty;
}

/// <summary>
/// Represents normalized token and cost usage.
/// </summary>
public sealed class UsageDto
{
    [Display(Name = "Input Tokens")]
    public int InputTokens { get; init; }

    [Display(Name = "Output Tokens")]
    public int OutputTokens { get; init; }

    [Display(Name = "Cached Input Tokens")]
    public int CachedInputTokens { get; init; }

    [Display(Name = "Estimated Cost Micros")]
    public long EstimatedCostMicros { get; init; }
}

/// <summary>
/// Defines the adapter boundary for provider-hosted or local models.
/// </summary>
public interface IModelAdapter
{
    /// <summary>
    /// Gets the provider name exposed by the adapter.
    /// </summary>
    string ProviderName { get; }

    /// <summary>
    /// Executes a single non-streaming inference request.
    /// </summary>
    /// <param name="request">The canonical inference request.</param>
    /// <param name="cancellationToken">The cooperative cancellation token.</param>
    /// <returns>The normalized inference response.</returns>
    Task<InferenceResponseDto> GenerateAsync(
        InferenceRequestDto request,
        CancellationToken cancellationToken);

    /// <summary>
    /// Executes a streaming inference request.
    /// </summary>
    /// <param name="request">The canonical inference request.</param>
    /// <param name="cancellationToken">The cooperative cancellation token.</param>
    /// <returns>A stream of normalized inference events.</returns>
    IAsyncEnumerable<InferenceEventDto> StreamAsync(
        InferenceRequestDto request,
        CancellationToken cancellationToken);
}

/// <summary>
/// Represents a normalized streaming event.
/// </summary>
public sealed class InferenceEventDto
{
    [Display(Name = "Type")]
    public string Type { get; init; } = string.Empty;

    [Display(Name = "Delta")]
    public string? Delta { get; init; }

    [Display(Name = "Response Id")]
    public string? ResponseId { get; init; }

    [Display(Name = "Usage")]
    public UsageDto? Usage { get; init; }
}

A clean registration pattern uses typed options plus named/typed clients.

using CeRuntime.Contracts;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.DependencyInjection.Extensions;
using Microsoft.Extensions.Options;

namespace CeRuntime.Bootstrap;

/// <summary>
/// Provides runtime service registration helpers.
/// </summary>
public static class ServiceCollectionExtensions
{
    /// <summary>
    /// Registers CE runtime core services and provider adapters.
    /// </summary>
    /// <param name="services">The service collection.</param>
    /// <returns>The updated service collection.</returns>
    public static IServiceCollection AddCeRuntime(this IServiceCollection services)
    {
        services.AddOptions<RuntimeOptions>()
            .BindConfiguration("Runtime")
            .ValidateDataAnnotations()
            .ValidateOnStart();

        services.AddOptions<OpenAiAdapterOptions>()
            .BindConfiguration("Providers:OpenAI")
            .ValidateDataAnnotations()
            .ValidateOnStart();

        services.AddOptions<AzureOpenAiAdapterOptions>()
            .BindConfiguration("Providers:AzureOpenAI")
            .ValidateDataAnnotations()
            .ValidateOnStart();

        services.AddHttpClient("openai");
        services.AddHttpClient("azure-openai");
        services.AddHttpClient("anthropic");
        services.AddHttpClient("gemini");

        services.TryAddSingleton<ITokenizer, CompositeTokenizer>();
        services.TryAddSingleton<IPromptManager, PromptManager>();
        services.TryAddSingleton<IInferenceOrchestrator, InferenceOrchestrator>();

        services.AddKeyedSingleton<IModelAdapter, OpenAiModelAdapter>("openai");
        services.AddKeyedSingleton<IModelAdapter, AzureOpenAiModelAdapter>("azure-openai");
        services.AddKeyedSingleton<IModelAdapter, AnthropicModelAdapter>("anthropic");
        services.AddKeyedSingleton<IModelAdapter, GeminiModelAdapter>("gemini");

        return services;
    }
}

/// <summary>
/// Represents runtime-wide configuration.
/// </summary>
public sealed class RuntimeOptions
{
    [Display(Name = "Default Model")]
    [Required]
    public string DefaultModel { get; init; } = string.Empty;

    [Display(Name = "Max Concurrent Requests")]
    [Range(1, int.MaxValue)]
    public int MaxConcurrentRequests { get; init; } = 128;

    [Display(Name = "Enable Semantic Cache")]
    public bool EnableSemanticCache { get; init; } = true;
}

Runtime engineering for concurrency, memory, performance, and scale

The runtime should be async all the way to the transport edge. Provider API calls are naturally I/O-bound and stream-oriented, and both OpenAI and Anthropic explicitly document SSE streaming, while Gemini documents streamGenerateContent over SSE as well. A sync-over-async API surface will artificially serialize network latency, consume thread-pool threads unnecessarily, and make cancellation far less reliable.

Inside the service, use bounded Channel<T> pipelines for admission control and batching. Microsoft’s channels library is explicitly designed for asynchronous producer/consumer pipelines, supports bounded and unbounded modes, exposes backpressure behavior, and uses ValueTask to reduce allocations. Bounded channels are the right primitive for request queues, streaming fan-out buffers, and microbatch schedulers.

For outbound provider calls, use IHttpClientFactory with named or typed clients or long-lived HttpClient instances with PooledConnectionLifetime, but do not create/dispose HttpClient per request. Microsoft documents that each HttpClient has its own connection pool, that repeated disposal causes connection recreation and can contribute to port exhaustion, and that recommended lifetime strategies are either factory-created clients or long-lived clients with connection lifetime management. IHttpClientFactory additionally centralizes logging and outgoing middleware.

The biggest concurrency decision is not “threaded or async”; it is which workloads stay on async network paths and which workloads move onto dedicated compute workers.

For batching, the tradeoff is not purely throughput versus latency. It is throughput versus fairness versus cancellation complexity.

The caching strategy should also be layered, because “cache” means different things in different parts of the runtime.

Provider-native caching is worth surfacing in the adapter. OpenAI documents request-level prompt caching with prompt_cache_retention, including 24h support and token thresholds; Anthropic documents prompt caching via cache_control, a default 5-minute cache lifetime, optional 1-hour TTL, cache usage details, and cache isolation properties. Those are not interchangeable features, which is exactly why they belong behind a provider-capability interface.

On the memory side, the primary hosted-runtime strategy is minimize transient allocations. Use ArrayPool<T> or ObjectPool<T> for serialization buffers, stream frame builders, and token arrays; keep request DTOs immutable; and avoid concatenating large prompt strings repeatedly. For local/native backends, move heavyweight tensor buffers off the managed heap entirely. Microsoft documents NativeMemory for native memory allocation and aligned allocation, and ONNX Runtime’s C# guidance explicitly recommends OrtValue reuse for fixed-size numeric tensors because it reuses underlying buffers, pins managed buffers, and reduces data transfer overhead.

For compute-heavy local inference, the attached references are directionally correct: use aligned unmanaged allocations, memory-mapped model weights, layout-aware prepacking, paged KV caching, and SIMD/hardware intrinsics for hot kernels. Microsoft’s SIMD guidance recommends System.Numerics.Vector<T> where possible and runtime checks such as Vector.IsHardwareAccelerated; the intrinsics API exposes Avx2.IsSupported-style capability checks; and llama.cpp’s official materials emphasize quantization across multiple bit widths, concurrency, and OpenAI-compatible serving for GGUF models.

If you support ONNX locally, treat opset/version compatibility as a first-class policy. ONNX is strongly typed, does not support implicit casts, and every graph carries opset versions that determine operator semantics. That means your local-backend compatibility gate should reject unsupported opsets or domains before execution begins.

For managed runtime tuning, prefer server GC in the API tier and use low-latency modes sparingly and only after measurement. Microsoft documents that server GC is intended for server applications that need throughput and scalability, uses dedicated threads per logical CPU, and is faster on the same heap size; SustainedLowLatency suppresses some generation 2 collections but can increase heap size and fragmentation, so it is appropriate only for measured, contained latency windows rather than as a blanket setting.

Native interop should be explicit and narrow. Source-generated P/Invoke via [LibraryImport] is the right interop baseline for CUDA, DirectML, vendor tokenizers, or specialized local kernels. Native AOT is attractive for isolated sidecars and constrained environments because it reduces startup time and memory and runs without JIT, but Microsoft also documents serious limitations, including no dynamic loading and no runtime code generation. That makes Native AOT a fit for stable worker binaries, not for reflection-heavy plugin hosts.

Security, governance, and compliance posture

The security design should assume a multi-tenant enterprise service exposed to untrusted prompts and trusted internal systems. That means you need separate controls for caller auth, service identity, provider credentials, data handling, tool invocation, and LLM-specific abuse patterns such as prompt injection and insecure output handling. OWASP’s LLM Top 10 remains a practical threat-modeling taxonomy for this layer, covering prompt injection, insecure output handling, model denial of service, sensitive information disclosure, excessive agency, and model theft.

Authentication should be layered. For external callers, use OIDC/JWT validation at the edge, with tenant and model authorization policies in the runtime. For outbound provider access, use the strongest identity offered by the provider. OpenAI uses bearer authorization headers; Azure OpenAI explicitly recommends token-based authentication over API keys and documents Microsoft Entra ID and managed identities; Gemini’s public API examples use the x-goog-api-key header. Where provider workload identity or managed identity is available, prefer it over long-lived static secrets.

Secrets should never live in application code, container images, or long-lived config maps. Microsoft documents Azure Key Vault as a centralized service for secrets, keys, and certificates, with authentication and authorization gates; OWASP’s secrets management guidance recommends centralization, standardization, automated rotation, dynamic secrets where possible, and reducing the time secrets remain in memory.

The threat model below is the minimum viable enterprise security map.

At the transport and data layer, require TLS in transit, encryption at rest for retained conversation state, tenant-scoped cache keys, and explicit retention policies for prompts, responses, and telemetry. OpenAI and Anthropic both document provider-specific caching and retention semantics, so your runtime must treat retained provider-side state as a configurable policy choice rather than an implicit default.

For governance, adopt NIST AI RMF as the umbrella for AI trustworthiness and NIST SSDF as the secure software delivery baseline. NIST describes the AI RMF as a voluntary framework to incorporate trustworthiness considerations into the design, development, use, and evaluation of AI systems, while the SSDF provides a core set of high-level secure software development practices intended to reduce vulnerabilities and mitigate their impact. That pairing fits an enterprise CE platform very well: AI RMF for model/runtime governance, SSDF for implementation and delivery discipline.

Testing, quality engineering, and CI/CD

This runtime should be validated as both a software platform and an LLM integration surface. That means the test strategy cannot stop at ordinary unit tests. It needs unit, integration, contract, property-based, fuzz, chaos, performance, and end-to-end coverage, with promotion gates tied to the risk of the change. NIST SSDF explicitly recommends integrating secure development practices into the SDLC, which supports treating the CI/CD system as an enforcement mechanism rather than a convenience tool.

A practical test matrix looks like this:

The most valuable harness pattern is a deterministic fake-adapter stack. Build FakeModelAdapter, FakeTokenizer, and FakeClock implementations for domain-level tests; a record/replay HTTP harness for provider contract tests; and a stream event replayer for SSE parser validation. For local backends, build a golden-corpus differential harness against reference outputs and a benchmark harness that captures throughput, latency, and memory regression. The attached references are especially valuable for this local path because they frame the correctness risks around quantization layouts, memory ownership, KV cache semantics, and native interop.

Provider contract tests should specifically verify:

• header handling for rate limits and retries,
• structured output schema enforcement,
• streaming delta ordering and terminal events,
• usage extraction and cached-token handling,
• error normalization into internal codes.

That focus is grounded in the provider docs: OpenAI documents structured outputs, response error codes, rate-limit headers, and random exponential backoff; Anthropic documents retry-after, rate-limit headers, and SSE error events; Gemini documents project-level rate limits and SSE generation endpoints.

A representative unit-test pattern is below.

using System.Collections.Generic;
using System.Runtime.CompilerServices;
using CeRuntime.Contracts;
using Xunit;

namespace CeRuntime.Tests;

/// <summary>
/// Verifies orchestration behavior using deterministic fakes.
/// </summary>
public sealed class InferenceOrchestratorTests
{
    [Fact]
    public async Task ExecuteAsync_returns_normalized_response_from_adapter()
    {
        // Arrange
        var adapter = new FakeModelAdapter();
        var tokenizer = new PassThroughTokenizer();
        var promptManager = new PassThroughPromptManager();
        var orchestrator = new InferenceOrchestrator(adapter, tokenizer, promptManager);

        var request = new InferenceRequestDto
        {
            Tenant = "tenant-a",
            Model = "gpt-like-model",
            Messages =
            [
                new MessageDto
                {
                    Role = "user",
                    Content = "Say hello."
                }
            ]
        };

        // Act
        InferenceResponseDto response = await orchestrator.ExecuteAsync(request, CancellationToken.None);

        // Assert
        Assert.Equal("adapter-response-1", response.Id);
        Assert.Equal("hello", response.OutputText);
        Assert.Equal("fake-provider", response.Provider);
        Assert.Equal(3, response.Usage.InputTokens);
        Assert.Equal(1, response.Usage.OutputTokens);
    }

    private sealed class FakeModelAdapter : IModelAdapter
    {
        public string ProviderName => "fake-provider";

        public Task<InferenceResponseDto> GenerateAsync(
            InferenceRequestDto request,
            CancellationToken cancellationToken)
        {
            var response = new InferenceResponseDto
            {
                Id = "adapter-response-1",
                Model = request.Model,
                OutputText = "hello",
                FinishReason = "stop",
                Provider = ProviderName,
                Usage = new UsageDto
                {
                    InputTokens = 3,
                    OutputTokens = 1,
                    CachedInputTokens = 0,
                    EstimatedCostMicros = 25
                }
            };

            return Task.FromResult(response);
        }

        public async IAsyncEnumerable<InferenceEventDto> StreamAsync(
            InferenceRequestDto request,
            [EnumeratorCancellation] CancellationToken cancellationToken)
        {
            yield return new InferenceEventDto
            {
                Type = "delta",
                Delta = "hel"
            };

            yield return new InferenceEventDto
            {
                Type = "delta",
                Delta = "lo"
            };

            await Task.CompletedTask;
        }
    }

    private sealed class PassThroughTokenizer : ITokenizer
    {
        public ValueTask<TokenCountResult> CountTokensAsync(
            InferenceRequestDto request,
            CancellationToken cancellationToken) =>
            ValueTask.FromResult(new TokenCountResult(3));

        public ValueTask<InferenceRequestDto> TruncateAsync(
            InferenceRequestDto request,
            int budget,
            CancellationToken cancellationToken) =>
            ValueTask.FromResult(request);
    }

    private sealed class PassThroughPromptManager : IPromptManager
    {
        public ValueTask<PromptEnvelope> RenderAsync(
            InferenceRequestDto request,
            CancellationToken cancellationToken) =>
            ValueTask.FromResult(new PromptEnvelope(request));

        public ValueTask<PromptEnvelope> CompactAsync(
            InferenceRequestDto request,
            int budget,
            CancellationToken cancellationToken) =>
            ValueTask.FromResult(new PromptEnvelope(request));
    }
}

The CI/CD pipeline should enforce a strict progression: restore/build → static analysis → unit/integration/property/fuzz tests → provider contract tests → performance regression tests → image build/signing/SBOM → canary deployment → post-deploy verification. Where provider credentials are required, use short-lived service identities or workload identity patterns instead of embedded long-lived secrets; Azure’s managed identity guidance is directly aligned with that approach.

Deployment, observability, versioning, and risk management

The production deployment model should separate the CE API tier from any local compute tier. The API tier is a horizontally scalable stateless service; the local inference tier, if present, is a specialized worker pool with different memory and scheduling characteristics. This separation lowers blast radius, simplifies autoscaling, and prevents native/backend concerns from contaminating the API process. That recommendation is strongly supported by the attached references, which show how far local inference pulls the design toward unmanaged memory, driver interop, and custom scheduling.

For Kubernetes deployment, use Deployments with rolling updates, readiness probes, liveness probes, and Horizontal Pod Autoscaling. Kubernetes documents that readiness probes prevent traffic from reaching not-yet-ready containers, liveness probes restart broken ones, Deployments support rolling updates via maxUnavailable and maxSurge, and HPA can scale on both standard and custom metrics.

Observability should be based on OpenTelemetry from day one. OpenTelemetry .NET documents support for traces, metrics, and logs, and marks the three major components as stable. That is enough to justify a single telemetry contract spanning request traces, provider calls, stream lifecycle events, queue depth, token usage, cache hits, cost estimates, and local-worker performance counters.

The minimum useful SLI/SLO set is:

Versioning should be two-dimensional: runtime version and model version must evolve independently. For provider-hosted models, store a ModelManifest that records provider, endpoint family, model identifier, capability flags, tokenizer version, prompt cache capability, streaming support, tool support, and deprecation status. For local ONNX backends, include IR/opset compatibility; for GGUF backends, include quantization family, tokenizer artifact version, and backend requirements. This is not optional: ONNX graphs carry explicit opset rules, provider models change over time, and Azure’s current documentation makes clear that model availability and API capability are versioned surfaces.

A safe migration strategy is:

• shadow traffic for adapter or model swaps,
• golden-corpus regression on representative prompts,
• canary rollout by tenant or route,
• dual manifest support during transition,
• automatic rollback on SLO breach,
• explicit deprecation windows for contract changes.

The risk register below reflects the most material enterprise concerns.

Open questions and limitations

This report is intentionally high-confidence and architecture-first. A few decisions still depend on constraints that were not specified in the prompt.

If local inference is a hard requirement from day one, you should decide early whether the first local backend is ONNX Runtime C# or a GGUF-native worker. ONNX gives you a more standardized model contract and strong C# bindings; GGUF aligns more directly with the attached llama.cpp-oriented references and quantized local LLM serving.

If hard compliance requirements exist, such as region-locked processing, PHI, export control, or customer-managed key mandates, the final deployment topology may shift materially toward Azure OpenAI with Entra ID and managed identities, or toward a local-only controlled backend. The current evidence supports those paths, but the exact governance posture still depends on the customer environment and data classification.

If extreme throughput is the primary driver, the batching, cache, and local-worker design should be benchmarked against your actual prompt distributions before adopting continuous batching or specialized low-latency GC modes. Microsoft’s GC guidance is explicit that low-latency modes can increase fragmentation and memory pressure, and SIMD/intrinsics only help when the surrounding memory behavior is disciplined.