Executive Recommendation

Full report

On this page

Architect the NPC chat system as event-driven and quota-aware. NPCs should only invoke LLM calls in response to human-driven triggers (chat messages or game events), and remain silent otherwise (especially in empty rooms). Core game logic (rules, puzzles, movement) should run deterministically (FSMs, utility AI, scripted behaviors) and only use the LLM for flavor text. For example, one design calls a symbolic decision engine (e.g. “reject bribe”) and then prompts the LLM only to generate the dialogue line. We should also tier model usage by NPC role: use smaller, cheaper models for background NPCs (1–3B parameters) and larger models only for key characters. This “LOD-of-intelligence” tiering dramatically cuts cost in practice. Apply token-aware rate limiting and budgeting: limit calls and tokens per NPC/room/game, since identical-seeming requests vary widely in cost. In short, only call models on-demand, use caching/summarization to trim context, and enforce strict per-NPC and per-game limits so that even heavy use stays within a budget comparable to fixed costs like voice actors.

Call-Admission Decision Table

Table: Admission policy for NPC LLM calls. We admit human-driven requests if under per-NPC and per-game budgets; we coalesce or delay overlapping triggers to avoid duplicate work; and we prohibit calls in empty rooms or when limits are hit. This follows token-aware limiting (favoring token count over request count) as recommended in AI platforms.

Suggested Budgets & Timeouts

• Per-Call Limits:
• Token limits: e.g. 2,048–4,096 input tokens (context + memory) and ∼256–512 output tokens per call. (Practical context limit is often ~8–16 turns of chat.)
• Timeout: 10–15 seconds wall-clock per model call, then abort. (This keeps game responsiveness acceptable.)
• Per-NPC Budgets:
• Call rate: ~1 call per NPC every 5–10 seconds (6–12 calls/minute) maximum.
• Token budget: e.g. 50k–100k tokens per NPC per 30 minutes.
• Memory retrieval: max ~5 memory items per call to limit context size.
• Per-Room/Game Budgets:
• Concurrent NPCs: ≤6 active NPCs per room as spec.
• Call rate: e.g. 30–60 LLM calls total per minute per room (across all NPCs), enforced by token-bucket or sliding-window counters.
• Overall tokens: e.g. 500k tokens per game per hour.
• Timeouts and Degradation:
• Queue wait: If calls exceed concurrency (e.g. GPU busy), queue up to a few seconds (with backpressure).
• Rate-limit: Return a polite canned response or skip if an NPC is still formulating or if quotas are used up.

These values are tunable defaults; actual limits depend on model latency and cost targets. The key is to set a clear “budget” of calls/tokens per NPC and game session, after which the system downgrades NPC behavior.

Workload Estimates with Formulas

Let H = number of active human players, N = NPC count, r = average messages per player per minute, and e = avg NPC responses per player message. Then over T = 30 minutes:

• Call count: approximately $$\text{Calls} = H \times r \times T \times e.$$
• Token usage: if each call consumes ≈ $T_{in}$ input and $T_{out}$ output tokens, total tokens ≈ $\text{Calls} \times (T_{in} + T_{out})$.
• Cost: let $C$ = cost per 1,000 tokens (symbolic). Then cost ≈ $C \times \dfrac{\text{total tokens}}{1000}$.

Using sample assumptions ($T_{in}\approx400$, $T_{out}\approx100$ tokens):

• Quiet (light): H=1, N=1, $r≈0.5$/min (1 chat/2 min), e≈0.5. Then Calls≈$1×0.5×30×0.5=7.5≈8$. Tokens≈$8×500=4{,}000$. Cost≈$4C$. (Nearly zero – NPC rarely speaks.)
• Typical: H=5, N=3, $r≈1$/min, e≈1. Then Calls≈$5×1×30×1=150$. Tokens≈$150×500=75{,}000$. Cost≈$75C$. For $C≈\$0.03$ (per 1k), that’s ~$2.25.
• Busy: H=10, N=6, $r≈2$/min, e≈1.5. Calls≈$10×2×30×1.5=900$. Tokens≈$900×500=450{,}000$. Cost≈$450C$ (~$13.50 at \$0.03/k).
• Worst-Case: (Adversarial) H=10, N=6, $r≈5$/min (heavy spam), e≈2. Calls≈$10×5×30×2=3{,}000$. Tokens≈$1{,}500{,}000$. Cost≈$1{,}500C$ (~$45).

(These are illustrative; actual usage will vary. The system’s budget (e.g. 500k tokens) would cut off before the worst case.)

Degradation Ladder

If NPC interaction budgets are exhausted, gracefully fall back through these tiers:

1. Full LLM replies (normal): Rich, context-aware responses using memories and conversation history.
2. Restricted LLM mode: Trim context (e.g. fewer memory items), limit output length, or switch to a smaller model to preserve budget.
3. Constrained/templated text: Use rule-based or template responses. For example, if an NPC is “interrupted”, return a canned phrase (“I need to think...” or a nod). NPCs might acknowledge input in generic terms (e.g. “Interesting...” or emoji) without new facts.
4. Silent / very generic: If calls still disallowed, NPC offers no new dialogue or only the minimal deterministic action (e.g. completing its turn of conversation).
5. Unresponsive endpoint: Ultimately, if absolutely over budget, NPCs remain silent until the session resets. (This preserves game integrity by making NPCs inert rather than hallucinating.)

Each step down reduces LLM usage. We can signal degrade progress to designers (e.g. colors or system flags) so NPCs don’t break immersion abruptly. This ladder ensures the game is still playable (players might notice “the NPC seems distracted”), but prevents runaway costs.

Cost Amplification & Denial-of-Wallet Risks

• Noisy-neighbor abuse: A single fast-talking player could induce many NPC calls. Mitigation: per-NPC call caps and token-rate limiting (e.g. token bucket) ensure no one player or NPC monopolizes resources.
• Prompt loops: Malicious inputs might provoke recursive or expensive behavior (e.g. an NPC that calls another NPC repeatedly). Use guards: reject or sanitize triggers that would create loops. Also apply sliding-window checks to detect repeats.
• Flooding: If dozens of messages arrive (e.g. at scene load), we coalesce or drop some calls. Introduce exponential backoff: if an NPC is already “thinking”, ignore new requests until a delay (similar to HTTP 429 + Retry-After).
• Memory overload: Endless memory recall (game secrets) costs tokens. Limit memory retrieval per call. Disallow cross-game memory leaks by tagging memory with game ID (as required) so no private hints leak.
• Rate-limit evasion: Attackers might send many tiny messages to slip under RPS caps. By focusing on token-rate, even many short messages incur cost. We use token-bucket logic (consuming tokens per call) so attackers can’t cheaply bypass limits.
• Replay/jailbreak: An adversary might try different prompts to incite maximum output. Use timeouts, output length caps, and constraint checking (as in [34]) to avoid runaway generations.

In all cases, the mitigations are tied to telemetry (see below) and quotas. By throttling based on real resource use rather than just counts, we prevent “denial-of-wallet” where a user or bug explodes costs.

Telemetry Metrics

To tune and monitor the system without logging private chat content, collect aggregated and per-run metrics:

• Usage & Cost: Count of LLM calls, total tokens consumed per NPC and per game session. (As one guide advises, track “AI-specific usage… token counts”.)
• Latency & Errors: Response time per call and number of call timeouts or failures.
• Queue/Backpressure Stats: Instances where calls were deferred or dropped due to rate limits.
• Player & NPC Activity: # of active players and NPCs per room, messages per minute, idle vs active time. (No transcript text is stored.)
• Tier distribution: Which model tier was used (e.g. 3B vs 8B) and how often.
• Memory DB metrics: Memory reads/writes per call (to detect heavy recall).
• Budget exhaustion events: How often an NPC or game hit its call/token limit and triggered a degrade step.
• Error Cases: Schema validation or prompt-rejection counts (if using constrained output/parsing).

Collect these as time-series metrics and traces (e.g. via OpenTelemetry). Dashboards can show trends (e.g. spikes in calls per game) and alarms when usage nears thresholds. This ensures we spot runaway costs and adjust budgets — without ever logging the content of player or NPC messages.

Sources: Industry guidance on AI agent rate limiting and observability; game-AI architecture best practices (neuro-symbolic pipelines, model tiering, memory limits). The above plan synthesizes those patterns for a safe, cost-effective NPC chat system.