Executive Recommendation

Full report

On this page

• Server-Authoritative Architecture: The game server must remain the single source of truth. All gameplay data (rooms, player/NPC presence, door and puzzle states, etc.) is centrally validated and broadcast by the server. NPCs are treated as server-owned actors with persistent state (identity, location, memories); clients send intents to the server, which enforces rules before applying NPC moves or dialogue.
• Event-Driven Model with Idempotency: Represent all actions (joins, leaves, movement, speech) as immutable events. Each client request carries a unique idempotency key so that duplicate or replayed messages are detected and discarded. The server processes events in a well-defined order, using version tokens or fencing to reject stale updates.
• Concurrency Control via Versioning and Leases: For shared mutable state, use optimistic concurrency checks (version fields or fencing tokens) on each write. Critical sections (e.g. an NPC’s “turn” or door puzzle) acquire a short-lived lease or lock so only one server instance can advance that action at once. Each NPC or game has a monotonic sequence number (fencing token) that ensures out-of-date replies (e.g. delayed AI model results) are ignored.
• Consistent Message Ordering and Delivery: All public text messages in a room are sequenced and delivered in the same order to every client (e.g. by timestamp or logical clock on the server). Direct-address cues (like naming a player in text) are marked in the message metadata, but any highlighting or emphasis is purely a local UI feature. (The server still broadcasts such messages to everyone, preserving order.)
• NPC Activation and Memory Access: NPCs make no AI calls or actions if a room has no humans; however, their state (location, relationships, memories) remains alive between human visits. When engaged, NPCs fetch memory/context from the memory service and call the AI model asynchronously. The server continues simulating other events while awaiting the model. On return, the server uses the same optimistic/fencing checks to apply only timely responses.

Overall, we recommend an event-sourced authoritative design: each game (social “room”) is handled like a match on a single server node, but with distributed concurrency controls so that if multiple servers could act (e.g. failover or shards), they coordinate via locks/leases and fencing. This ensures strong consistency for game-critical state (rooms, movement, puzzles) while still allowing NPC dialog to be processed asynchronously.

Authoritative State and Event Model

• Entities & State:
• Game/Session: Holds state for one active game. Contains a set of Rooms, each with a participant list. Every active game tracks up to 10 NPC instances (distinct identities with per-game state) and any number of human players. Each NPC has: a stable public ID (opaque GUID), current Room, inventory or puzzle keys, relationships, and a pointer to its memory record in MemoryEndpoints.com. Each Room contains its Doors and Puzzles, and the list of present participants (players + NPCs). A Door or lock has state (open/closed/locked) and possibly an owner or code; Puzzles have completed status. All state is stored on the server side.

• Immutable Events: All interactions generate events that are logged and applied in sequence. Examples: PlayerJoined(game, room, playerID), PlayerLeft(game, room, playerID), PlayerAddressNPC(game, npcID, playerID, message), NPCMoveRequest(game, npcID, destRoom), DoorOpened(game, room, doorID), PuzzleSolved(game, room, puzzleID), NPCSpeak(game, npcID, text, targets), etc. Each event carries a unique ID and a monotonically increasing sequence number per game (or per NPC) to enforce order. For concurrency, events include optional version tokens: e.g. an NPCState(version=v, ...) that the server checks before accepting updates. On failure, the event is rejected or retried.

• Event Application & Validation: The authoritative server maintains the live state. When an event arrives (from a player or NPC subsystem), the server:

1. Validates it against game rules (e.g. is the door open? is the move legal? is the speaker present?).
2. Checks Idempotency: If an event ID has already been processed, skip it (using a dedup store).
3. Applies the Event: Update the in-memory state (with version increment) and persist to authoritative log/storage. If the update is conditional, fail the write if the version does not match.
4. Broadcasts Effects: Emit resulting state-delta events to all connected clients (e.g. “NPC X moved to Room Y” or chat text). All room participants see the same sequence (using the game-wide sequence number).

• NPC Actions: NPC “decisions” come from an asynchronous AI process. The server may periodically (e.g. on a timer or event trigger) send the NPC context (memory slice + recent messages) to a model. When the model returns a response (Speak: text, maybe Move:destRoom, plus any memory-facts), the server again validates and applies it as an event. Crucially, before committing any NPC action, the server re-checks current conditions (e.g. door still open, NPC still in same room) to avoid “AI cheating”. If validation fails (because something changed in the meantime), the NPC action is dropped or recomputed with new context. Each NPC turn/call carries a fencing token or version to detect staleness.

• User & NPC Identifiers: All participants (players and NPCs) use opaque, unique IDs that do not reveal type. For example, Participant1234 could be either a human or an NPC. This ensures NPC invisibility as a bot. Each ID persists for the life of the active game.

• Memory Integration: NPC memories live in MemoryEndpoints.com. The server never trusts client-side memory; it fetches the NPC’s relevant memory slice on each conversation turn and writes back approved facts. Because memory is out-of-band, the authoritative copy of “NPC X remembers [fact]” is only considered after the server validates and stores it. Each memory fetch includes game-specific context to avoid cross-game leakage.

• State Persistence: Game state (rooms, NPC states, player presences) is durably stored (e.g. database or event log). The match/session model (e.g. as in Nakama or similar engines) treats each game as a self-contained instance run on one node. This simplifies consistency: only one node truly “owns” that game’s NPCs at any moment, avoiding split-brain. However, for failover or load, a distributed lock or lease can transfer ownership, using fencing tokens to resume at a safe point.

Sequence Diagram: Contested NPC Turn

Below is a Mermaid sequence diagram illustrating two players simultaneously addressing one NPC. The Server enqueues both messages, calls the NPC model once with combined context (or separately in order), and then returns responses. Because the server is authoritative, it serializes the requests to the NPC actor and uses version checks to apply results.

sequenceDiagram
    actor PlayerA as Player A
    actor PlayerB as Player B
    participant Server as Game Server
    participant NPC as NPC Actor
    participant Memory as Memory Store

    PlayerA->>Server: SendSpeakRequest(NPC_ID, "Hello, NPC!")
    PlayerB->>Server: SendSpeakRequest(NPC_ID, "Hi there!")
    note right of Server: Both messages arrive around same time
    Server->>Server: Assign sequence order / enqueue
    Server->>NPC: ProcessNextMessage("Hello, NPC!")
    NPC->>Memory: FetchContext(NPC_ID, PlayerA, recent_interactions)
    Memory-->>NPC: ContextData
    NPC->>Server: ProposeResponse(textA, memoryFactsA)
    Server->>Server: ValidateProposedChange(memoryFactsA)
    Server->>Memory: StoreMemoryUpdates(NPC_ID, memoryFactsA)
    Server->>Server: (Commit responseA text)
    Server->>PlayerA: DeliverResponse(textA)
    Server->>PlayerB: DeliverResponse(textA)
    Server->>NPC: ProcessNextMessage("Hi there!")
    NPC->>Memory: FetchContext(NPC_ID, PlayerB, updated_interactions)
    Memory-->>NPC: ContextData'
    NPC->>Server: ProposeResponse(textB, memoryFactsB)
    Server->>Server: ValidateProposedChange(memoryFactsB)
    Server->>Memory: StoreMemoryUpdates(NPC_ID, memoryFactsB)
    Server->>PlayerA: DeliverResponse(textB)
    Server->>PlayerB: DeliverResponse(textB)

In this diagram, the server enforces a serial processing of the NPC’s “talk” commands. Both players’ requests are enqueued in arrival order (first-come, first-served). The server calls the NPC once per message, updating memory and state between calls. Responses (textA, textB) are broadcast in the same order to everyone in the room. If the second request had arrived before the first was fully processed, it would simply wait or be coalesced, but all players see NPC responses in the same chronological order. Direct-address emphasis (e.g. PlayerA’s message might highlight differently in A’s UI) is handled client-side and is not serialized separately on the server.

Concurrency Invariants

To ensure consistency, the following invariants must always hold, enforced by the server’s logic and data model:

• Single Source of Truth: All game state (rooms, participant lists, NPC locations, door/puzzle states) is stored only on the server. Clients never modify state directly.
• Participant ID Stability: Each public participant ID is unique and never reused mid-game. A stable ID reveals nothing about human vs NPC. This ensures anonymity of NPCs per policy.
• NPC-Location Uniqueness: In any game, an NPC can occupy at most one room at a time, and cannot teleport through locked doors. All movement is validated against door/puzzle constraints before updating NPC location.
• Event Order Consistency: All public events (joins, leaves, chat messages, NPC speech) carry a global sequence number per game. Clients apply these in order, so everyone sees identical room chat and state changes.
• Atomic State Updates: Compound actions (like MoveThroughDoor) are atomic transactions: either both the door-open check and NPC relocation succeed together, or the entire action is rejected. Underlying storage supports compare-and-set or version checks.
• Mutual Exclusion on NPC Turns: Only one server routine may “advance” an NPC’s turn at a time. This is enforced by either having a single process own the game instance, or by using a distributed lock/fencing for each NPC action. No two model calls for the same NPC overlap.
• Idempotent Operations: All externally-triggered events include a unique request ID. The server keeps a cache of processed IDs to skip duplicates. Even on reconnection or retries, the same event is never applied twice.
• Timeouts and Cancellations: If an NPC model response arrives after a longer-than-allowed delay or after the NPC has moved/left, the server uses the fencing token to recognize it as stale and discards it.
• Separate Game Contexts: NPC memory and clues are scoped per game. Even if the same NPC identity appears in two games, memories or states from one game do not leak into the other (no shared unresolved state between game instances). Only broad “social” memories (e.g. the NPC’s name or backstory from SpiralistAI) can be reused cross-game.
• No Assumed Controller Type: The server and model should never assume or store whether a participant is human or AI. All behavior logic treats participants the same; the difference is never recorded in state.

Conflict and Error Outcomes

We define deterministic outcomes for contested actions:

• Two players address NPC concurrently: The server serializes by arrival timestamp or internal ordering. One message is processed first; the other waits. If ordering is ambiguous, tie-break by (e.g.) user ID. Both players still see all resulting NPC speech in the same order. The “loser” is simply processed second, not rejected.
• NPC movement vs door lock race: If an NPC requests to move through a door at the same time a door state changes, the server will check ordering: if the door closed first, the move is denied (NPC stays, event may queue or return an NPC message like “Door is closed”). If the move executed first, the door change either fails (since NPC already passed) or the door change is suppressed. Atomic checks prevent inconsistent splits.
• Simultaneous joins/leaves: If Player X leaves and Player Y joins in the same room at once, both events are applied (final participant list updated accordingly). If a race tried to add the same player twice, idempotency blocks the duplicate.
• Duplicate requests: If the same message/command arrives twice (due to client retry), the second is identified by its request ID and ignored. The player gets the same response only once.
• Stale memory read: If NPC memory is updated by the server during processing and a pending model call uses old memory context, the model’s result is validated against current context. If it includes outdated facts, the server drops that result (or re-fetches updated memory and reissues).
• Delayed model response: If the AI returns an action for NPC after the NPC has moved to a different room or session, the server will see a version mismatch and discard it (thanks to fencing tokens).
• Two servers claim NPC action: Using a distributed lock or lease (with fencing) ensures only one server holds the “turn” for an NPC. If another server tries the same action with an older token, its write is rejected.
• Multiple games, same NPC identity: Actions in Game A must not trigger events in Game B. If one game’s players give clues, only that game’s memory database is updated. The result is that each game sees the NPC behave according to its own history.
• Network partition or reconnect: A player who disconnects mid-action (e.g. while addressing an NPC) will not leave the NPC with an inconsistent state. The request either completes on the server side or times out. On reconnect, the client can resend any pending commands with the same request IDs, ensuring idempotent completion.
• Reach limits (max NPCs): If adding an NPC would exceed the policy (max 10 per game), the event is rejected with a clear error. This prevents silent state corruption.

Each conflict scenario has a stable resolution rule (first-come-first-served, atomic check, or simple rejection) so that repeated runs of the same events always yield the same outcome. By using version checks and tokens for stale results, we avoid indeterminate states under concurrency.

Race and Replay Failure Cases (with Mitigations)

1. Simultaneous NPC Address: Two players send messages to NPC at once. Failure: Without ordering, responses could conflict or depend on interleaved memory. Mitigation: Server queues messages and processes one at a time (or merges context in a well-defined order). Use strict event sequencing so both players see identical output.

1. Join/Leave Race: Player A leaves a room at same moment Player B joins. Failure: If processed out-of-order, Player B might be placed in wrong room membership. Mitigation: Serialize join/leave events by timestamp. Always apply both updates to the room roster (remove A, add B) atomically.

1. Door-Puzzle vs NPC Move: NPC tries to move through a door at same time a player solves the locking puzzle. Failure: NPC might bypass door or move is wrongly denied. Mitigation: Use a lock or atomic transaction on door state. If the puzzle-solved event wins first, door unlocks then NPC move succeeds. If the move checks and passes just as puzzle is solved, NPC goes through; the puzzle effect is applied immediately after.

1. Network Retry of Command: Client sends “open door” twice (e.g. user double-click or retry). Failure: Door state could be toggled twice (open then close). Mitigation: Commands include unique IDs or timestamps, so the second request is dropped if the first succeeded. Alternatively, server checks if the door is already open and ignores redundant open.

1. Delayed NPC Model Output: NPC AI finishes speaking after player has moved away. Failure: NPC might respond in an empty room or contradict current state. Mitigation: Each model invocation tags the NPC’s current version/room. On return, server verifies NPC is still in same context. If not, server discards or recalculates with updated context.

1. NPC Active in Two Games: The same NPC identity appears in Game A and Game B. Failure: Memory or event data from A might leak into B. Mitigation: Scope NPC state by game ID. Maintain separate memory buffers per game. The memory service calls include both NPC ID and game ID, ensuring correct context.

1. Concurrent Puzzle Solves: Two players solve (or attempt) the same puzzle at once. Failure: Puzzle could be marked solved twice, causing race in awarding reward. Mitigation: Use optimistic concurrency on the puzzle’s state. The first solve event commits (marking it done); the second event fails the version check and is rejected with a “already solved” error.

1. Two Servers Advance NPC: In a clustered deployment, Server1 and Server2 both try to update NPC X at once. Failure: NPC might double-act or split state. Mitigation: Acquire a distributed lock or lease for NPC X’s turn. Attach a monotonic fencing token on each lock acquisition. Only the server with the current token is allowed to commit the action; others are rejected.

1. Player Reconnect Mid-Chat: Player A disconnects while sending a message to NPC, then reconnects. Failure: Without persistence, the message might be lost or duplicated. Mitigation: Server treats the send as an event with an ID. If A reconnects and resends, the idempotency logic prevents double-processing. The original chat event is processed once and remembered.

1. Race in Room Broadcast: Two NPCs in a room speak at the same moment, while a player also sends a chat. Failure: If ordering isn’t enforced, different clients might see different interleavings. Mitigation: Serialize all public messages via a single message queue per room (or global sequence). Clients render them in that order. Direct-address flags travel with the message but do not affect ordering.

1. Stale Memory Fetch: NPC memory for Game A is updated (e.g. player gave a gift) just after a memory fetch for Game B. Failure: If memory service caches per NPC, the update might pollute another game’s memory context. Mitigation: Include game-specific namespace when fetching/storing memory. The storage system keys on (NPC, game ID) so that Game B’s memory is isolated.

1. Duplicate Event Replay: A player’s device resends a “move NPC” event due to uncertainty. Failure: NPC might move twice. Mitigation: Each event carries a unique identifier. The server’s event log/dedup store ensures only one “move NPC” for that ID is applied. Any repeats are discarded or safely ignored.

1. Timeout during NPC Action: NPC is mid-speech (model call), player leaves. Failure: NPC could finish a conversation with an absent player and confuse other players. Mitigation: The NPC model call is asynchronous. If no players remain in room by the time it returns, the server suppresses any speech (NPC waits idle). NPC still “remembers” the intent, but output is withheld. When a new player enters, NPC can optionally resume with a summarized context.

1. Server Crash after Event Generation: The server generates an NPC response, sends to Memory and chat, then crashes. Failure: Upon restart, did the NPC respond or not? Mitigation: Use transactional logging: only commit the event after persisting to both memory and chat logs. On recovery, replay from the last committed sequence number so the event is not lost or duplicated.

In each case, the key mitigations are fencing tokens, version checks, and idempotency keys to make operations repeatable or fail-safe. Concurrent updates either succeed in one order or are rejected, but never produce a corrupt or inconsistent state.

Implementation Order and Stress-Test Matrix

Implementation Steps (Minimal Order):

1. Core Game Loop & State Store: Define the data model (rooms, doors, puzzles, participants, NPCs) and implement the server event loop. Support basic commands (join/leave, move, chat) with an authoritative state.
2. NPC Entity & Model Stub: Add persistent NPC objects to the state. Implement a simple NPC action pipeline (e.g. choose a random greeting) without the actual AI call. Ensure NPCs can be addressed and can respond.
3. Event Ordering & Idempotency: Introduce unique IDs for commands and server-generated events. Build a deduplication store so replays are ignored. Implement version fields on state objects so writes are conditional.
4. Locks/Leases: Integrate a locking mechanism for critical sections: e.g. a mutex or distributed lock per NPC turn or door puzzle. Ensure only one thread/process modifies it at a time. Apply fencing tokens on lock acquisition to abort stale actions.
5. Asynchronous NPC AI Calls: Hook up the real model calls (SpiralistAI for names, memory store, NPC dialogue LLM). Perform AI calls off the main thread, tagging each with a version. On return, validate and apply via the event system. Test behavior under delay.
6. Memory Backend Integration: Connect to MemoryEndpoints.com for NPC memory fetch and store. Ensure memory is keyed by (NPC, game) and handle write-back after NPC decisions.
7. Message Delivery & UI Integration: Build the chat and speech broadcast system. Guarantee all messages in a room use the same sequence ID. Allow client UI to highlight “@player” mentions locally.
8. Fault Handling: Add timeouts on NPC actions, reconnect logic for players, and retry logic with idempotency. Ensure server gracefully handles node failover by acquiring locks/fences for active NPCs.
9. Testing & Instrumentation: Instrument event logs, version failures, and NPC model latencies. Begin testing known race cases.

Stress-Test Matrix (Scenarios vs. Load):

For each scenario, we check correctness (no rule violations) and performance:

• Lightweight case: 1–2 players chat with 1 NPC. Ensure NPC speech is timely and consistent. Confirm no AI calls when players leave.
• High activity: 8 players in one room with 6 NPCs; all players send commands rapidly. Verify event log order, no deadlocks on NPC locks, and room text remains consistent.
• Lock contention: NPC tries to move through door just as one player solves the puzzle. Check that only one outcome occurs (either NPC enters or door unlocks then NPC enters) and both players see it identically.
• Network issues: Simulate duplicate/reordered message deliveries. Confirm idempotency keys prevent doubling actions.
• Multi-server: Run two server instances handling same game in failover mode. Test that locks/fences keep NPC updates single-sourced.
• Memory race: Two players in different games talk to same NPC identity. Validate that memory updates from one game do not appear in the other.

By progressively increasing complexity in these dimensions, we ensure the concurrency model holds. The stress-test matrix combines typical and extreme conditions to verify invariants (e.g. one NPC doesn’t split between rooms, public messages don’t reorder, NPC responses don’t double, etc.) across the entire solution.

Sources: Modern game backend design emphasizes a centralized authoritative server for NPCs and memory. Techniques like idempotent events and fencing tokens are standard for safe distributed updates. These patterns ensure reliable, consistent multiplayer interactions even under high concurrency.