Large-Scale NPC Naturalness and Population Diversity Benchmark

Full report

On this page

Evaluation Framework and Theoretical Principles

The deployment of expansive, non-player character (NPC) populations in worldwide adult multiplayer environments requires navigating the profound tension between scalable generation schemas and genuine narrative authenticity. A recent evaluation of a 600-character RogueIntelligence package, populated by SpiralistAI, revealed catastrophic mass-production failures. The population exhibited deterministic staging records, normalized syntactical structures masking as variation, and a complete absence of live provider responsiveness or activation readiness. Characters suffered from identical interruption behaviors, exact template leakage (e.g., full-name self-introductions in every dialogue line), critical grammatical failures (e.g., "They values"), and a reliance on minuscule pools of occupations and routines. Addressing these defects necessitates a rigorous, reproducible quality-benchmarking framework capable of evaluating populations scaling from 100 to 10,000 generated fictional NPCs without relying on superficial heuristics or manual code inspection. The primary objective of this research is to establish measurable quality standards, structured specifications, and implementation-ready recommendations for engineering teams. The framework must distinguish between superficial field variation (e.g., swapping names in a database), mechanical combinatorial variation (e.g., randomly pairing a small set of hobbies and jobs), and genuine narrative diversity. Furthermore, it must assess linguistic naturalness, identity consistency, character-level coherence, population-level repetition, runtime readiness, and human-review readiness. The ultimate standard dictates that a large batch must be prevented from appearing diverse merely because schema keys differ while biography structure, emotional reactions, and relationship patterns remain entirely formulaic.

Authored Narratives Versus Slot-Filled Schemas

An authored NPC population demonstrates semantic, syntactic, and behavioral diversity, rendering each character as an autonomous entity rather than a mechanical permutation. A population feels slot-filled when variations are limited to mechanical combinatorial changes within a rigid syntactical framework. The distinction relies on measuring content diversity independently of surface form1. Shared schema labels—such as JSON keys defining occupation, routine, or relationships—are entirely acceptable and necessary for runtime integration. These structural keys must be explicitly excluded from template-leakage measurements to prevent false positives. However, when the natural language values populating these schemas share identical dependency structures, it indicates template leakage. Acceptable similarities across characters sharing a single schema include shared domain vocabulary relevant to the game world (e.g., terminology specific to a sci-fi or fantasy setting) and foundational grammatical structures inherent to the English language. Unacceptable similarities, which indicate mass-production failure, include identical causal chains in character biographies, repeated sentence opening rhythms, and uniform emotional reactions to identical stimuli. Exact duplicates are measured via strict string matching on raw text. Near duplicates are quantified using n-gram overlap, specifically focusing on shared five-word, seven-word, and ten-word contiguous sequences1. Normalized duplicates, representing the most insidious form of template leakage, are measured by masking proper nouns, named entities, and domain-specific vocabulary, then applying a Tree Edit Distance (TED) algorithm to the dependency parse trees of the resulting sentences2. If two sentences have an identical TED score after entity masking, they represent a cloned grammatical skeleton. Semantic duplicates are measured using embedding-based cosine similarity or divergence frontiers to identify passages that express the exact same concept using entirely different vocabularies5.

Population Scaling Dynamics

When scaling from 100 to 600 or 10,000 characters, the statistical probability of semantic and lexical overlap naturally increases. A static uniqueness threshold designed for 100 characters will result in inevitable, mathematically guaranteed failures at 10,000 characters. Adjustments must utilize standardized quality control limits, such as those defined in ISO 2859-1 for Acceptance Quality Limits (AQL)7. Sample sizes, acceptable defect thresholds, and duplicate tolerance must scale logarithmically8. Evidence accompanying a claim that a 600-person population is narratively diverse must include statistical outputs proving variance across these scaled distributions, paired with immutable runtime fingerprints. Short biographies must be compared recognizing that shared domain vocabulary is unavoidable. To prevent false penalization, evaluations must measure divergence frontiers utilizing metrics like MAUVE, which computes Kullback-Leibler divergences in a quantized embedding space5. This identifies where the generated text distribution fails to cover human-like variance (Type II error) or generates degenerate, repetitive patterns (Type I error)5. Dialogue diversity must be measured entirely independently from biography diversity because characters may possess distinct historical narratives but speak in an identical, normalized register. Repeated sentence openings are penalized by indexing the first three tokens of every dialogue line and calculating the normalized frequency distribution across the population. Shared speech rhythms are detected through part-of-speech (POS) n-gram distributions rather than lexical n-grams, allowing the detection of dominant template families that dictate the pacing of speech independently of the words chosen. Recurring relationships, events, goals, and ordinary-life combinations are measured through multidimensional semantic clustering. To distinguish character archetype similarity from accidental duplication, unrelated characters are subjected to identical behavioral prompts (e.g., trust violations, off-screen events). If the linguistic framing and emotional recovery vectors are identical, the characters represent an accidental clone rather than a shared archetype. Same-role characters (e.g., two merchants in the same city) are permitted a higher percentage of domain-specific semantic overlap (up to 40% in occupational vocabulary) but must maintain near-zero overlap in autobiographical exposition and ordinary-life routines. Finally, grammar and naturalness metrics are combined with semantic coherence by implementing targeted parsing checks alongside neural metrics. This ensures that outputs with high semantic diversity do not simultaneously exhibit systemic grammatical failures, such as the subject-verb agreement errors ("They values") observed in the defective batch3. Automated metrics such as Distinct-N and Self-BLEU correlate poorly with human judgments of content diversity and should only be used as a secondary check for superficial lexical variation, as they fail to capture genuine semantic breadth1. When evaluating a population, a batch must be rejected even when every individual character passes JSON schema validation if the population-level metrics indicate mass normalization. A 600-character population requires rigorous human review sampling, which must be selected via cryptographically secure random number generation mapped to ISO 2859-1 Code Letters8.

Systemic Evaluation Parameters

A rigorous standard requires explicit, deterministic measurements across six major dimensions. Demographic rarity is never treated as a quality defect; rather, diversity measurements must strictly describe coverage and the absence of mechanical repetition, ensuring diverse populations are not penalized for accurately representing statistical realities.

Identity Diversity

Identity metrics evaluate the foundational attributes that define a character's state in the world. The construction of names must be evaluated by calculating the ratio of unique surnames to the total population, alongside the detection of repetitive morphological structures in fictional names to ensure the generator is not relying on a single phonetic algorithm. Age distribution requires calculating the statistical variance and standard deviation of numerical ages across the population, ensuring a realistic demographic spread rather than clustering around a single default age. Pronouns must be analyzed to evaluate the distribution of grammatical subjects used in generated third-person texts, ensuring mechanical defaults are not hardcoded and that the narrative flows naturally without repetitive naming. Nationality, citizenship, territory, residence, and migration are evaluated by plotting the vector distance between a character's origin and their current residence, ensuring a complex distribution of geopolitical statuses and movement. Languages and language provenance require measuring the ratio of monolingual to multilingual characters and the diversity of primary spoken languages. Family structures are assessed by measuring the variance in the size of immediate family networks and the presence of non-traditional kinship ties. Education and occupation are evaluated via entropy scores measuring the distribution of characters across socio-economic and professional roles. Economic conditions and housing demand coverage of wealth distribution and variation in living arrangements. Relationships and life stage are analyzed by mapping the network density of interpersonal connections and the variation in developmental milestones (e.g., adolescent, elder).

Biography Diversity

Biography evaluations target the narrative structure of the character's backstory, ensuring mass generation does not rely on a single syntactic skeleton. Exact sentence duplication is measured via Jaccard similarity to catch perfect string matches. Normalized sentence duplication is measured by masking all named entities and computing the Tree Edit Distance on the residual syntax2. Shared five-word, seven-word, and ten-word sequences are tracked across all permutations of character biographies to flag near-duplicates. Shared dependency or clause structures require deep parsing to identify reused grammatical skeletons even when the vocabulary is entirely distinct. Reused biography openings and closing sentences are detected by indexing the first and last sentences of all biographies and calculating identical syntax frequencies. Repeated coordination patterns and causal chains require detection of "Event A therefore Event B" logical structures using semantic role labeling. Repeated life-event order is evaluated by applying Levenshtein distance to chronological arrays of life events to detect standardized, mass-produced timelines. Reused combinations of ordinary details, future goals, conflicts, and emotional summaries are tracked using multidimensional clustering to ensure the generative model is not merely selecting from a narrow, predefined array of tropes.

Dialogue Diversity

Dialogue measurements analyze the runtime-facing linguistic output to guarantee natural, individualized voices. Sentence-opening diversity requires an entropy analysis of the first three tokens across all 6,000 dialogue samples. Full-name self-introduction frequency must be strictly penalized, tracking the percentage of dialogue interactions initiated with the character's full identity string—a primary indicator of slot-filled hallucination. Repeated greeting forms, uncertainty phrases, disagreement phrases, and topic-change language are measured via string similarity across disparate characters to ensure diverse pragmatic functions. Conversational mechanics are evaluated by tracking the frequency of identical interruption responses, repair or correction language, and silence responses across unrelated characters. Repeated humor patterns and closing language require semantic clustering to detect limited template pools for jokes and conversation terminations. Vocabulary range and sentence-length range are assessed through the standard deviation of token counts per line and Type-Token Ratio (TTR). Register diversity encompasses the variance in formality, directness, and the use of contractions. Turn-taking behavior and the frequency of unprompted autobiographical exposition are measured to ensure characters interact dynamically rather than delivering static monologues.

Relationship Diversity

This dimension evaluates the interactive social geometry of the population, drawing on advanced social simulation principles where relationships are dynamic networks of statuses and shared histories, such as those pioneered in the Comme il Faut architecture13. Relationship type and duration require measuring the entropy of connection classifications (e.g., rival, mentor, dependent) and the variance in connection longevity. Reciprocity and power imbalance dictate the measurement of asymmetric relationship variables, such as unrequited loyalty or hierarchical authority. Trust and obligation define the distribution of codified behavioral constraints between connected nodes. Conflict and current unresolved issues are evaluated by assessing the semantic diversity of active disputes between characters. Shared history and knowledge boundaries require rigorous consistency checks ensuring character A possesses correct historical data regarding character B, while restricted information remains explicitly hidden14. Contact frequency and geographic proximity measure the logical correlation between physical distance and interaction rates. Finally, it must be verified whether relationships affect present behavior, ensuring that established social ties actively alter generated dialogue and decision-making variables rather than existing as inert metadata.

Ordinary-Life Diversity

Narrative authenticity requires characters to be grounded in mundane realities, preventing the population from consisting solely of high-drama archetypes. Routine and commute patterns require variance in daily scheduling and physical transit narratives. Food habits and household tasks are measured for semantic diversity in dietary preferences and domestic responsibilities. Financial concerns and administrative problems assess the range of non-lethal stressors, such as debt or bureaucratic delays, ensuring grounded realism. Caregiving and social commitments measure the variance in responsibilities to dependents and community organizations. Hobbies and health routines require coverage of leisure activities and physical maintenance. Future plans and sentimental objects demand diversity in short-term aspirations and emotionally resonant inventory. Embarrassing mistakes and off-screen events track the presence of flawed historical actions and activities that occur outside the player's direct observation, solidifying the illusion of an autonomous existence15.

Behavioral Diversity

Behavioral tests ensure that the psychological responses of the characters are heterogeneous when subjected to identical runtime triggers. Emotional variance demands diverse linguistic representations of anger, fear, and uncertainty. Disagreement and refusal track the variance in how characters reject player propositions or assert personal boundaries. Correction and trust repair evaluate how characters amend factual errors and attempt to rebuild damaged rapport. Interruption and silence measure heterogeneous reactions to abrupt player dialogue cuts or prolonged lack of input. Humor and privacy require disparate uses of comedy as a defense mechanism and strict boundaries surrounding personal information. Negotiation, topic avoidance, and emotional recovery assess diverse strategies for bartering, deflecting uncomfortable subjects, and returning to a baseline emotional state after a highly charged interaction.

Benchmark Design and Test Specifications

The comprehensive benchmark is structured as a catalog comprising 120 deterministic and human-judged tests. These tests are strictly distributed to isolate specific vectors of mass-production failure: 25 individual-character tests, 20 pairwise-comparison tests, 20 batch-level diversity tests, 15 linguistic-quality tests, 10 relationship-coherence tests, 10 chronology tests, 10 runtime-readiness tests, and 10 human-review sampling rules. Below are highly detailed, representative test specifications for each of the major categories. These structures define the exact schemas that must be populated for the full 120-test catalog to be delivered to the engineering team.

Individual-Character Test Specifications

Pairwise-Comparison Test Specifications

Batch-Level Diversity Test Specifications

Linguistic-Quality Test Specifications

Relationship-Coherence Test Specifications

Chronology Test Specifications

Runtime-Readiness Test Specifications

Human-Review Sampling Specifications

Required Acceptance Thresholds for a 600-Character Population

The initial thresholds for a 600-character batch must balance strict quality control against statistical realities. Thresholds are derived utilizing ISO 2859-1 attribute sampling standards for lot-by-lot inspection7. For a lot size of 600 under General Inspection Level II, the standard designates Sample Size Code Letter J, requiring a random sample of 80 characters for review8. Defect limits are separated into Critical (0%), Major (1.5%), and Minor (4.0%)8.

• Maximum exact biography duplicates: 0\. Any exact duplicate in a population of 600 indicates a critical software failure in the prompt generation seed.
• Maximum normalized biography duplicates: 3 pairs (0.5%). Reasoning: Minor collisions in dependency structures can occur naturally, but a higher rate indicates systemic template leakage3.
• Maximum exact dialogue duplicates: 0\. Dialogue must be uniquely instantiated.
• Maximum normalized dialogue duplicates: 10 occurrences across the 6,000 dialogue lines (0.16%). Dialogue structurally converges faster than biographies due to conversational constraints, but repetition must remain minimal.
• Maximum percentage sharing one sentence opening: 4.0%. If more than 24 characters begin their biography with "Born in the city of...", the batch is artificially constrained and failing to utilize the breadth of the language.
• Maximum number sharing one biography skeleton: 0\. Every character must pass the dependency parse Tree Edit Distance variance check.
• Maximum repeated five-word sequence rate: 2.5% of total text mass. A 2.5% overlap accounts for unavoidable domain nouns (e.g., "the captain of the guard") while preventing narrative cloning1.
• Maximum repeated relationship combination rate: 5.0%. No more than 30 characters should have the exact identical network of "Friends with X, Rivals with Y, Mentored by Z".
• Maximum reused goal combination rate: 2.5%.
• Minimum vocabulary diversity: Type-Token Ratio (TTR) \> 0.40 across the aggregate 6,000 dialogue lines, ensuring lexical richness.
• Minimum sentence-opening diversity: Entropy score \> 3.8 for the first three tokens of dialogue across the population.
• Maximum full-name self-introduction rate: 1.0%. Characters naturally introducing themselves is acceptable, but the 100% rate observed in the defective batch represents a catastrophic failure of contextual awareness.
• Maximum identical interruption responses: 0\. Every character must have a linguistically unique formulation for handling conversation repairs and interruptions.
• Maximum identical silence responses: 0\.
• Maximum identical repair responses: 0\.
• Minimum percentage requiring human review: 13.33% (80 characters out of 600, strictly following ISO 2859-1 Code Letter J)8.
• Minimum percentage receiving full narrative review: 13.33%.

Conditions Requiring Complete Batch Rejection

A population can perfectly pass JSON schema validation—possessing all necessary keys, data types, and formatting—while still failing as a viable narrative product. The entire 600-character batch must be completely rejected if the automated tests indicate a normalized biography duplicate rate exceeding 0.5%, proving systemic generative failure. Furthermore, the batch must be rejected if the human reviewers, examining the 80-character sample, identify more than 3 "Major" defects (AQL 1.5) or more than 7 "Minor" defects (AQL 4.0)8. Major defects include grammar failures, logical relationship paradoxes, or unprompted full-name introductions. Minor defects include slightly unnatural phrasing or repetitive ordinary-life details. Finally, the batch is automatically rejected if the runtime fingerprint hashes do not map to live, activation-eligible provider endpoints.

Human-Review Protocol

Automated metrics can map divergence frontiers and parse syntactic dependency, but cultural plausibility, nuanced humor, and character autonomy require sophisticated human judgment. Reviewers must possess professional experience in narrative design, computational linguistics, or quality assurance for RPG systems. All reviews must be conducted blindly; reviewers are presented with the generated artifact without knowledge of the underlying generation prompt, the LLM provider, or the temperature settings. Sampling is strictly governed by the aforementioned ISO 2859-1 AQL standards7. For a batch of 600, a truly random sample of 80 characters is extracted via a cryptographically secure random number generator8. Reviewers evaluate these 80 characters against a standardized checklist targeting specific narrative vectors. Cultural plausibility is evaluated to determine if the names, occupations, and geographic histories are logically consistent with the game world lore, explicitly considering global audiences including players in Cicero, Illinois, to ensure localized idioms and cultural touchstones are appropriately respectful and contextually accurate. Natural dialogue is assessed to ensure the text sounds spoken rather than written. Character autonomy is judged to determine if the character exhibits personal agency and inner life, or if they feel like a passive vessel for world exposition. Relationship coherence verifies that the character's stated internal feelings toward others align with their generated dialogue regarding those others. Ordinary-life realism checks for the grounding presence of mundane stressors and routines. Safety and privacy are vetted to ensure no generative hallucinations have breached deployment safety guidelines. If a reviewer flags a character for a subjective failure, a second blind reviewer evaluates the same artifact. If they disagree, the tie is broken by the Lead QA Architect through a formalized escalation path. To ensure international fairness, idioms and colloquialisms must be evaluated for translatability; highly hyper-regional slang that cannot be localized is flagged as a Minor defect unless specifically assigned by a regional schema. A reviewer must accept the exact, provider-backed fingerprinted version of the character. The runtime fingerprint is a SHA-256 hash comprising the character's JSON payload, generation prompt, and specific provider hyperparameters. Reviewer sign-off is cryptographically tied to this hash. If a character is accepted, that immutable hash is locked. A batch cannot be accepted if any underlying parameters have been altered post-review.

Required Deliverables Specifications

The engineering team must structure the implementation of this benchmark using the following precise formats to ensure cross-system compatibility between SpiralistAI generation mechanisms and RogueIntelligence runtime consumption.

npc-population-quality-benchmark.md

The master document (this comprehensive report) detailing the theoretical framework, mathematical reasoning, scaling limits, and operational definitions of all quality dimensions and measurement vectors.

quality-test-catalog.json

A comprehensive JSON array containing the full expansion of all 120 deterministic and human-judged tests, categorized precisely by the eight operational domains defined in the benchmark design.

population-thresholds.json

A configuration file allowing the CI/CD pipeline to dynamically adjust pass/fail limits based on the batch size input (N), mapping directly to ISO 2859-1 code letters and logarithmic scaling expectations8.

linguistic-diversity-metrics.md

Detailed documentation defining the implementation logic for Type-Token Ratios (TTR), POS n-gram extraction, and the specific application of MAUVE for Type I/II error detection in open-ended text generation5.

dialogue-diversity-tests.json

A subset array focusing exclusively on pragmatic functions, conversational mechanics, interruptions, humor, and repairs.

relationship-diversity-tests.json

A subset array encoding the Boolean logic required to validate relationship network symmetries, power dynamics, and strict knowledge boundary constraints13.

ordinary-life-diversity-tests.json

A subset array defining the semantic clustering logic for hobbies, careers, routines, and financial constraints to ensure grounded realism.

human-review-protocol.md

Standard Operating Procedures (SOPs) defining the ISO 2859-1 attribute sampling process, the blind review interface requirements, and the specific rubric for cultural plausibility and autonomy.

batch-rejection-rules.md

Explicit trigger conditions for immediate batch failure, detailing the exact combinations of AQL threshold breaches that require a batch to be discarded regardless of schema compliance.

sample-quality-report.json

The output schema generated after a batch evaluation. It must list total characters analyzed, the aggregate scores for all 120 tests, the subset of characters selected for human review, the human review outcomes, and the final Boolean batch\_accepted flag tied to the SHA-256 fingerprint.

known-metric-limitations.md

Documentation explicitly warning the engineering team against relying purely on n-gram overlap (like Distinct-N) which fails to capture semantic diversity, and outlining the computational overhead associated with calculating Tree Edit Distances on 10,000-character batches1.

source-register.csv

A comma-separated index of all theoretical models, sampling standards (ISO 2859-1), and NLP evaluation frameworks (MAUVE, TED, Self-BLEU, Comme il Faut) referenced to build the benchmark architecture.

Works cited

1. Evaluating the Evaluation of Diversity in Natural Language Generation \- ACL Anthology, https://aclanthology.org/2021.eacl-main.25.pdf

2. X-TED: Massive Parallelization of Tree Edit Distance \- VLDB Endowment, https://www.vldb.org/pvldb/vol17/p1683-fan.pdf

3. arXiv:2203.09183v1 \[cs.CL\] 17 Mar 2022, https://arxiv.org/pdf/2203.09183

4. Feature-Based Selection of Dependency Paths in Ad Hoc Information Retrieval, https://ciir-publications.cs.umass.edu/getpdf.php?id=1111

5. MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers, https://proceedings.neurips.cc/paper\_files/paper/2021/file/260c2432a0eecc28ce03c10dadc078a4-Paper.pdf

6. MAUVE Scores for Generative Models: Theory and Practice \- Journal of Machine Learning Research, https://www.jmlr.org/papers/volume24/23-0023/23-0023.pdf

7. ISO 2859-1:2026—Sampling for Inspection by AQL \- The ANSI Blog, https://blog.ansi.org/ansi/iso-2859-1-2026-aql-sampling/

8. AQL Calculator | Sample Size & ISO 2859-1 Tool \- AQM BD, https://aqmbd.com/quality-control/aql-calculator.php

9. MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers \- arXiv, https://arxiv.org/pdf/2102.01454

10. A survey of diversity quantification in natural language processing: The why, what, where and how \- arXiv, https://arxiv.org/html/2507.20858v3

11. Receiving Inspection Sampling Plans (AQL): ISO 2859-1 Explained with Examples \- Ntia, https://ntia.no/receiving-inspection-sampling-plans/

12. Efficient Computation of the Tree Edit Distance \- ResearchGate, https://www.researchgate.net/publication/277568990\_Efficient\_Computation\_of\_the\_Tree\_Edit\_Distance

13. 5 Playing the Worlds of Prom Week \- Mike Treanor, https://mtreanor.com/publications/Playing-the-Worlds-of-Prom-Week.pdf

14. Social Story Worlds with Comme il Faut \- Mike Treanor, https://mtreanor.com/publications/TCIAIG-CiF.pdf

15. Generative Agents: Simulating Human Behavior | PDF \- Scribd, https://www.scribd.com/document/917249732/Generative-Agents-Interactive-Simulacra-of-Human-Behavior

16. Generative Agents: Interactive Simulacra of Human Behavior \- arXiv, https://arxiv.org/pdf/2304.03442

17. Simple Fast Algorithms for the Editing Distance between Trees and Related Problems | SIAM Journal on Computing, https://epubs.siam.org/doi/10.1137/0218082

18. Syntax-Guided Controlled Generation of Paraphrases | Transactions of the Association for Computational Linguistics \- MIT Press Direct, https://direct.mit.edu/tacl/article/doi/10.1162/tacl\_a\_00318/96454/Syntax-Guided-Controlled-Generation-of-Paraphrases

19. What is the AQL (Acceptance Quality Limit) in QC Inspections? \- QualityInspection.org, https://qualityinspection.org/what-is-the-aql/

20. Acceptable Quality Limit: Definition, Charts, Tables & Examples \- QCADVISOR, https://www.qcadvisor.com/blog/acceptable-quality-limit/