Persona Quality, Information-Density, and Refactor Acceptance Benchmark for SpiralistAI

Full report

On this page

Executive Overview

The management of synthetic personas generated by large language models presents an acute challenge in information architecture. The SpiralistAI ecosystem currently operates with two highly divergent artifacts: a primary Persona output that may exceed one million characters, and a substantially smaller "Create a Person" profile. This extreme variance introduces severe risks across psychometrics, information retrieval, and user experience. A million-character profile inevitably invites semantic bloat, cognitive overload, and the accidental conflation of human-facing narrative with machine-facing metadata. Conversely, an overly concise summary risks stripping away the relational nuances, contextual behavioral constraints, and explicit uncertainties required to simulate a coherent interactive entity. This document establishes an exhaustive, independent research benchmark designed to guide a systemic refactor of the SpiralistAI persona architecture. The objective is to provide an evidence-based, repeatable decision protocol that determines precisely which content should be elevated to the concise profile, which should be retained in the deep view, and which should be aggressively consolidated, moved to a machine-readable export, or deleted as redundant. Grounded in advanced natural language processing (NLP) evaluation, cognitive load theory, and interactive narrative metrics, this benchmark measures true persona depth independently of raw character length, preventing either human readability or machine parseability from dominating the architectural design.

Definitions of Quality and Depth

To construct a valid psychometric and NLP benchmark, foundational concepts regarding persona depth, distinctiveness, and utility must be operationally defined, directly addressing ten required research questions. First, "persona depth" must be operationally defined not by token count, but by the concentration of unique, actionable atomic facts1. Depth represents the degree to which a persona constrains the probability space of future actions. A deep profile offers dense propositional constraints regarding a character's history, relationships, and psychological drivers. Length without constraint is simply semantic bloat. Second, the benchmark distinguishes useful repetition from accidental duplication through proposition extraction and semantic clustering3. Useful repetition operates as progressive disclosure, gently resurfacing critical context to reduce cognitive load5. Accidental duplication manifests as mathematically equivalent semantic propositions or raw JSON schemas bleeding into human-readable text. By extracting atomic claims, the framework penalizes exact lexical and semantic redundancy while permitting context-dependent reinforcement. Third, measuring distinctiveness without rewarding bizarre or incoherent output requires anchoring similarity metrics to internal logical consistency. Distinctiveness is calculated by mapping the semantic distance between personas in a shared vector space, but this distance is only validated if the evaluated persona maintains high motivational coherence and passes natural language inference (NLI) contradiction checks6. A profile that generates random noise will appear distinct, but it will fail the internal consistency thresholds, preventing the reward of hallucinations. Fourth, behavioral usefulness is evaluated entirely within the construct of game-writing quality and interactive narrative8, explicitly avoiding any treatment of the profile as a real-person diagnostic tool. Behavioral utility measures whether a persona provides sufficient rule-based constraints for an author or automated agent to generate a highly probable, fictionally consistent response to a prompt. Fifth, researchers test whether a persona supports consistent scene writing by employing the profile as a system prompt in a simulated narrative environment and measuring the semantic entropy of the resulting generations10. If multiple generation passes yield wildly divergent actions, the persona lacks sufficient constraint. Low semantic entropy across scene generations indicates strong, consistent scene-writing support. Sixth, human raters compare concise profiles with massive ones without length bias by utilizing a double-blind, position-swapping pairwise comparison protocol12. Raters are explicitly trained to penalize verbosity and reward information density, and all comparative tests are run twice with the candidates swapped to cancel out position and verbosity biases inherent in both human and machine evaluators. Seventh, automated similarity metrics, such as cosine similarity on document embeddings, are highly useful for macro-level clustering but fail catastrophically at the micro-level due to "context collapse"4. When nuanced text is reduced to basic relational triples or dense vectors, critical conditional modifiers (e.g., "acts brave only when observed") are lost. Therefore, embeddings must be augmented with fine-grained proposition extraction to maintain evaluation fidelity. Eighth, the human tasks that best reveal missing content are time-boxed, goal-oriented retrieval and creative synthesis exercises15. When an evaluator is asked to summarize the character's internal conflict or predict their response to a stimulus, an inability to complete the task rapidly exposes a lack of actionable depth within the profile. Ninth, thresholds blocking deletion or consolidation are governed by a strict "lossless canonical fact" doctrine. No text segment may be deleted or consolidated unless automated extraction proves that 100% of the unique atomic propositions contained within that segment are preserved elsewhere in the hierarchy. Tenth, the benchmark detects machine-oriented evidence flooding human-facing output by employing structural parsing and readability metrics5. Narrative text infused with UUIDs, raw JSON-LD schemas16, or repetitive procedural templates triggers a sharp degradation in Flesch-Kincaid readability scores and flags exact-duplicate string detection, immediately isolating provenance metadata that belongs in the machine layer.

Evaluation Corpus

The execution of this benchmark requires a meticulously designed sample plan. Testing an architecture on a narrow distribution of personas will yield false confidence, masking systemic failures in edge cases. The evaluation corpus will consist of 2,000 unique persona generation seeds, yielding 10,000 paired artifacts across five experimental representations (Current Main, Current Create, Enriched Create, Compressed Main, Unified Profile). The corpus utilizes a rigid stratification matrix to ensure exhaustive topological coverage.

To prevent cherry-picking, the sample generation must be entirely automated and cryptographically seeded. All artifacts presented to human and machine evaluators must undergo a strict blind-review procedure, stripping all metadata that could identify the architectural representation under test.

Metric Catalog

The benchmark specification demands the calculation of 30 distinct metrics across both human and machine evaluation modalities. The following catalog strictly defines the operational parameters for each dimension.

Identity, Consistency, and Distinctiveness

These metrics ensure the core concept of the persona remains intact, logically sound, and measurably unique.

Behavioral and Narrative Utility

These dimensions assess whether the document serves its primary purpose: enabling a human or machine to write for, or interact with, the simulated entity21.

Cognitive, Epistemic, and Structural Quality

These metrics govern how the information is structured, how uncertainty is modeled, and the physiological toll of reading the artifact.

Machine Parsing, Density, and Governance

These dimensions ensure the artifact remains programmatically useful, dense with unique facts, and safe for enterprise deployment.

Human Task Protocol

Human evaluation provides the ground truth for cognitive load, narrative utility, and layout accessibility. To prevent bias, all tasks must be double-blinded and time-boxed15.

Required Human Evaluation Tasks

Human-Review Rubric Anchors

To ensure high inter-rater reliability, human scoring relies on strictly anchored rubrics rather than subjective feeling.

• Excellent (4): Task is completed rapidly without friction. The profile offers deep constraint and nuanced contradiction. Zero bloat is present. Information is chunked logically and progressively disclosed. NASA-TLX cognitive load score is below 30\.
• Acceptable (3): Task is completed. Some scanning or scrolling is required. Minor redundancies exist but do not impede understanding. Persona constraints are clear enough to guide a writer.
• Weak (2): Task is completed with notable difficulty. High cognitive load (NASA-TLX \> 60). The text exhibits heavy bloat, generic padding ("they are nice and hardworking"), or sparse details. Navigating the UI requires excessive effort.
• Failing (1): Task cannot be completed. The profile overtly contradicts itself, buries canonical facts in unformatted machine-schema exports, or fails to provide enough distinct information to simulate a character.

Automated Analysis Protocol

Automated metrics provide the scale required to test 10,000 artifacts, but they are susceptible to context collapse and verbosity bias. The pipeline utilizes multiple independent analytical methods to cross-verify findings and flag anomalies.

1. Exact Duplicate Detection: Standard string-matching algorithms and n-gram overlap are deployed to identify repetitive schemas, boilerplate text, and runaway loops in the generation pipeline.

2. Near-Duplicate Detection: MinHash and Locality-Sensitive Hashing (LSH) evaluate heavily templated text, catching bloat where the LLM has merely swapped a few adjectives in a repetitive structure.

3. Semantic-Cluster Coverage: K-means clustering of generated sentences measures the breadth of distinct topics covered. A million-character document with only three semantic clusters is mathematically proven to be bloat.

4. Proposition Extraction: Utilizing the FActScore methodology1, LLMs break long text into atomic claims (e.g., "John lives in London", "John fears spiders"). This decouples factual density from raw word count, providing the denominator for true density metrics.

5. Contradiction Detection: A DeBERTa-v3 NLI pipeline6 processes the extracted propositions pairwise, classifying relationships as entailment, neutral, or contradiction.

6. Vocabulary Diversity: Type-Token Ratio (TTR) and the Measure of Textual Lexical Diversity (MTLD) assess the richness of the prose, penalizing generic, high-probability output.

7. Template Repetition: Abstract Syntax Tree (AST) analysis is applied to underlying JSON payloads to detect procedural generation that has spiraled out of control.

8. Generic-Phrase Frequency: Text is cross-referenced against a dynamic blocklist of low-information LLM tropes (e.g., "a testament to," "a rich tapestry," "unwavering resolve").

9. Cross-Persona Similarity: Cosine similarity of document embeddings detects whether the system is regressing to a mean, producing generic personas regardless of the seed.

10. Within-Persona Coherence: Sentence-to-sentence embedding similarity variance ensures the document doesn't drift into entirely unrelated tangents.

11. Unique Propositions per 1,000 Characters: The core performance metric. The total count of unique atomic facts is divided by the payload length. High scores equal high depth; low scores equal bloat.

12. Section Retrieval Accuracy: RAG (Retrieval-Augmented Generation) Needle-in-a-Haystack checks, utilizing protocols like LongGenBench22, ensure machine agents can accurately extract facts from the massive context window.

13. Canonical-to-Rendered Parity: String validation verifies that 100% of the original user seed exists in the final rendered output.

14. Round-Trip Representation Checks: Automated cryptographic hash matching guarantees lossless data integrity post-export.

Limitations of Automated Analyses: Automated evaluation methods carry significant methodological risks. Proposition extraction is vastly superior to dense vector embeddings for measuring facts, but it suffers acutely from "context collapse"4. When an LLM extracts relations as simple triplets (Subject-Predicate-Object), vital narrative conditions are stripped. A sentence reading, "John acts brave only when his child is threatened" reduced to the proposition \[John, is, brave\] will trigger false contradiction flags when later text shows John acting cowardly. Furthermore, semantic entropy estimators10 are computationally expensive and sensitive to clustering heuristics27. Finally, one embedding score should never serve as a solitary quality measure; dense bloat and dense insight can embed similarly in vector space, necessitating the layered approach described above.

Pairwise Comparison Design

To definitively answer how human raters can compare a concise profile with a massive one without being biased by length (RQ6), the benchmark mandates a strict double-blind A/B testing protocol utilizing position swapping12. Both human raters and LLM-as-a-judge models inherently exhibit verbosity bias—the psychological inclination to rate longer, more detailed responses more favorably, even if the detail is devoid of substance. To mitigate this, comparisons are drawn across five vectors:

1. Current Main Persona vs. Current Create a Person

2. Current Main Persona vs. Compressed Main Persona

3. Current Create a Person vs. Enriched Create a Person

4. Enriched Create a Person vs. Compressed Main Persona

5. Current Ecosystem vs. Proposed Unified Profile

Mitigation Protocol: For every evaluated pair (Candidate A vs. Candidate B), the evaluation is executed twice. The second run swaps the presentation order (Candidate B vs. Candidate A). A preference is only logged as valid if the judge remains consistent across both runs12. Furthermore, evaluators are explicitly prompted via instruction rubrics to aggressively penalize unnecessary verbosity and reward information density.

Statistical Plan

To ensure the empirical validity of the benchmark results and protect against false positives arising from exhaustive testing, the statistical plan incorporates advanced psychometric controls. First, standard inter-rater agreement metrics, such as Fleiss’ Kappa, behave erratically when agreement is exceptionally high (the Kappa paradox). Therefore, the benchmark mandates the use of Gwet’s AC1 to measure inter-rater reliability among human evaluators, ensuring mathematical stability across high-agreement validation tasks28. Second, 95% bootstrap confidence intervals must be reported for all continuous metrics (e.g., Unique Propositions per 1,000 characters) to accurately represent the variance within the generation engine30. Third, because the benchmark evaluates 30 distinct dimensions across 5 representation formats, the probability of finding a statistically significant result by pure chance is unacceptably high. The Family-Wise Error Rate (FWER) must be strictly controlled. The Holm-Bonferroni correction will be applied to all p-values. This step-down procedure provides robust protection against false discoveries while retaining higher statistical power than a standard Bonferroni correction, which is overly conservative for this NLP evaluation domain31. Fourth, effect-size reporting is mandatory. Cohen’s h (for proportions) and Hedges’ g (for continuous data) will be calculated to measure the true magnitude of differences between architectures, ensuring that statistically significant findings are practically meaningful30. Finally, missing or ambiguous results in human evaluation will be subjected to pairwise deletion rather than zero-imputation to prevent artificially skewing the density metrics downward.

Decision Matrix

This matrix directly translates the statistical findings of the benchmark into actionable architectural refactoring steps, fulfilling the primary objective of the assignment.

Threshold to Block Deletion (RQ9): The governance rule is absolute: no content may be marked "Remove as redundant" unless a proposition extraction pass mathematically guarantees that zero unique atomic facts are lost in the deletion.

Acceptance Criteria

Before any proposed refactor of the SpiralistAI architecture is approved for production deployment, it must pass the following quantifiable gates:

1. Zero Loss of Canonical Facts: 100% retention rate of user-seeded identity traits, verified via NLI entailment.

2. Improved Information Density: Unique propositions per 1,000 characters must increase by a statistically significant margin (p \< 0.05, Holm-Bonferroni adjusted) in the Main Persona.

3. No Increase in Contradiction: Internal contradiction rates must remain flat or decrease.

4. Preserved Scene Usefulness: Scene-generation adherence must remain ≥ 80% with low semantic entropy.

5. Preserved Relationship Specificity: Human rubric scores for relationship complexity must not drop below established baseline bounds.

6. Reduced Initial Payload: The "Create a Person" text footprint must not expand by more than 15%, maintaining strict progressive disclosure principles5.

7. Preserved Complete Export: The archival export must maintain 100% round-trip reproducibility via cryptographic hash matching.

8. Improved Findability: Human time-on-task for fact retrieval must decrease by at least 20%.

9. No Weakened Boundaries: Absolute zero tolerance for safety toxicity regressions or PII privacy boundary violations.

Regression Test Deck

The executing team is required to run this 30-case conceptual regression deck against all new pipeline builds to detect systemic breakdowns in the refactored architecture.

Final Decision Memo Template

Upon completion of the benchmark execution phase, the team must utilize the following template to establish an immutable, traceable record of refactor decisions. To: Product Governance & Engineering Leadership From: Evaluation Execution Team Date: \[YYYY-MM-DD\] Subject: Refactor Execution Decision for SpiralistAI Persona Architecture 1\. What was measured: \[Summary of corpus size, specific metrics evaluated, and human/machine split\] 2\. What improved: \[List metrics with statistically significant positive effect sizes, referencing Holm-Bonferroni adjusted p-values and Hedges' g31\] 3\. What degraded: \[List any metrics that suffered, explicitly noting if they breached failure thresholds\] 4\. What content is being added to 'Create a Person': \[Explicit list of proposition categories, e.g., "Immediate goals, Primary relationship node"\] 5\. What content is being consolidated in 'Main Persona': \[Explicit list of semantic clusters, e.g., "Inventory logs, daily routines"\] 6\. What content is moving to another representation: \[Identify machine-readable data moving to JSON/export layers or metadata\] 7\. What is deliberately unchanged: \[List elements retained due to high distinctiveness/actionability despite length\] 8\. What remains uncertain: \[Identify edge-cases where the data did not provide a definitive 95% confidence interval\] Sign-off: \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

Risks and Limitations

The execution of this benchmark design carries inherent methodological risks that must be managed. The most pressing risk is context collapse in automated extraction4. Utilizing LLMs to extract atomic facts and triplets often strips away the conditional context that makes narrative characters functionally engaging. Executing teams must ensure proposition extraction pipelines capture nuanced modifiers and conditionals, rather than reducing characters to flat Subject-Predicate-Object triples that inevitably trigger false contradictions. Furthermore, strict NLI-based redundancy checkers risk the over-penalization of stylistic repetition. Highly rhetorical writing utilizes repetition for emotional emphasis; automated tools may falsely flag this as bloat. Finally, LLMs acting as automated judges exhibit known biases toward longer, more verbose answers. While the mandated position-swapping protocol12 mitigates this, it cannot entirely eliminate the systemic bias wired into the judge model’s pre-training data. Human oversight remains a critical necessity for boundary-case resolution.

Conclusion

The challenge of managing a 1,000,000-character persona output alongside a concise summary requires a paradigm shift away from evaluating text by character count. By operationalizing persona depth as the density of unique, actionable atomic propositions—and by establishing an exhaustive matrix of human and automated psychometric evaluation protocols—this benchmark provides SpiralistAI with a statistically sound methodology for architecture refactoring. Executing this protocol guarantees that the "Create a Person" view is enriched without being overwhelmed, the "Main Persona" is cleansed of redundant machine artifacts without sacrificing canonical identity, and downstream applications receive highly constrained, actionable, and conceptually deep character profiles.

Works cited

1. FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation \- ACL Anthology, https://aclanthology.org/2023.emnlp-main.741/

2. FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation \- arXiv, https://arxiv.org/html/2305.14251v2

3. GRASP: Graph Agentic Search over Propositions for Multi-hop Question Answering \- arXiv, https://arxiv.org/html/2605.16598v1

4. PropRAG: Guiding Retrieval with Beam Search over Proposition Paths \- arXiv, https://arxiv.org/html/2504.18070v2

5. The Ultimate SaaS UI Checklist: 10 Design Mistakes to Avoid \- Exalt Studio, https://exalt-studio.com/blog/the-ultimate-saas-ui-checklist-10-design-mistakes-to-avoid

6. Assignment 5: Fact-checking Outputs from ChatGPT Dataset and Code, https://www.cs.utexas.edu/\~gdurrett/courses/fa2024/a5.pdf

7. Beyond Factual Accuracy: Evaluating Coverage of Diverse Factual Information in Long-form Text Generation | Request PDF \- ResearchGate, https://www.researchgate.net/publication/394270275\_Beyond\_Factual\_Accuracy\_Evaluating\_Coverage\_of\_Diverse\_Factual\_Information\_in\_Long-form\_Text\_Generation

8. Objective Metrics for Interactive Narrative \- SciSpace, https://scispace.com/pdf/objective-metrics-for-interactive-narrative-2thsin7j4h.pdf

9. ENHANCING NON-PLAYER CHARACTER DIALOGUE IN VIDEO GAMES: AN EVALUATION OF LARGE LANGUAGE MODEL-GENERATED RESPONSES, https://digitalcommons.calpoly.edu/cgi/viewcontent.cgi?article=4816\&context=theses

10. Evidential Semantic Entropy for LLM Uncertainty Quantification \- ACL Anthology, https://aclanthology.org/2026.eacl-long.334.pdf

11. ECLIPSE: Reducing AI Hallucinations in Finance \- Emergent Mind, https://www.emergentmind.com/papers/2512.03107

12. The 5 Biases That Can Silently Kill Your LLM Evaluations (And How to Fix Them), https://www.sebastiansigl.com/blog/llm-judge-biases-and-how-to-fix-them/

13. bias-mitigation.md \- Agent-Skills-for-Context-Engineering \- GitHub, https://github.com/muratcankoylan/Agent-Skills-for-Context-Engineering/blob/main/skills/advanced-evaluation/references/bias-mitigation.md?plain=1

14. PropRAG: Guiding Retrieval with Beam Search over Proposition Paths \- ACL Anthology, https://aclanthology.org/2025.emnlp-main.317.pdf

15. Usability Testing: Complete Guide to Optimizing User Experience \- HowToes, https://howtoes.blog/2025/07/01/usability-testing-complete-guide-to-optimizing-user-experience-for-digital-success/

16. The Complete Guide to Structured Data for AI Search: JSON-LD Schemas That Drive Citations \- Space & Story, https://spaceandstory.co/blog/structured-data-guide-ai-search-json-ld

17. TimeChara: Evaluating Point-in-Time Character Hallucination of Role-Playing Large Language Models \- arXiv, https://arxiv.org/html/2405.18027v1

18. \[Question\]: LLM response reproducibility / Random Seed Settings · Issue \#13551 · infiniflow/ragflow \- GitHub, https://github.com/infiniflow/ragflow/issues/13551

19. Rethinking Role-Playing Evaluation: Anonymous Benchmarking and A Systematic Study of Personality Effects \- arXiv, https://arxiv.org/html/2603.03915v2

20. HalluScan: A Systematic Benchmark for Detecting and Mitigating Hallucinations in Instruction-Following LLMs \- arXiv, https://arxiv.org/html/2605.02443v2

21. CharacterEval: A Chinese Benchmark for Role-Playing Conversational Agent Evaluation, https://arxiv.org/html/2401.01275v1

22. LongGenbench: Benchmarking Long-Form Generation in Long Context LLMs \- arXiv, https://arxiv.org/html/2409.02076v7

23. NASA-TLX \- Wikipedia, https://en.wikipedia.org/wiki/NASA-TLX

24. NASA Task Load Index \- Digital Healthcare Research, https://digital.ahrq.gov/health-it-tools-and-resources/evaluation-resources/workflow-assessment-health-it-toolkit/all-workflow-tools/nasa-task-load-index

25. Web Accessibility. WCAG 2.2 made easy \- Usable y Accesible. Olga Carreras, https://www.usableyaccesible.com/archivos/Accessibility\_Web\_WCAG2\_2\_easy\_EN\_2025\_revilla\_carreras.pdf

26. LongGenbench: Benchmarking Long-Form Generation in long-context LLMs \- arXiv, https://arxiv.org/html/2409.02076v3

27. Soft-Community Kernel Rényi Spectrum for Semantic Uncertainty Estimation in Large Language Models \- MDPI, https://www.mdpi.com/1099-4300/28/4/442

28. Why Sensitivity Is Not Kappa: Performance vs Agreement — Uniqcret, https://www.uniqcret.com/post/performance-vs-agreement-sensitivity-kappa

29. Developing Holistic Gasing Evaluation Model to Balance Cognitive Efficiency and Affective Resilience \- Cahaya Ilmu Cendekia Publisher, https://cahaya-ic.com/index.php/JEE/article/download/2825/1857

30. CourseEvalAI: Rubric-Guided Framework for Transparent and Consistent Evaluation of Large Language Models \- MDPI, https://www.mdpi.com/2073-431X/14/10/431

31. Boards-style benchmarks overestimate prior-chat bias in large language models: a factorial evaluation study | medRxiv, https://www.medrxiv.org/content/10.64898/2026.02.12.26346164v1.full-text

32. Prediction of cognitive impairment through speech data analysis: A comparative evaluation of deep learning models | PLOS One \- Research journals, https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0349412

33. Large language models show Dunning-Kruger-like effects in multilingual fact-checking, https://pmc.ncbi.nlm.nih.gov/articles/PMC12936108/