Human Review, Curriculum Governance, And Contamination Control Before Training

Full report

On this page

1. Governance Principles and Explicit Assumptions

The development and refinement of a compact local assistant depend entirely on the cryptographic integrity and semantic quality of its training data. Scaling laws for alignment demonstrate that massive, noisy datasets are inferior to highly curated, structurally perfect conversational records1. As an independent training-data governance lead, conversation-quality researcher, and evaluation-contamination auditor, the overarching mandate is to design a rigorous, deterministic process for candidate record evaluation. This governance architecture rests upon several explicit assumptions and non-negotiable boundaries. First, no agent or auditor may claim that it inspected, ran, or validated private TinyRustLM code, evaluation evidence, or unseen proprietary datasets. Consequently, the two known opaque local candidate collections—comprising a small 42-record role-history set and a larger 668-record draft set—cannot be judged or authorized for training sight unseen. The process must be designed to scale and govern these sets strictly through procedural gating, cryptographic hashing, and blinded human review. Under no circumstances should an Large Language Model (LLM) judge be assumed to replace human semantic review for the authorization of training data, as LLMs frequently exhibit preference leakage, evaluator bias, and a lack of grounded reasoning when scoring complex qualitative dimensions4. The behavioral goals of the local assistant dictate the foundational rules of the corpus. Ordinary, helpful conversation must dominate the training manifold to ensure the model remains eager to assist rather than defaulting to evasion1. When faced with ambiguity, the assistant must produce useful clarifying questions rather than fabricating assumptions. Advice rendered by the model must honor practical resource constraints and provide feasible fallbacks. Furthermore, current user instructions must strictly supersede stale profile memory, and any quoted, retrieved, or remembered instructions must be treated as untrusted data payloads that cannot override core behavioral boundaries. Conflicting authorities within a prompt must be acknowledged and resolved honestly, without fabricating false consensus. Lawful adult questions must receive substantive, balanced treatment, while concrete security and nonconsensual-harm boundaries remain exceptionally narrow and strictly enforced. Responses involving summaries, rewrites, extractions, JSON, and code explanations must be highly useful and structurally sound. Finally, positive examples must substantially outnumber refusals, and sealed evaluations—along with their expected answer terms—must remain entirely excluded from the training mixture5.

2. Versioned Record and Provenance Schema

To enforce strict lineage, reproducible training configurations, and exact data boundaries, every single-turn and multi-turn conversational example must conform to a heavily versioned cryptographic schema. The schema must encapsulate textual data, generation metadata, lineage pointers, and a precise loss-masking topology.

The most critical component of the schema for supervised fine-tuning (SFT) is the assistant\_loss\_mask. In standard instruction tuning, the objective is to teach the model to generate the assistant's response, not to predict the user's prompt or the system instructions12. If loss were calculated over user messages, the model would waste capacity learning to predict the prompt distribution, degrading its conversational utility. The schema maintains an array of binary integers matching the tokenized sequence length, populated as 0 for system and user tokens, and 1 for assistant tokens. Because causal language modeling relies on next-token prediction—where the label for token [Figure omitted from source export] is token [Figure omitted from source export]—the loss mask must be mathematically shifted (or "rolled") by one position12. If the assistant's response begins at index [Figure omitted from source export], the loss mask at index [Figure omitted from source export] must be set to 1 so the model is properly penalized for failing to predict the first assistant token given the final user token.

3. Reviewer Manual and Critical-Failure Taxonomy

Human review remains the irreplaceable gold standard for high-fidelity alignment1. The reviewer manual must strictly separate objective factual assessments from subjective conversational quality, requiring raters to evaluate records across multiple dimensions without conflating them.

A simple numerical score is entirely insufficient for severe defects, as averaging a catastrophic safety failure with a high formatting score produces a deceptively acceptable median. Therefore, the presence of any critical failure requires immediate rejection of the record.

4. Exact Protocol for the Opaque 42-Record Set

The small collection of 42 role-history records represents a high-leverage data asset that is too small for statistical sampling. At this microscopic scale, the risk of a single defective record disproportionately shifting the model's behavior is severe. Therefore, every single record in this collection must undergo exhaustive, dual-independent human review. Every record in the 42-item set is assigned to two independent human raters who are strictly blinded to each other’s evaluations. Each rater applies the multidimensional rubric and actively scans for the critical failures outlined in the taxonomy. The raters must operate in total isolation to prevent anchoring bias. If the two raters disagree on the presence of a critical failure, or if their ordinal quality scores diverge by more than one point on a standard 5-point Likert scale, the record is flagged for mandatory adjudication. A third, senior auditor—acting as the tie-breaker—reviews the record alongside the masked, written justifications of the first two raters. The senior auditor's decision is final, but the disagreement is logged for subsequent inter-rater reliability analysis. Before any record from this 42-item set is cryptographically signed and promoted to the trainable manifest, the system must record irrefutable evidence of its quality. This evidence requires the cryptographic identities of both independent raters (and the adjudicator, if applicable), a verified boolean assertion that zero critical failures were detected, and an average quality score meeting the predefined promotion threshold (e.g., [Figure omitted from source export]). Only upon the generation of this immutable approval payload is the review\_status flipped from PENDING\_REVIEW to APPROVED.

5. Exact Protocol for the Opaque 668-Record Set

The larger draft set of 668 records occupies a transitional scale: it is large enough that 100% dual review is resource-intensive and potentially inefficient, but small enough that aggressive random sampling might fail to detect concentrated pockets of critical failures. Therefore, a hybrid approach combining active-learning triage and rigorous statistical sampling is mandated. To determine if a sample is sufficient to authorize the unreviewed remainder, the protocol relies on the Clopper-Pearson exact binomial confidence interval, which provides a strictly conservative guarantee of coverage without relying on large-sample normal approximations14. Given [Figure omitted from source export] observed defects in a sample of size [Figure omitted from source export], the upper bound [Figure omitted from source export] of the true defect rate at a [Figure omitted from source export] confidence level is derived using the Beta distribution quantile function [Figure omitted from source export]14: [Figure omitted from source export] When zero defects are observed ([Figure omitted from source export]), the formula simplifies significantly, and the statistical "Rule of Three" serves as a robust heuristic: a 95% confidence upper bound on the defect rate is approximately [Figure omitted from source export]16. For a population of 668 records, if the governance mandate requires 95% confidence that the true critical failure rate is below 1%, the required sample size yielding zero defects would be roughly 300 records16. Sampling nearly half the dataset to blindly authorize the other half represents an inefficient allocation of human resources. Instead of simple random sampling, the protocol dictates an active-learning triage phase to heavily stratify risk18. An auxiliary evaluator model—acting solely as a triage heuristic, not a final judge—scores all 668 records. Records with high predictive uncertainty, structural anomalies (e.g., extreme sequence lengths, unusual character ratios), or high semantic similarity to known failure modes are flagged as high-risk. The 668 records are then partitioned into a high-risk stratum and a low-risk stratum. One hundred percent of the high-risk stratum undergoes dual human review. The low-risk stratum is subsequently sampled using the aforementioned binomial distribution. Crucially, if a single critical failure is detected in the low-risk sample, the assumption of safety is shattered. The statistical upper bound spikes exponentially, rendering the sample entirely insufficient to authorize the unreviewed remainder. In this event, the active-learning triage is deemed a failure, and the entire 668-record collection must default to 100% human review before authorization.

6. Sampling Statistics and Inter-Rater Analysis

The reliability of human labels is the bedrock of the training data. If reviewers fundamentally disagree on what constitutes a helpful or safe response, the resulting model will suffer from mode collapse or erratic behavioral drift. To quantify inter-rater agreement (IRA) while accounting for chance, missing data, and varying levels of measurement, the protocol utilizes Krippendorff's alpha ([Figure omitted from source export])20. The overarching formulation calculates the ratio of observed disagreement ([Figure omitted from source export]) to expected disagreement by chance ([Figure omitted from source export])21: [Figure omitted from source export] The calculation of [Figure omitted from source export] adapts to the metric nature of the evaluation dimension. For critical failures, a nominal metric is used, employing an identity function where disagreement is 1 if the boolean labels differ, and 0 if they match22. For quality scores, an ordinal metric is used, employing ranked distances that account for the ordering of categories; two adjacent categories (e.g., a score of 4 versus 5\) are penalized less severely than two categories far apart (e.g., 1 versus 5\)22. A threshold of [Figure omitted from source export] is mandated for high-reliability dimensions such as factual correctness and safety, while [Figure omitted from source export] is tentatively acceptable for highly subjective dimensions such as tone22. Before evaluating the live 42-record or 668-record sets, reviewers must undergo blinded calibration against synthetic "gold" standard items and intentional counterexamples. Continuous drift checks—where previously evaluated gold items are silently re-inserted into the queue—are required to ensure reviewers do not develop lenient or draconian tendencies over time. To combat cognitive degradation, the protocol enforces strict reviewer fatigue limits: no rater may evaluate conversational data for more than 120 consecutive minutes without a mandated lockout period. When systematic disagreement occurs—particularly regarding narrow safety boundaries or lawful adult content—it often indicates ambiguity in the reviewer manual rather than rater error. If Krippendorff's alpha drops below 0.667 on the "Over-Refusal & Tone" dimension, review must be halted immediately. Policy must be clarified to ensure raters do not inject personal moralizing into the dataset, guaranteeing that lawful adult inquiries receive neutral, substantive treatment.

7. Repair, Rejection, Preference, and Deletion Lineage

When a candidate record fails review, simply discarding it wastes valuable scenario configurations. However, silently repairing it obscures the origin of model failures and erases critical data necessary for preference optimization. The governance protocol establishes strict rules for record derivation. When a record requires repair, reviewers must choose the optimal path. Rejecting a record is necessary for catastrophic safety violations. Minimally editing a record—modifying a few tokens to fix a hallucination or format error—is permitted but strongly discouraged for significant rewrites, as human annotators often introduce unnatural linguistic artifacts or "reviewer dialect" that diverges from the target model's natural manifold. Regenerating the response from the exact same scenario factors using a different random seed or adjusted temperature is the preferred method for repair. If the primary teacher model repeatedly fails, obtaining a second teacher answer from a distinct frontier model may be used, provided the licensing schema permits it. Alternatively, preserving the defective output as a negative preference example is highly valuable for alignment algorithms. Regardless of the path chosen, every repair operation must be strictly tracked. The original defective record is preserved with a REJECTED status, and the new record inherits a derived\_from field pointing to the original record\_hash. This deletion lineage ensures that researchers can audit exactly which prompts induced failures and how they were mitigated, without exposing the training workers to the defective text. Not all rejected records should be deleted. If a record fails due to verbosity, slight structural deviation, or minor inaccuracies, it serves as an optimal "hard negative" for preference optimization algorithms27. However, records containing critical safety failures, PII, or explicit hallucinations must be wholly removed from the dataset to prevent the model from assigning any probability mass to catastrophic token sequences. For the records retained as preference pairs, the selection of alignment algorithms is critical. Direct Preference Optimization (DPO) has traditionally served as the standard, but it requires calculating log probabilities against a frozen reference model, doubling the memory overhead28. Given the resource constraints of training a compact local assistant, reference-free algorithms are required: Odds Ratio Preference Optimization (ORPO) merges SFT and preference alignment into a single monolithic loss function by appending a log odds ratio term to the negative log-likelihood loss30. It explicitly penalizes the rejected response while rewarding the chosen one, handling severe class imbalances highly efficiently33. The objective is defined as: [Figure omitted from source export] Simple Preference Optimization (SimPO) eliminates the reference model by utilizing a length-normalized average log probability as the implicit reward28. This naturally aligns the training objective with the generation metric, preventing the model from exploiting length (where it artificially inflates probabilities for verbose, lower-quality answers)29. [Figure omitted from source export] Kahneman-Tversky Optimization (KTO) leverages an asymmetric utility function based on prospect theory, requiring only a binary signal of whether an output is desirable or undesirable35. This is ideal for utilizing legacy records where no clear alternative response was generated.

8. Scenario-Factor and Curriculum Design

To build a compact local assistant, the training curriculum must instill generalized reasoning rather than rote memorization. This requires a systematic approach to scenario generation. Rather than generating prompts randomly, the dataset must be constructed as a combinatorial matrix of scenario factors.

Training is executed in staged curriculum phases, transitioning from format compliance to deep reasoning38. A universal optimum does not exist; instead, data volumes must scale logarithmically with capability milestones. The 1K Pilot Corpus is focused purely on structural compliance, role adherence, and tone. This corpus teaches the model how to talk, avoiding chatty filler and adhering to JSON formats. The 10K Validation Corpus expands into domain-specific knowledge and multi-turn state changes. This phase teaches the model to navigate conflicting instructions and utilize context effectively. The 100K+ Training Corpus instills deep domain mastery, complex reasoning, and vast scenario-factor variation, solidifying the model's ability to generalize across unseen tasks.

9. Contamination Threat Model and Controls

Benchmark leakage silently inflates evaluation performance, destroying the integrity of model validation by allowing the model to "remember" answers rather than reason through them8. The governance framework demands a multi-tiered contamination detection pipeline that operates on both syntactic and semantic levels. Before any record enters the trainable manifest, it must undergo strict deduplication and overlap analysis against known public benchmarks (e.g., MMLU, GSM8K, ARC). All text is standardized using Unicode Normalization Form C (NFC)42, lowercased, and stripped of punctuation. The pipeline utilizes 64-permutation MinHash combined with Locality-Sensitive Hashing (LSH) to identify near-duplicates at scale across the corpus, catching paraphrased or slightly edited copies10. Furthermore, a sliding token n-gram and character n-gram window is run against the benchmark index. Any candidate exhibiting a high Longest Common Subsequence (LCS) ratio or high n-gram overlap with a benchmark item is automatically flagged and rejected10. Sophisticated LLMs can suffer from semantic contamination, where the exact phrasing differs, but the core entity relationships, semantic neighbors, and structural logic of an evaluation set are leaked8. The system must scan for entity-substituted overlap and specific expected-answer overlaps (e.g., highly specific numerical answers or unique terminology) within the context of similar semantic clusters5. Additionally, teacher memorization poses a severe threat; if a frontier model acting as a teacher was contaminated during its own pre-training, it may inject benchmark paraphrases into the synthetic data. Output distributions exhibiting suspiciously high probabilities on obscure benchmark topics must be quarantined. To protect proprietary evaluations, a sealed-holdout exclusion registry is maintained. This registry binds the generator version, seed, template family, private prompt hash, expected-term hash, exclusion reason, and deletion lineage into a secure ledger. By storing only one-way cryptographic hashes of the private evaluation prompts and expected terms, training workers and data parsers can check candidate hashes against this registry without ever exposing the holdout content to the workers or the training infrastructure. If a hash collision occurs, the record is silently discarded, ensuring the private evaluations remain completely unseen and valid5.

10. Teacher-Generation and Factual-Review Controls

When relying on larger frontier models to bootstrap data for the compact assistant, rigorous generation controls are required to ensure the resulting text is safe, verifiable, and free of malicious artifacts. Responses should be generated using a multi-model ensemble. If the constituent models diverge significantly in their semantic outputs (ensemble disagreement), the prompt is deemed too ambiguous or factually unstable and is routed to human review. Generation must rely on fixed seeds and temperature settings logged in the schema, guaranteeing deterministic replay; if a systemic error is found, the exact generation state can be reproduced and debugged. Advanced teacher models often produce internal reasoning tokens (e.g., \<think\> or \<scratchpad\> blocks). These hidden-reasoning blocks must be programmatically stripped before the SFT phase; training a compact model on raw reasoning blocks intended for massive models often degrades the compact model's fluency and causes it to leak internal logic to the user11. Output claims must undergo factual verification against a localized, whitelisted retrieval database. Furthermore, outputs must be scrubbed for malicious policy artifacts and evasion behaviors unique to the teacher model (e.g., "As an AI developed by X, I cannot..."). Automated regex and named-entity recognition (NER) scanners actively strip potential personally identifiable information (PII) and ensure strict license compliance. Any deletion requests from a rights-holder triggers a targeted purge traversing the deletion lineage ledger to wipe all derived records.

11. Trainable-Manifest Promotion Gate

The final step before data reaches the GPU is the promotion gate. No individual record can be trained on unless it is explicitly included in a cryptographically signed trainable manifest. The manifest operates as an immutable Merkle tree of record\_hash identifiers. It explicitly lists the approved records, the specific reviewer pseudonyms that approved them, and the threshold metrics that justified the promotion. A parallel removal manifest explicitly blacklists rejected hashes, guaranteeing they cannot bypass future data pipelines. During this compilation, the gate strictly enforces the assistant-only loss mask11. The manifest validates that every record has a reproducible mask where all system and user tokens carry a weight of 0, ensuring no gradients are calculated for prompt prediction. The manifest also verifies that the mask rolling parameters are mathematically correct for the chosen causal attention architecture, preventing off-by-one errors that derail training stability.

12. Pilot Design and Stop Rules

Training and data ingestion must immediately halt if any of the following stringent stop rules are triggered. First, an excessive defect rate requires a halt; if post-training evaluation reveals a critical failure rate exceeding 0.1% on held-out data, the data pipeline is compromised. Second, inadequate ordinary-chat coverage triggers a stop; if statistical analysis shows the corpus is over-indexed on adversarial refusals or complex edge cases, causing the model to degrade in standard, helpful dialogue, the curriculum must be rebalanced. Third, reviewer disagreement mandates a pause; if Krippendorff’s alpha falls below the 0.667 threshold for critical dimensions, it indicates a breakdown in the reviewer consensus model. Fourth, any detection of contamination—such as sudden, inexplicable performance spikes on public benchmarks—suggests a failure in the LSH/N-gram filtration pipeline and requires an immediate audit. Fifth, license uncertainty or changes in upstream terms of service require freezing operations until the provenance schema confirms compliance. Finally, the failure of a small pilot to improve unseen behavior triggers a halt; if the 1K pilot fails to meaningfully improve the model's structural adherence on unseen prompts, it indicates that the base model requires continued pre-training before it can effectively absorb SFT data.

13. Unknowns Requiring Inspection of Private Records

Because the 42-record and 668-record collections remain entirely unseen and uninspected, several critical variables cannot be resolved through abstract protocol design alone. The actual execution of this governance framework will require discovering the baseline toxicity of the collections, which dictates the inherent frequency of unsafe or malicious requests within the specific domain. The linguistic distribution—comprising the proportion of non-standard dialects, technical jargon, or code snippets—remains unknown and is necessary to evaluate tokenizer efficiency and vocabulary coverage. Furthermore, the context length variance, or the distribution of token counts across the records, is currently unknown but is strictly required to optimize fixed-length bin packing (e.g., 4096 or 8192 token bins) to minimize truncation and padding inefficiencies11. Finally, the presence of implicit bias within the prompt formulation cannot be measured without inspection; discovering such biases is necessary to determine if counterfactual augmentation is required to balance the curriculum.

14. Annotated Primary-Source Bibliography

The formulation of this governance architecture is grounded in current literature and technical documentation regarding LLM alignment, statistical sampling, and data contamination.

Works cited

1. A Closer Look at the Limitations of Instruction Tuning \- arXiv, https://arxiv.org/html/2402.05119v4
2. UNDERSTANDING LAYER SIGNIFICANCE IN LLM ALIGNMENT \- OpenReview, https://openreview.net/pdf?id=7ha61H73pg
3. DATA SELECTION VIA OPTIMAL CONTROL FOR LANGUAGE MODELS \- ICLR Proceedings, https://proceedings.iclr.cc/paper\_files/paper/2025/file/9ad4891facabf17aa11580686bacfe4e-Paper-Conference.pdf
4. Keys to Successful LLM-as-a-Judge and HITL Workflows \- Kili Technology, https://kili-technology.com/blog/keys-to-successful-llm-as-a-judge-and-hitl-workflows
5. ResearchQA: Benchmarking Citation-Grounded Question-Answering on Scientific Papers, https://arxiv.org/html/2607.11074v1
6. Benchmarking MLLMs on Topological Reasoning of Chemical Reaction Diagrams | OpenReview, https://openreview.net/forum?id=N5YcOxEcV8
7. When Helpfulness Backfires: LLMs and the Risk of Misinformation Due to Sycophantic Behavior \- PMC, https://pmc.ncbi.nlm.nih.gov/articles/PMC12045364/
8. SSA: Semantic Contamination of LLM-Driven Fake News Detection \- ACL Anthology, https://aclanthology.org/2025.emnlp-main.744/
9. Large Language Model Benchmarks: A Taxonomy of Capabilities, Scientific Quality Assessment, and Saturation Analysis \- MDPI, https://www.mdpi.com/2504-4990/8/6/141
10. L20-Edu-135M: An Auditable Single-GPU Study of Data-Efficient Small Language Modeling, https://arxiv.org/html/2606.22189v1
11. Stage 1: Supervised Fine-Tuning (SFT) — Nemotron \- NVIDIA Documentation, https://docs.nvidia.com/nemotron/latest/nemotron/super3/sft.html
12. Stage 1: Supervised Fine-Tuning (SFT) — Nemotron \- NVIDIA Documentation, https://docs.nvidia.com/nemotron/nightly/nemotron/nano3/sft.html
13. Bittensor Agent Arenas as a Trajectory Primitive: Distilling a Shopping Agent from ShoppingBench Subnet Traces \- arXiv, https://arxiv.org/html/2606.10064v1
14. Clopper-Pearson Interval Calculator \- MetricGate, https://metricgate.com/docs/clopper-pearson-interval/
15. Better Binomial Confidence Intervals \- Digital Commons @ Wayne State, https://digitalcommons.wayne.edu/cgi/viewcontent.cgi?article=1132\&context=jmasm
16. A Concise Guide to the Statistical Rule of Three \- Statology, https://www.statology.org/a-concise-guide-to-the-statistical-rule-of-three/
17. Confidence Interval for a Proportion: Formula, Steps & Calculator \- Statistics Fundamentals, https://statisticsfundamentals.com/confidence-intervals/confidence-interval-for-proportion/
18. Do We Still Need Humans in the Loop? Comparing Human and LLM Annotation in Active Learning for Hostility Detection \- arXiv, https://arxiv.org/html/2604.13899v1
19. HILTS: Human-LLM Collaboration for Effective Data Labeling \- NSF PAR, https://par.nsf.gov/servlets/purl/10653923
20. Inter-rater reliability \- Grokipedia, https://grokipedia.com/page/Inter-rater\_reliability
21. Krippendorff's alpha \- Grokipedia, https://grokipedia.com/page/Krippendorff's\_alpha
22. Krippendorff's Alpha Calculator \- MetricGate, https://metricgate.com/docs/krippendorffs-alpha/
23. Computing Krippendorff's Alpha-Reliability, https://www.asc.upenn.edu/sites/default/files/2021-03/Computing%20Krippendorff%27s%20Alpha-Reliability.pdf
24. Krippendorff's alpha \- Wikipedia, https://en.wikipedia.org/wiki/Krippendorff%27s\_alpha
25. Krippendorff's Alpha Basic Concepts \- Real Statistics Using Excel, https://real-statistics.com/reliability/interrater-reliability/krippendorffs-alpha/krippendorffs-alpha-basic-concepts/
26. K-Alpha Calculator–Krippendorff's Alpha Calculator: A user-friendly tool for computing Krippendorff's Alpha inter-rater reliability coefficient \- PMC, https://pmc.ncbi.nlm.nih.gov/articles/PMC11636850/
27. From Imitation to Discrimination: Progressive Curriculum Learning for Robust Web Navigation \- arXiv, https://arxiv.org/html/2604.12666v1
28. Simple Preference Optimization (SimPO) \- Emergent Mind, https://www.emergentmind.com/topics/simple-preference-optimization-simpo
29. SimPO: Simple Preference Optimization with a Reference-Free Reward \- arXiv, https://arxiv.org/html/2405.14734v1
30. ORPO Trainer \- Hugging Face, https://huggingface.co/docs/trl/orpo\_trainer
31. Welcome to my blog\! \- Odds Ratio Preference Optimization (ORPO), https://thinamxx.github.io/blog/posts/ORPO/orpo.html
32. ORPO: Monolithic Preference Optimization without Reference Model \- arXiv, https://arxiv.org/html/2403.07691v2
33. DPO Isn't Enough: The Modern Post-Training Stack — SimPO, ORPO, KTO and Beyond | by James Fahey | Medium, https://medium.com/@fahey\_james/dpo-isnt-enough-the-modern-post-training-stack-simpo-orpo-kto-and-beyond-d82e52a1ee6c
34. Simpo: Simple preference optimization with a reference-free reward \- arXiv, https://arxiv.org/pdf/2405.14734
35. Stable Preference Optimization: Learning preference is more important than imitation | OpenReview, https://openreview.net/forum?id=1NIyjrtxJU
36. KTO: Model Alignment as Prospect Theoretic Optimization \- arXiv, https://arxiv.org/pdf/2402.01306
37. Kahneman-Tversky Optimization(KTO): Revolutionizing Language Model Training with Prospect Theory | by Yatin Arora | Medium, https://medium.com/@SpielmitDaten/kahneman-tversky-optimization-kto-revolutionizing-language-model-training-with-prospect-theory-99f30c50481e
38. Teaching According to Talents\! Instruction Tuning LLMs with Competence-Aware Curriculum Learning \- arXiv, https://arxiv.org/pdf/2509.13790
39. IT2ACL Learning Easy-to-Hard Instructions via 2-Phase Automated Curriculum Learning for Large Language Models \- ACL Anthology, https://aclanthology.org/2024.lrec-main.822.pdf
40. A Comprehensive Survey of Contamination Detection Methods in Large Language Models, https://arxiv.org/html/2404.00699v4
41. NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark | OpenReview, https://openreview.net/forum?id=KivNpBsfAS¬eId=2ZoRYDfF6u
42. InPage-to-Unicode Converter \- Emergent Mind, https://www.emergentmind.com/topics/inpage-to-unicode-converter
43. (OSCAR) Open Super-large: Crawled ALMAnaCH coRpus — Case Study \- Medium, https://medium.com/@jolalf/oscar-open-super-large-crawled-almanach-corpus-case-study-abb146f963bb
44. SplitGuard: A Resource Efficient Framework for Auditing Train and Eval Overlap and Near Duplicate Contamination in NLP Datasets \- TechRxiv, https://www.techrxiv.org/doi/pdf/10.36227/techrxiv.177205027.77074976
45. Aman's AI Journal • Primers • Data Filtering, https://aman.ai/primers/ai/data-filtering/
46. Evaluating Latent Knowledge of Public Tabular Datasets in Large Language Models \- arXiv, https://arxiv.org/html/2510.20351v2