Scientific Evaluation of Registry-Backed Semantic Resolvers

Full report

On this page

1. Executive Summary

The transition from heuristic-based information retrieval to robust, registry-backed semantic resolution necessitates a fundamental paradigm shift in model evaluation methodologies. Within the operational framework of EmbeddedSemantics.com, the resolver architecture explicitly decouples candidate retrieval from semantic authority. Vector embeddings are strictly relegated to their optimal role: serving as a high-recall filter to retrieve a plausible candidate set of Concept IDs from an immense combinatorial search space. The subsequent decision layer, acting as the definitive semantic authority, evaluates the evidentiary weight of these retrieved candidates to either assert a definitive identity resolution or intentionally abstain. This architectural decoupling renders generic embedding benchmarks, such as the Massive Text Embedding Benchmark (MTEB), fundamentally insufficient for deployment validation. High performance on generalized semantic textual similarity (STS) tasks does not correlate with the strict, zero-tolerance identity resolution required by a clinical or enterprise registry system1. General benchmarks fail to penalize overconfidence on out-of-distribution inputs, ignore the topological distortions caused by hard negatives, and critically omit a system’s ability to selectively abstain when presented with open-set phenomena4. This report details an exhaustive, mathematically rigorous evaluation methodology tailored specifically for the expression-to-concept pipeline. It establishes comprehensive protocols for calibrating probabilistic outputs, establishing conformal prediction thresholds, mining topological hard negatives, detecting embedding drift in production, and validating models through a rigorous, multi-stage acceptance lifecycle.

2. What the Resolver Is Actually Being Evaluated For

A registry-backed semantic resolver is not evaluated on its ability to place loosely related concepts in spatial proximity; it is evaluated exclusively on its ability to assert absolute identity and to refuse resolution when identity cannot be unequivocally established. The architecture dictates that embeddings serve only as a computationally efficient sieve. The true metric of systemic success is the model's calibration and its capacity for selective prediction. The evaluation framework measures the system across three distinct, sequentially dependent phases. The first phase, search space reduction, evaluates whether the vector space reliably places the true, ground-truth Concept ID within the top\-[Figure omitted from source export] neighborhood, recognizing that downstream semantic authorities cannot resolve concepts that are never surfaced4. The second phase, semantic authority, tests whether the decision layer can distinguish the exact target concept from a closely related but distinct ontology neighbor, essentially demanding a transition from semantic similarity to exact entity linking7. The final phase, selective prediction, assesses whether the system accurately quantifies its own epistemic uncertainty, allowing it to abstain safely when presented with unseen concepts, nonsensical inputs, or deeply ambiguous expressions8. A model that perfectly ranks candidates on in-distribution data but fails to abstain on an unresolvable query is fundamentally flawed and poses a severe risk in a registry context.

3. Retrieval Metrics

The evaluation of the candidate retrieval phase assumes a closed-world retrieval task where the primary objective is to surface the correct Concept ID from a static registry. The metrics deployed in this phase assess the topological health of the embedding space, ensuring that the necessary raw material is provided to the semantic authority layer.

4. Identity Resolution Metrics

Once a candidate set is successfully retrieved, the system must establish definitive identity. The metrics in this phase evaluate the exactness and ambiguity of the final selection, transitioning from retrieval evaluation to classification evaluation.

5. Abstention Metrics

The requirement that the system must be allowed to abstain transforms the evaluation paradigm from standard multiclass classification to selective classification, historically known as classification with a reject option9. Treating abstention as a first-class outcome demands metrics that evaluate the risk-coverage tradeoff. The definitive metric for assessing selective classification performance is the Area Under the Risk-Coverage Curve (AURC)15. To compute the AURC, the dataset is sorted monotonically by the model's confidence scores. For any given coverage fraction [Figure omitted from source export] (representing the percentage of total queries the model elects to answer), there exists a corresponding empirical risk [Figure omitted from source export], which is the error rate within that covered subset. The AURC represents the integral of this curve, expressed mathematically as [Figure omitted from source export]15. A lower AURC indicates that the model effectively defers its most error-prone, high-uncertainty predictions to a human-in-the-loop or fallback system16. Beyond the AURC, operational telemetry requires tracking Abstention Precision and Abstention Recall. Abstention Precision measures the percentage of abstained queries that were genuinely unresolvable due to out-of-distribution characteristics, deep ambiguity, or missing ontology entries. Conversely, Abstention Recall measures the percentage of all genuinely unresolvable queries in the test set that the model successfully identified and refused to answer8. Finally, the Empirical Coverage Rate must be continuously monitored to determine the actual throughput of the automated system at a given confidence threshold, ensuring the system does not silently degrade into total abstention.

6. Calibration

Calibration is the mathematical transformation of raw semantic similarity scores—such as cosine similarities or inner products—into statistically defensible probability distributions. A perfectly calibrated model ensures that an output confidence of 0.90 correlates precisely with a 90% empirical correctness rate19. Establishing a defensible resolution threshold relies entirely on rigorous calibration rather than the invention of arbitrary cosine cutoffs, which fluctuate wildly between models and embedding spaces21. The evaluation of calibration relies on the Expected Calibration Error (ECE) and the Brier score. ECE is calculated by partitioning predictions into [Figure omitted from source export] equally spaced bins based on their confidence levels. It computes the weighted average of the absolute difference between the empirical accuracy and the average confidence in each bin19. While ECE provides an intuitive scalar for miscalibration, the Brier score serves as a proper scoring rule that measures the mean squared difference between predicted probabilities and actual binary outcomes. The Brier score can be algebraically decomposed into three distinct components: calibration (reliability), resolution (discrimination), and uncertainty, offering deeper diagnostic insights into model behavior23. To correct miscalibration, models must apply post-hoc calibration algorithms. Platt Scaling, or temperature scaling, utilizes a parametric approach by fitting a logistic regression model to map raw logits to probabilities. It is highly effective for deep neural networks but assumes a strictly sigmoidal distortion22. Isotonic Regression, alternatively, is a non-parametric method that fits a piecewise constant, monotonically increasing function to minimize the Brier score. Isotonic regression is superior for mapping complex, non-linear miscalibrations but requires a significantly larger calibration dataset to prevent overfitting14. In a registry-backed semantic resolver, calibration must be executed at a highly granular level. Per-model calibration is standard, but per-language calibration is strictly required, as a model may exhibit excellent calibration in English while demonstrating dangerous overconfidence in Spanish due to disparities in pre-training data volume. Furthermore, the decision layer can utilize margin-based evidence—the mathematical gap between the top-1 and top-2 candidate probabilities—as a critical feature for a secondary calibrated classifier to define dynamic abstention regions6.

7. Open-Set Recognition

Because the semantic resolver operates in an open world, it is statistically guaranteed to encounter user expressions that map to concepts entirely outside the current registry. Open-Set Recognition (OSR) and Out-of-Distribution (OOD) detection are vital defense mechanisms to prevent the confident misclassification of unknown entities28. Distance-based OOD methodologies improve upon standard cosine thresholds by utilizing the Mahalanobis distance within the embedding space. Unlike Euclidean distance, the Mahalanobis distance incorporates the covariance structure of the in-distribution training data, allowing the system to detect when a query falls outside the established information geometry of known concept clusters29. Alternatively, energy-based OOD models translate the final layer logits into a thermodynamic energy score. Energy functions naturally assign lower energy states to in-distribution data and higher energy states to OOD data, creating a more mathematically separable distribution than softmax probabilities, which notoriously suffer from overconfidence when processing inputs far from the decision boundary31. The most statistically rigorous approach to OSR is Conformal Prediction (CP), a distribution-free framework that constructs prediction sets guaranteed to contain the true concept with a user-specified probability, [Figure omitted from source export], without requiring underlying distributional assumptions32. Using Split Conformal Prediction, a non-conformity score—often defined as [Figure omitted from source export], or based on feature-space distances—is calibrated on an exchangeable hold-out set35. In the context of a semantic resolver, CP directly powers the abstention mechanism. If the generated conformal prediction set is empty, the input is rigorously mathematically identified as OOD or highly anomalous, triggering an immediate abstention37. If the conformal set size expands to multiple candidates, the model has identified epistemic uncertainty, which safely triggers a reranking phase or defers the decision to human review38.

8. Hard Negatives

Hard negative mining fundamentally alters the embedding space topology. By forcefully separating difficult edge cases during contrastive learning, the embedding space is reshaped to prioritize deep semantic distinctions over superficial lexical overlap. The evaluation suite must classify negatives precisely to prevent topological collapse and ensure that the embedding space accurately reflects the ontology13.

9. Multilingual Evaluation

Evaluating multilingual embedding models requires identifying and mitigating benchmark bias, where high aggregate performance driven by resource-rich languages (like English) masks catastrophic degradation in low-resource languages1. Multilingual evaluation must test cross-language equivalents, verifying whether an expression in French resolves to the exact same abstract Concept ID as its English counterpart, thereby proving true language-agnostic semantic alignment within the vector space. Furthermore, the system must be evaluated on translation mismatch detection. This measures the model's ability to abstain when a culturally specific term or localized colloquialism lacks a direct, rigorous semantic equivalent in the target registry. Calibration metrics, specifically ECE and Brier scores, must be disaggregated and reported per language. A unified threshold is mathematically indefensible; resolution thresholds must be dynamically parameterized per language to account for varying levels of epistemic uncertainty across different linguistic corpora.

10. Dataset Design

The evaluation dataset must be meticulously stratified to test the extreme limits of semantic resolution; it cannot rely on randomly sampled query-document pairs typical of generic IR tasks. Every query within the dataset must operate as a golden anchor, possessing a verified, human-adjudicated link to a specific Concept ID, or possessing a verified "unresolvable" flag. Dataset design must enforce strict class imbalance constraints. Real-world concept registries follow Zipfian distributions, where a small number of concepts account for the vast majority of queries. However, standard marginal conformal prediction severely under-covers rare, minority classes6. Therefore, the evaluation dataset must oversample the tail of the distribution to ensure statistical validity and guarantee that class-conditional (Mondrian) conformal prediction maintains its target coverage across rare but critical concepts6. The dataset must also incorporate explicit forbidden pairs—mappings of queries to Concept IDs that the model must explicitly reject—acting as a direct, penalizing test of semantic hard negative separation.

11. Benchmark Leakage

Large language models and deep embedding encoders often inadvertently ingest public evaluation benchmarks during pre-training, rendering metric improvements entirely illusory13. To ensure the integrity of the evaluation suite, rigorous decontamination protocols must be enforced. This involves executing [Figure omitted from source export]\-gram overlap hashing and semantic embedding similarity scans against the model's known pre-training corpora to identify and filter out leaked samples. Furthermore, to combat generalized benchmark leakage, a significant portion of the evaluation suite must consist of dynamic benchmarks. These are continuously generated, novel expressions—synthesized from recent news corpora or real-time user query logs—that mathematically guarantee the expressions have not been seen in historical pre-training runs, ensuring the evaluation measures true generalization rather than data memorization.

12. Adversarial Cases

The semantic resolver must prove its robustness against inputs intentionally designed to force a hallucinated resolution or exploit vector space vulnerabilities. Adversarial evaluation includes testing against prompt injections, assessing whether instruction-tuned embedding models can be hijacked by hidden commands designed to force a specific concept retrieval. The evaluation must also measure performance on contextual ambiguity. For example, the query "The bat broke" could refer to an animal or sports equipment. A robust system must abstain or produce a multi-item conformal prediction set rather than confidently guessing based on training data priors. Finally, typographical perturbations—including character-level noise, Levenshtein distance mutations, and simulated OCR errors—must be injected to ensure the model retains its semantic focus without degrading into lexical fragility40.

13. Proposed EmbeddedSemantics Evaluation Suite

To enforce this rigorous methodology, EmbeddedSemantics requires a dedicated, programmatic evaluation harness designed strictly for the expression-to-concept pipeline. This suite executes a fully automated, multi-stage pipeline:

1. Ingestion: Processes the model artifacts, vocabulary, and tokenizers in an isolated environment.

2. Indexing: Encodes the entire current state of the concept registry into the vector space, establishing the semantic baseline.

3. Retrieval Phase: Executes dense retrieval against the 15 predefined case categories, calculating Recall@K, MRR, and NDCG to assess search space reduction.

4. Scoring Phase: The secondary decision layer generates probabilities and constructs conformal prediction sets utilizing the calibrated non-conformity scores.

5. Calibration & Reporting Phase: Computes ECE, Brier score, and AURC, generating reliability diagrams to visualize the risk-coverage tradeoff.

6. Gatekeeping: Automatically compares the generated metrics against the previous production candidate using statistical significance testing to determine pipeline progression.

14. Case Categories

The evaluation dataset must be uniformly structured and partitioned across 15 distinct, linguistically and semantically challenging test families to expose specific model vulnerabilities.

15. Minimum Dataset Sizes

Statistical significance demands rigorous dataset sizing; inadequate sampling guarantees that reported metric improvements are merely stochastic noise. To estimate overall general performance with a 95% confidence level and a 1% margin of error, a minimum of 10,000 carefully curated, golden samples is required. Furthermore, to ensure statistical power for independent analysis, each of the 15 case categories requires a strict minimum of 1,000 samples. Split conformal prediction also dictates the necessity of a dedicated, pristine calibration set—typically comprising 10% to 20% of the total available data. This calibration set must be strictly quarantined from both the training regimen and the primary metric evaluation datasets to prevent data leakage and ensure the mathematical validity of the conformal thresholds35.

16. Human Review Methodology

The generation of the golden evaluation sets requires extensive human expert adjudication. Relying on simple percentage agreement between human annotators is statistically flawed and obscures underlying biases. The methodology must utilize robust Inter-Annotator Agreement (IAA) metrics. Fleiss' Kappa must be deployed for categorical assignments across multiple annotators, while Krippendorff's Alpha is required due to its flexibility in handling missing data and varying data types across the complex annotation schema42. To ensure the integrity of the ground truth, an adjudication threshold must be established: any query-concept pair failing to achieve a Krippendorff's Alpha of [Figure omitted from source export] is deemed highly ambiguous. These pairs are subsequently routed to a master taxonomist for final resolution or are explicitly reclassified as "Entity Ambiguity" cases to test the model's abstention capabilities.

17. Model Comparison Protocol

It is a critical methodological error to assume a model's superiority based on raw metric increases (e.g., a \+0.5% gain in Hit@1). Statistical significance testing is mandatory to isolate genuine algorithmic improvements from random stochastic variance introduced by training dynamics or data batching43. McNemar's Test is utilized for comparing the predictive accuracy of two models on the same test set. It constructs a [Figure omitted from source export] contingency table evaluating instances where Model A succeeds and Model B fails, and vice versa, determining if the marginal frequencies are statistically equivalent43. For complex, non-binary metrics such as ECE, MRR, or AURC, Permutation Tests must be employed. These tests iteratively and randomly swap the predictions of the baseline and candidate models to build a null distribution, allowing for the calculation of an empirical [Figure omitted from source export]\-value regarding the observed performance difference43. Furthermore, bootstrap resampling must be used to generate 95% confidence intervals for all metrics; a candidate model is only deemed superior if its confidence interval demonstrates non-overlapping superiority over the baseline.

18. Regression Testing

A robust CI/CD pipeline for machine learning must enforce strict regression bounds to maintain trust in the semantic resolver. This requires the establishment of Concept Stability protocols. When updating models, previously correctly resolved, highly critical concepts—such as life-saving medical codes or high-value financial entities—must remain correctly resolved. Introducing a new model that improves overall accuracy but induces regressions on critical infrastructure terms is unacceptable. To enforce this, automated gates execute the evaluation suite against a static, immutable "Golden 1,000" dataset. Any degradation in performance on this critical subset immediately fails the build pipeline, preventing catastrophic forgetting from reaching production environments.

19. Drift Detection

Once a model is deployed into production, the underlying data distribution will inevitably shift. Monitoring this shift is as critical as offline evaluation. Embedding drift occurs when the semantic representation of incoming queries shifts over time—driven by emerging slang, new domain terminology, or altered user behavior—and must be measured without relying on ground-truth labels, which are unavailable in real-time46. The Population Stability Index (PSI) is deployed to quantify the shift in the distribution of predicted confidence scores or embedding cluster assignments. An observed [Figure omitted from source export] indicates moderate drift requiring investigation, while a [Figure omitted from source export] indicates severe drift necessitating model retraining48. Additionally, the Wasserstein Distance (Earth Mover's Distance) calculates the topological effort required to transform the current production embedding distribution into the baseline training distribution. This metric is highly effective for detecting complex, multi-dimensional semantic shifts in unstructured text data48.

20. Production Monitoring

Real-time production telemetry must extend far beyond standard software engineering metrics like latency and throughput, delving into the statistical health of the model. Abstention Rate Monitoring tracks the empirical coverage rate in real-time. A sudden spike in abstentions strongly indicates an influx of OOD data or malicious inputs, whereas a sudden, unexplained drop indicates calibration collapse, wherein the model has become dangerously overconfident on novel data. Furthermore, monitoring the relationship between Latency and Accuracy is vital. If the conformal prediction sets generated by the model consistently grow larger—indicating rising epistemic uncertainty—the downstream processing required to evaluate those sets will increase, degrading system latency. A latency spike driven by expanded conformal sets serves as a leading indicator of a fundamental mismatch occurring in the upstream retrieval embeddings.

21. What Counts as Model Improvement

A candidate model is formally recognized as an improvement, and thereby eligible for production consideration, if and only if it meets all of the following multi-dimensional criteria simultaneously:

22. What Must Block Production Activation

A model must be immediately blocked from production activation, regardless of how highly it scores on generic benchmarks like MTEB or STS, if any of the following critical failures occur:

• Catastrophic Forgetting: Any observed regression on the "Golden 1,000" dataset exceeds 0.5%, indicating instability in mission-critical concept resolution.
• Calibration Collapse: The overall ECE exceeds 0.10, indicating severe overconfidence that will bypass abstention thresholds and force erroneous resolutions.
• OOD Blindness: The model fails to abstain on [Figure omitted from source export] of the queries within the "Nonsensical Input" or "Unseen Concept" datasets, proving a failure in conformal risk control.
• Benchmark Contamination: Algorithmic audits reveal that the specific test suite strings were present in the model's pre-training corpus, invalidating the evaluation results.

23. Proposed Public Benchmark Pages

To establish industry leadership and redefine the standards for semantic resolution, EmbeddedSemantics.com should host a public leaderboard distinct from existing generic platforms like MTEB. This leaderboard must rank models based on their AURC, rather than raw hit accuracy, thereby explicitly penalizing models that are incapable of abstaining safely. Reporting must be disaggregated; models should be scored separately across the 15 Case Categories, highlighting their specific structural vulnerabilities (e.g., demonstrating strong performance on Paraphrase tasks but failing catastrophically on Polysemy). Finally, the benchmark page must enforce strict model and version provenance, utilizing cryptographic hashing of model weights, tokenizer configurations, and dataset versions to guarantee absolute reproducibility for the research community.

24. Proposed Charts and Visualizations

Reports and production dashboards must visually communicate epistemic uncertainty and calibration health to stakeholders. Reliability Diagrams (Calibration Curves) must plot empirical accuracy against predicted confidence across defined bins, allowing reviewers to immediately visualize instances of over- or under-confidence20. Risk-Coverage Curves should be utilized to visualize the fundamental tradeoff between the error rate (risk) and the percentage of queries the model is permitted to answer (coverage). The geometry of this curve visually defines the system's abstention efficacy. For embedding analysis, UMAP or t-SNE projections should highlight the spatial separation of the six Hard Negative categories, providing a visual audit of topological density and boundary collapse within the vector space.

25. Proposed Technical Articles

To solidify thought leadership in the domain of applied semantic resolution, the following technical articles should be authored and published based on this methodology:

1. Beyond MTEB: Why Registry Resolution Requires Selective Classification.

2. Calibrating the Void: Implementing Split Conformal Prediction for Entity Abstention.

3. The Topology of Hard Negatives: Hubness, Distortions, and Embedding Collapse.

4. Measuring the Unseen: Energy-Based OOD Detection in Dense Retrieval Architectures.

26. Research Gaps

A rigorous review of current academic and industry literature reveals several methodological gaps requiring ongoing research and development. The management of dynamic calibration sets remains challenging; determining how to continuously update the conformal calibration set in a live production environment without inducing cyclic instability requires further statistical research. Cross-lingual OOD detection presents a significant hurdle, specifically the difficulty in mathematically differentiating between a truly out-of-distribution concept and a poorly translated valid concept. Finally, a rigorous comparison between generative versus contrastive extraction is required to assess whether LLM-based generative extraction, constrained by strict decoding rules, mathematically outperforms calibrated contrastive vector retrieval in handling extreme edge-case taxonomy resolution7.

27. Annotated Bibliography

The methodology defined within this exhaustive report is grounded directly in the latest research spanning uncertainty quantification, selective classification, and model evaluation metrics. The foundational requirement for models to abstain is rooted in the extensive literature on selective classification, pioneered by Chow (1970) and modernized by El-Yaniv and Wiener (2011), with Geifman and El-Yaniv (2017) formally introducing the Area Under the Risk-Coverage Curve (AURC) as the definitive metric for evaluating systems equipped with a rejection option9. The implementation of mathematically defensible abstention thresholds relies heavily on Conformal Prediction and Conformal Risk Control. Originating from the distribution-free statistical guarantees formulated by Shafer and Vovk (2008), recent advances in Split Conformal Prediction and Bayesian Conformal Prediction have enabled highly scalable methods to construct bounded prediction sets33. The utilization of non-conformity scores, notably advanced by Sadinle et al. (2019), allows the system to transition from raw logits to calibrated abstention regions36. Probability calibration techniques are synthesized from the works of Niculescu-Mizil and Caruana, demonstrating the necessity of transforming model confidence to align with empirical accuracy via tools like Isotonic Regression and Platt Scaling, effectively measured through Expected Calibration Error (ECE) and Brier Score decomposition19. Out-of-Distribution detection methodologies draw upon energy-based models and the application of Mahalanobis distance to separate semantic shifts in deep feature spaces28. Finally, the overarching critique of generalized benchmarks, such as MTEB, is supported by recent findings demonstrating their failure to capture non-local entity references, handle hierarchical ontology mapping, and resist benchmark contamination, affirming the necessity of the bespoke evaluation suite outlined in this report1.

Works cited

1. (PDF) MTEB: Massive Text Embedding Benchmark \- ResearchGate, https://www.researchgate.net/publication/364516382\_MTEB\_Massive\_Text\_Embedding\_Benchmark

2. Bridging Language and Items for Retrieval and Recommendation: Benchmarking LLMs as Semantic Encoders \- ACL Anthology, https://aclanthology.org/2026.acl-long.147.pdf

3. A new “sense of position”: from the logic of subject indexing to the geometry of embeddings | Journal of Documentation \- Emerald Insight, https://www.emerald.com/jd/article/82/1/136/1317811/A-new-sense-of-position-from-the-logic-of-subject

4. On the Theoretical Limitations of Embedding-Based Retrieval \- arXiv, https://arxiv.org/html/2508.21038v1

5. A benchmark of text embedding models for semantic harmonization of Alzheimer's disease cohorts \- PMC, https://pmc.ncbi.nlm.nih.gov/articles/PMC12811766/

6. Least Ambiguous Set-Valued Classifiers With Bounded Error Levels | Request PDF, https://www.researchgate.net/publication/307604226\_Least\_Ambiguous\_Set-Valued\_Classifiers\_With\_Bounded\_Error\_Levels

7. SPR-RAG: Semantic Parsing Retriever-Enhanced Question Answering for Power Policy, https://www.mdpi.com/1999-4893/18/12/802

8. Conformal Prediction for Natural Language Processing: A Survey \- ResearchGate, https://www.researchgate.net/publication/386030579\_Conformal\_Prediction\_for\_Natural\_Language\_Processing\_A\_Survey

9. Trusting the Untrustworthy: A Cautionary Tale on the ... \- OpenReview, https://openreview.net/notes/edits/attachment?id=fDdJWYpsWA\&name=pdf

10. KG-Orchestra: An Open-Source Multi-Agent Framework for Evidence-Based Biomedical Knowledge Graphs Enrichment | bioRxiv, https://www.biorxiv.org/content/10.64898/2026.02.18.706536v1.full-text

11. Full article: From query to conversation: LLMs for earth system science dataset discovery, https://www.tandfonline.com/doi/full/10.1080/17538947.2026.2689233

12. IRC-Bench: Recognizing Entities from Contextual Cues in First-Person Reminiscences, https://apartsinprojects.github.io/ImplicitEntities/

13. IRC-Bench: Recognizing Entities from Contextual Cues in First-Person Reminiscences, https://www.mdpi.com/2504-4990/8/7/186

14. Metadata Reconstruction from Values Alone: Recovering Column Semantics in Undocumented Warehouses \- arXiv, https://arxiv.org/html/2608.07946v1

15. AURC — TorchUncertainty Docs, https://torch-uncertainty.github.io/generated/torch\_uncertainty.metrics.classification.AURC.html

16. 1 INTRODUCTION \- arXiv, https://arxiv.org/html/2410.15361v1

17. A Novel Characterization of the Population Area Under the Risk Coverage Curve (AURC) and Rates of Finite Sample Estimators \- ICML 2026, https://icml.cc/virtual/2025/poster/45606

18. Reliable Text-to-SQL with Adaptive Abstention \- arXiv, https://arxiv.org/html/2501.10858v1

19. Uncertainty Quantification | IBM, https://www.ibm.com/think/topics/uncertainty-quantification

20. Estimating Expected Calibration Errors \- ResearchGate, https://www.researchgate.net/publication/354517176\_Estimating\_Expected\_Calibration\_Errors

21. Reliability Diagram for Calibration Calculator \- MetricGate, https://metricgate.com/docs/reliability-diagram-calibration/

22. Calibration Calculator \- Isotonic & Platt Scaling \- MetricGate, https://metricgate.com/docs/calibration-isotonic-platt/

23. Full article: Boldness-Recalibration for Binary Event Predictions \- Taylor & Francis, https://www.tandfonline.com/doi/full/10.1080/00031305.2024.2339266

24. The Confidence Trap: Gender Bias and Predictive Certainty in LLMs \- arXiv, https://arxiv.org/html/2601.07806v1

25. A bias-corrected decomposition of the Brier score | Request PDF \- ResearchGate, https://www.researchgate.net/publication/260408950\_A\_bias-corrected\_decomposition\_of\_the\_Brier\_score

26. Why 99% Confidence Can Mislead AI Models \- Newline.co, https://www.newline.co/@Dipen/why-99percent-confidence-can-mislead-ai-models--79286d1f

27. A Survey of Calibration Process for Black-Box LLMs \- arXiv, https://arxiv.org/html/2412.12767v1

28. (PDF) Toward Open Set Recognition \- ResearchGate, https://www.researchgate.net/publication/236916472\_Toward\_Open\_Set\_Recognition

29. Improving Out-of-Distribution Detection via Dynamic Covariance Calibration \- arXiv, https://arxiv.org/html/2506.09399v1

30. Improving Out-of-Distribution Detection via Dynamic Covariance Calibration \- GitHub, https://raw.githubusercontent.com/mlresearch/v267/main/assets/guo25m/guo25m.pdf

31. OOD detection: overview, output-based methods \- The Carpentries Incubator, https://carpentries-incubator.github.io/fair-explainable-ml/instructor/7a-OOD-detection-output-based.html

32. Conformal inference for reliable single cell RNA-seq annotation \- Oxford Academic, https://academic.oup.com/bioinformatics/article/41/10/btaf521/8257682

33. Bayesian Conformal Prediction as a Decision Risk Problem \- arXiv, https://arxiv.org/html/2602.03331v1

34. Uncertainty-Aware Transformers: Conformal Prediction for Language Models \- arXiv, https://arxiv.org/html/2604.08885v1

35. Bayesian Conformal Prediction via Decision-Theoretic Threshold Selection \- arXiv, https://arxiv.org/html/2602.03331v2

36. Conformal Prediction without Nonconformity Scores \- GitHub, https://raw.githubusercontent.com/mlresearch/v286/main/assets/hanselle25a/hanselle25a.pdf

37. Conformal Predictions for Visual Animal Identification \- MDPI, https://www.mdpi.com/2227-7080/14/4/232

38. The limits of distribution-free conditional predictive inference \- ResearchGate, https://www.researchgate.net/publication/344932790\_The\_limits\_of\_distribution-free\_conditional\_predictive\_inference

39. Entity Alignment with Unlabeled Dangling Cases \- arXiv, https://arxiv.org/html/2403.10978v1

40. Towards Certification of Uncertainty Calibration under Adversarial Attacks \- OpenReview, https://openreview.net/forum?id=uuPkll6i7m

41. ICML Poster False Coverage Proportion Control for Conformal Prediction, https://icml.cc/virtual/2025/poster/45792

42. Annotation Quality: How to Measure It in 2026 \- Prudent Partners, https://prudentpartners.in/annotation-quality-metrics-inter-annotator-agreement/

43. A resampling-based method to evaluate NLI models | Natural Language Engineering, https://www.cambridge.org/core/journals/natural-language-engineering/article/resamplingbased-method-to-evaluate-nli-models/36425029D22D619BF9ADB1FEF910D047

44. An Evaluation of Classification Algorithms Using Mc Nemar's Test \- ResearchGate, https://www.researchgate.net/publication/267448337\_An\_Evaluation\_of\_Classification\_Algorithms\_Using\_Mc\_Nemar's\_Test

45. Accurate Clinical Entity Recognition and Code Mapping of Anatomopathological Reports Using BioClinicalBERT Enhanced by Retrieval-Augmented Generation: A Hybrid Deep Learning Approach, https://www.preprints.org/manuscript/202512.1075

46. LLM Observability: Tutorial & Best Practices | LaunchDarkly, https://launchdarkly.com/blog/llm-observability/

47. AI Model Drift Monitoring: Enterprise Guide to Continuous Evaluation \- Agility at Scale, https://agility-at-scale.com/ai/generative/continuous-evaluation-and-drift-monitoring/

48. AI model drift | AI Governance Lexicon | VerifyWise, https://verifywise.ai/lexicon/ai-model-drift

49. Guide to ML Model Monitoring: Metrics, Drift & Retraining | Snowflake, https://www.snowflake.com/en/artificial-intelligence/observability/model-monitoring/

50. Quantifying and Improving Adaptivity in Conformal Prediction Through Input Transformations, https://ojs.aaai.org/index.php/AAAI/article/view/39370/43331