Rigorous Semantic Generalization: A Scientifically Grounded Pathway for Multilingual Expression Resolution

Full report

On this page

1. Executive summary

The fundamental challenge in deploying multilingual semantic systems lies in bridging the gap between highly governed, exactly resolved expressions and the vast, unstructured, and often ambiguous manifold of natural language. The architecture defined by the Embedded Semantics paradigm demands a stringent operational standard. When an unseen natural language expression is encountered, the system must generate candidate representations, accumulate semantic evidence, and subsequently make a binary choice to either resolve the expression to a stable ConceptCode or abstain. Crucially, the system must exhibit a profound asymmetric risk profile, strongly preferring abstention over silently assigning an incorrect semantic identity. This research report presents a comprehensive, mathematically grounded pathway to architecting this experimental generalization subsystem. The analysis establishes that the optimal objective is not mere vector retrieval or standard multi-class classification, but rather a hybrid of open-set recognition and selective classification, rigorously governed by conformal prediction. By relying on distribution-free uncertainty quantification, the proposed architecture allows for the derivation of concept-specific and language-specific thresholds that mathematically bound the risk of false merges. This document exhaustively details the state-of-the-art in multilingual representation, cross-lingual calibration drift, prototype learning methodologies, evaluation protocols, and the strict dataset separation paradigms required to safely promote experimental models into production authority. The subsequent sections address all critical architectural questions, providing a blueprint for a highly reliable, linguistically equitable semantic engine.

2. Interpretation of the public Embedded Semantics architecture

The public architecture of Embedded Semantics establishes a bipartite mechanism for handling natural language concepts, fundamentally separating deterministic knowledge from probabilistic inference1. The primary track, Production Authority, is a strictly governed system. It operates on reviewed, exact expressions that map directly and immutably to a stable ConceptCode. This track functions as the ground-truth ontology, ensuring that known linguistic inputs bypass probabilistic inference entirely and are resolved with zero epistemic uncertainty. The embedded semantics within this layer provide a formal vocabulary that enables reasoning across diverse data sources without ambiguity1. The secondary track, Experimental Generalization, functions as a probabilistic triage system for unseen natural language expressions. The flow consists of candidate retrieval from a high-dimensional vector space, semantic evidence gathering through distance and entailment metrics, and a final adjudication step mandating a resolution or an abstention. The architectural directive prioritizing abstention over silent failure indicates a high-stakes operational environment where the cost of a false positive (a false semantic merge) vastly outweighs the cost of a false negative (an abstention requiring human review). Therefore, the experimental system is fundamentally an asymmetric risk engine. It must project a wide net during the candidate generation phase to maximize recall but apply draconian, rigorously calibrated thresholds during the semantic identity assignment phase. The architecture implies that experimental models act as active-learning feeders to the Production Authority; expressions successfully and confidently resolved in the experimental tier can, upon human review or strict statistical validation, be promoted to the exact-expression governed tier, thereby expanding the deterministic ontology.

3. Formal problem definition

To satisfy the architectural constraints of the Experimental Generalization track, the semantic assignment task must be mathematically formalized. The core research question asks when there is sufficient evidence to map an unseen expression to an existing governed concept. The correct mathematical objective is neither pure retrieval, which only ranks candidates, nor standard multi-class classification, which assumes a closed set of classes. Instead, the objective is open-set selective classification grounded in semantic entailment5. Let the space of all possible unseen multilingual expressions be denoted as [Figure omitted from source export], and let [Figure omitted from source export] represent the finite, governed set of existing ConceptCodes. The model [Figure omitted from source export] must map an input [Figure omitted from source export] to either a specific concept [Figure omitted from source export] or the abstention symbol [Figure omitted from source export]. The abstention symbol encapsulates two distinct failure modes. The first is epistemic uncertainty, wherein the model cannot confidently distinguish which [Figure omitted from source export] is correct due to ambiguity or overlapping semantic density. The second is out-of-distribution (OOD) novelty, wherein the expression [Figure omitted from source export] maps to a previously undefined concept [Figure omitted from source export]. To optimize this objective, the system requires a scoring function [Figure omitted from source export] that represents the semantic evidence that [Figure omitted from source export] entails [Figure omitted from source export], accompanied by a decision threshold [Figure omitted from source export]. The selective classifier is defined such that [Figure omitted from source export] if [Figure omitted from source export], and [Figure omitted from source export] otherwise. Because linguistic complexity, polysemy, and representation density vary drastically across languages and concepts, [Figure omitted from source export] cannot be a static, global scalar. The problem must be formulated as empirical risk minimization subject to a strict coverage constraint, utilizing the framework of Selective Conformal Risk Control (SCRC)7. This framework ensures that the probability of assigning the wrong ConceptCode, conditional on the system not abstaining, is mathematically bounded by a user-defined maximum risk level [Figure omitted from source export].

4. Candidate retrieval vs semantic identity distinction

Candidate generation and semantic identity assignment must be treated as two distinct computational phases with orthogonal optimization objectives. Conflating these two phases is a primary source of silent failure in modern retrieval-augmented systems. Candidate generation is inherently a high-recall, low-precision operation. Its singular goal is to rapidly scan a massive multilingual vector space to retrieve a manageable subset [Figure omitted from source export] of plausible concepts. This phase relies on Maximum Inner Product Search (MIPS) using dual-encoder architectures8. Candidate generation should intentionally ignore fine-grained entailment nuances, focusing instead on broad topical, lexical, and semantic proximity to ensure the true ConceptCode is not discarded prematurely. Semantic identity assignment, conversely, is a high-precision, low-recall operation focused exclusively on entailment verification. Once the candidate subset [Figure omitted from source export] is retrieved, the system must transition from measuring continuous vector proximity to executing a discrete logical verification, essentially asking if expression [Figure omitted from source export] strictly means Concept [Figure omitted from source export]. This distinction is critical because two phrases can be highly similar in embedding space, such as an assertion versus a negation, yet diverge fundamentally in semantic identity and intent9. Bi-encoders frequently gloss over these structural differences because they pool token representations into a single dense vector, causing a semantic bottleneck. Identity assignment must heavily penalize minor lexical deviations that invert meaning, such as directionality or modality. Therefore, while candidate generation utilizes cosine similarity over dense embeddings, identity assignment must rely on calibrated, temperature-scaled entailment scores, potentially leveraging cross-encoder rerankers or strict margin-based distance thresholds to confirm identity.

5. Review of current multilingual representation approaches

The landscape of multilingual embedding models has advanced significantly, transitioning from English-centric models with bolted-on multilingual capabilities to natively multilingual, multi-representational architectures. Evaluating the state of the art requires abandoning the assumption that a single dense vector is sufficient for capturing cross-lingual nuance. Models such as BGE-M3 (BAAI) have revolutionized representation by supporting over 100 languages simultaneously while generating dense, sparse, and multi-vector ColBERT-style embeddings in a single forward pass8. The integration of sparse BM25-style lexical matching and multi-vector representations is highly beneficial for exact expression resolution, as it directly mitigates the cross-lingual vocabulary collapse where rare, domain-specific terms are lost in dense vector bottlenecks. Other frontier models, such as Cohere Embed v4 and Jina Embeddings v3, have demonstrated state-of-the-art performance on the Massive Text Embedding Benchmark (MTEB) across diverse language pairs12. The Jina v3 architecture uniquely utilizes Matryoshka representation learning, allowing for flexible dimensionality reduction without catastrophic semantic loss8. This is highly advantageous for optimizing the memory footprint in vast candidate retrieval systems while preserving retrieval accuracy. The Granite Multilingual Encoder models employ extensive contrastive learning pipelines alongside knowledge distillation from strong teacher models, yielding highly robust semantic clusters across 1,800 languages15. Furthermore, architectures like ZeroEntropy's zembed-1 emphasize true multilingual parity by maintaining over 50% non-English training data, applying a unified relevance representation that avoids the structural degradation typically seen when non-English queries are mapped into an English-dominant semantic space16. For the Embedded Semantics architecture, relying exclusively on dense embeddings is suboptimal; a hybrid approach combining dense semantics with late-interaction ColBERT or sparse encoders offers the most scientifically defensible baseline for multilingual candidate generation.

6. Review of open-set recognition approaches

Open-set recognition (OSR) addresses the inevitable scenario where an unseen query does not belong to any governed ConceptCode within the Production Authority17. In the context of the Embedded Semantics system, failing to detect an open-set query results in a forced, erroneous merge, violating the core directive of the architecture. Open-set recognition is fundamentally linked to Out-of-Distribution (OOD) detection5. Standard classification models utilize softmax functions that artificially inflate probabilities for known classes, making them dangerously overconfident when presented with novel concepts. A query about a completely unregistered topic will still produce a vector that is nearest to something in the index, leading to a false positive if distance alone is used. Current approaches to OSR focus on distance-based metrics, energy-based models, and density estimation17. Distance-based methods, such as utilizing the Mahalanobis distance in the feature space, measure how far an input is from the Gaussian class centroids of known ConceptCodes18. Energy-based models utilize the denominator of the softmax function to estimate the probability density of the input, assigning higher energy scores to anomalies20. Recent theoretical advancements propose integrating conformal p-values specifically designed for open-set scenarios. By leveraging the Good-Turing estimator, these methods calculate the probability of observing a completely new semantic species based on the frequency of singletons in the calibration data21. This allows the algorithm to dynamically adjust prediction sets to include a placeholder "joker" class representing an unseen concept. The system can mathematically guarantee that no registered Concept is appropriate when the conformal p-value for the OOD class exceeds the evidence scores of all retrieved candidates, triggering an immediate abstention.

7. Review of calibration methods

Calibration ensures that a model's output scores accurately reflect the true empirical probability of correctness. If a semantic model outputs an identity score of 0.9 for a specific ConceptCode, the prediction should be empirically correct exactly 90% of the time. However, deep neural networks are notoriously miscalibrated, frequently exhibiting extreme overconfidence even when producing incorrect predictions6. For a multilingual system, this issue is heavily exacerbated by cross-lingual calibration drift. A model may be perfectly calibrated in a high-resource language like English, but highly overconfident and inaccurate when processing low-resource languages like Arabic or Swahili25. To calibrate a system across linguistic boundaries, global scaling methods like standard Platt scaling or unconstrained temperature scaling are mathematically insufficient. Recent research indicates that language-specific temperature scaling is strictly required25. This methodology involves partitioning the calibration dataset by language or script and learning independent scaling parameters for each linguistic subset. The calibrated probability is computed as [Figure omitted from source export], where [Figure omitted from source export] is the learned temperature specifically for language [Figure omitted from source export]. By applying language-specific scaling, the system compensates for the varying epistemic uncertainty inherent in different linguistic representations. Furthermore, advanced calibration techniques utilize validation datasets to non-parametrically map heuristic uncertainty scores into strict confidence intervals7. It is scientifically indefensible to calibrate a system solely on high-resource languages and assume that the calibration holds globally. Continuous monitoring of the cross-lingual expected calibration error (ECE) is mandatory to prevent the system from silently failing on minority languages26.

8. Review of abstention/selective-prediction methods

Selective prediction allows a machine learning model to abstain from making a decision when its confidence falls below a specific threshold, directly trading off predictive coverage for accuracy5. In the Embedded Semantics system, abstention is fundamentally preferred over incorrect assignment. The prevailing state-of-the-art framework for rigorously controlling this tradeoff is Conformal Prediction (CP)27. Conformal prediction operates as a post-hoc wrapper that transforms heuristic point predictions into statistically valid prediction sets with user-specified marginal coverage guarantees, ensuring that the true ConceptCode is contained within the output set with a defined probability5. Addressing the key architectural questions regarding thresholds: a global confidence threshold is scientifically indefensible in this domain. Different semantic regions (Concepts) and different languages possess varying levels of spatial density and semantic overlap28. Applying a static global threshold results in severe under-coverage for rare classes, known as tail classes, and minority languages29. Therefore, thresholds must be both Concept-specific and Language-specific. Frameworks such as Tail-Aware Conformal Prediction (TACP) and class-conditional conformal prediction demonstrate that thresholds must be dynamically adjusted based on the prevalence and spatial density of the target class29. Furthermore, Selective Conformal Risk Control (SCRC) explicitly formulates uncertainty control as a two-stage process: selecting confident samples based on a learned policy, then applying conformal risk control to bound the expected loss7. This allows the system to determine exactly when there is enough evidence to resolve an identity by calculating if the conformal prediction set contains exactly one ConceptCode. If the set contains multiple codes, indicating ambiguity, or zero codes, indicating an OOD concept, the system must abstain.

9. Prototype representation strategies

The Production Authority defines exact expressions that map deterministically to a ConceptCode. In the continuous vector space, these exact expressions serve as the ground-truth support vectors for prototype representation. The question of how much reviewed support is needed per Concept depends on the variance of the underlying semantic cluster. While a single exact expression can serve as a prototype for zero-shot or nearest-support classification, establishing a robust class centroid typically requires a minimum of five to ten linguistically diverse expressions to accurately capture morphological and syntactic variance. Uneven language coverage significantly impacts prototype construction. If a ConceptCode possesses fifty English support expressions and only one Arabic expression, a naive class centroid computed via global averaging will be heavily biased toward the English semantic subspace. Consequently, the Arabic prototype representation will collapse, causing future Arabic queries to be falsely rejected or misclassified. To prevent this, prototype learning must employ language-balanced representations. Rather than averaging all support vectors globally, the system should compute language-specific sub-centroids. When an unseen Spanish expression is queried, the system should compute its distance specifically against the Spanish sub-centroid of the ConceptCode, or utilize a k-Nearest Neighbors (k-NN) approach restricted to cross-lingual alignments. If language-specific support is entirely missing, cross-lingual transfer can be cautiously utilized by projecting the English centroid through a calibrated cross-lingual alignment matrix31, though this increases epistemic uncertainty and must trigger stricter abstention thresholds to maintain safety.

10. Hard-negative strategy

Constructing hard semantic neighbors is critical for fine-tuning embedding models and establishing precise decision boundaries around governed concepts. A false merge occurs specifically when two conceptually distinct phrases are embedded too closely in the vector space. Hard negatives are defined as expressions that are lexically similar, syntactically identical, or contextually adjacent, but belong to mutually exclusive ConceptCodes. The construction of hard semantic neighbors should follow a multi-tiered strategy. First, intra-batch hard negative mining during contrastive learning forces the model to separate highly related concepts that share overlapping vocabulary15. Second, syntactic perturbations should be generated computationally to create adversarial examples. This includes altering negations, flipping directionality, and swapping domain-specific entities. For example, the phrases "requires a permit" and "does not require a permit" must be treated as hard negatives, ensuring the model does not ignore crucial operational modifiers. Finally, historical false merges, where the experimental system previously failed during human review, must be recycled as explicit hard negatives in all subsequent training epochs. By minimizing the margin loss between the true prototype and these hard negatives, the embedding space is regularized to respect strict semantic boundaries rather than loose topical affiliations.

11. Multilingual evaluation methodology

Evaluating a semantic generalization subsystem requires measuring multilingual parity, defined as the degree to which the system provides equitable accuracy, calibration, and risk control across all supported languages16. Multilingual parity cannot be measured merely by averaging global accuracy across the entire dataset, as high-resource languages will invariably mask the catastrophic failure rates of low-resource languages. Instead, parity must be quantified using cross-lingual expected calibration error (ECE) and conditional selective risk25. The evaluation methodology must involve slicing the datasets rigidly by language family. To accurately measure parity, parallel translation corpora should be used to query the same underlying ConceptCode across fifty or more languages. The variance in Recall@k, the difference in conformal prediction set sizes at a fixed confidence level [Figure omitted from source export], and the abstention rate delta between the highest-resource and lowest-resource languages serve as the primary metrics for parity. A system achieves acceptable multilingual parity only when the selective risk, which is the error rate on non-abstained predictions, is statistically indistinguishable across language slices. It is acceptable for the coverage, or abstention rate, to naturally vary due to resource limitations, provided the risk remains firmly bounded.

12. Cross-script evaluation

Multilingual models often struggle heavily at the boundaries of different orthographic scripts, such as Latin, Cyrillic, Arabic, and Hanzi. These boundaries frequently lead to fragmented semantic spaces where cross-script alignment fails entirely, forcing the model to rely on character-level overlap rather than semantic intent. Cross-script evaluation must systematically isolate transliteration phenomena, code-switching, and script variations to ensure robust generalization. The evaluation protocol should intentionally inject exact expressions transliterated into non-native scripts, such as writing Arabic phrases in Latin characters or vice-versa, to measure embedding robustness8. The vector distance between the native script expression and its transliterated counterpart must be quantified. If the distance exceeds the intra-class variance of the target ConceptCode, the model exhibits severe script-bias and fails to capture underlying semantics. Furthermore, cross-script evaluation must heavily penalize models that cluster by script topology rather than meaning. Evaluation sets must include adversarial cross-script queries where lexical overlap is artificially high but semantic overlap is zero, ensuring that the model does not silently degrade into a naive string-matching engine when facing non-Latin inputs.

13. Ambiguity evaluation

Ambiguity arises when an unseen expression legitimately entails multiple ConceptCodes or lacks sufficient contextual information to specify exactly one. Representing ambiguity mathematically is a core strength of the conformal prediction framework27. Rather than forcing a singular softmax output and relying on arbitrary confidence thresholds, ambiguity is natively represented by the cardinality of the prediction set. If the conformal prediction set [Figure omitted from source export] contains more than one element, [Figure omitted from source export], the expression is formally ambiguous at the [Figure omitted from source export] confidence level. Ambiguity evaluation must rigorously measure the system's ability to output correctly sized sets for polysemous queries. The evaluation dataset must contain intentionally vague expressions that have been manually mapped by annotators to multiple valid ConceptCodes. The primary evaluation metric is the exact match of the predicted set to the ground-truth set of valid interpretations. A rigorous system will always trigger an abstention event when [Figure omitted from source export], routing the ambiguous expression back to human review for potential splitting, disambiguation, or clarification in the Production Authority.

14. Out-of-domain evaluation

Out-of-domain (OOD) evaluation tests the system's resilience to semantic drift and domain shifts19. For the Embedded Semantics system, OOD scenarios occur when users submit natural language expressions that are entirely unrelated to the governed ontology, or pertain to concepts that have not yet been formalized into a ConceptCode. The evaluation pipeline must inject extreme OOD data, such as medical queries fed into a financial ontology system, alongside near-OOD data, consisting of queries topically related but lacking an exact ConceptCode match. The primary evaluation metric for OOD resilience is the Abstention Rate on OOD inputs, which must theoretically approach 100%. Utilizing conformal p-values derived from Good-Turing estimators alongside distance-based thresholding, the evaluation will measure the Area Under the Receiver Operating Characteristic (AUROC) for distinguishing between in-domain unseen expressions and true OOD expressions18. A high conformal AUROC indicates that the system safely abstains when operating outside its epistemic bounds, avoiding catastrophic false merges.

15. Proposed benchmark architecture

The proposed benchmark architecture must be custom-built to reflect the strict dual-track nature and asymmetric risk profile of the Embedded Semantics system. Standard benchmarks like MTEB are insufficient because they prioritize generic topical retrieval and symmetric accuracy rather than exact entailment resolution and abstention dynamics12. The architecture comprises three sequential evaluation pipelines designed to test the system at every phase of generalization.

This benchmark must include specialized data partitions for Hard Negatives, Cross-Lingual Parity, and OOD data, ultimately outputting a unified "Semantic Generalization Trust Score" that penalizes false merges logarithmically heavier than abstentions.

16. Proposed train/calibration/test separation

To prevent threshold leakage and ensure that unseen-query test sets are kept truly held out, the data partitioning methodology must be strictly controlled. Standard random splitting is deeply flawed in highly imbalanced, open-set settings because it fails to preserve the long-tail distribution of the semantic space21. The data must be partitioned using Selective Sample Splitting based on label frequency and linguistic distribution.

1. Training Set (60%): Utilized exclusively for representation learning, fine-tuning embeddings via contrastive loss, and constructing prototype centroids.

2. Calibration Set (20%): Strictly held out from training. Used exclusively to compute non-conformity scores, apply language-specific temperature scaling, and establish the conformal thresholds [Figure omitted from source export]27. By learning the thresholds on data entirely unseen by the embedding model, threshold leakage is mathematically prevented.

3. Test Set (20%): Sequestered until final evaluation. This set is utilized to compute the unbiased empirical risk, False Coverage Rate, and coverage metrics.

Importantly, no ConceptCode overlapping exact expressions can exist between Train and Test splits. The Test set must consist solely of expressions that the model has never encountered, ensuring the evaluation reflects true zero-shot and few-shot generalization capabilities.

17. Recommended metrics

To effectively detect false semantic merges and evaluate system safety, traditional metrics such as global accuracy and F1-score must be subordinated to risk-focused metrics. The recommended metrics for the generalization subsystem are detailed below.

18. Proposed promotion gates

Promotion gates act as automated statistical barriers that experimental models must clear before they can be merged into the Production Authority track. These gates ensure that no model degrades the determinism of the governed ontology. The promotion sequence requires clearing four distinct gates. First, the model must pass the Base Retrieval Sufficiency Gate, where Recall@10 must exceed 99.0% on the held-out test set. If candidate generation fails to find the target, the entailment phase is irrelevant. Second, the model must pass the Conformal Risk Control Guarantee Gate. Using the calibration set, the model must demonstrate that it can achieve a bounded Selective Risk of [Figure omitted from source export] while maintaining a Coverage of at least [Figure omitted from source export]. If the model must abstain on [Figure omitted from source export] of data to hit the risk target, it is not viable for promotion. Third, the model must pass the Multilingual Parity Compliance Gate. The Selective Risk must not exceed [Figure omitted from source export] on any individual language sub-slice25. If English yields [Figure omitted from source export] risk but Arabic yields [Figure omitted from source export] risk, the model fails the parity gate. Finally, the model must clear the OOD Rejection Gate, proving it successfully abstains on [Figure omitted from source export] of injected open-set anomaly queries20.

19. Recommended model experiments ranked by expected information value

To empirically determine the strongest scientific path forward, the following experiments should be executed in order of highest expected information value.

20. Reranker decision criteria

A reranker, typically implemented as a cross-encoder, is scientifically justified when the bi-encoder exhibits high retrieval recall but poor entailment precision10. Bi-encoders compress the entire expression into a single vector, causing semantic bottlenecking where critical logical modifiers are irreversibly lost. A reranker should be invoked exclusively during the semantic identity assignment phase on the top [Figure omitted from source export] candidates, where [Figure omitted from source export] is small (e.g., [Figure omitted from source export]), retrieved by the bi-encoder. The decision to implement a reranker should be based on three criteria. First, the Precision Delta: the cross-encoder must improve Precision@1 by more than 5% absolute over the bi-encoder's dot-product scoring. Second, the Computational Budget: cross-encoders require full attention over the concatenated query-candidate pair, creating exponential latency scaling. It is justified only if the inference latency SLA permits a secondary transformer forward-pass. Third, Calibration Quality: rerankers trained on Natural Language Inference (NLI) datasets frequently yield superior calibrated probabilities compared to cosine similarities, making them highly synergistic with conformal prediction pipelines.

21. LLM-based adjudication analysis

The integration of generative Large Language Models (LLMs) into the generalization subsystem offers both profound capabilities and severe scientific risks. An LLM is useful primarily as a zero-shot logical adjudicator for highly complex, multi-clause expressions where dense embeddings fail to capture compositionality. Frameworks like ReCoVERR utilize LLMs to ask related questions, extract reliable evidence from the context, and verify semantic entailment before finalizing a prediction10. However, relying on an LLM makes the system significantly less trustworthy in several critical ways. First, LLMs are known to suffer from severe calibration drift, particularly in multilingual settings; they frequently output confidently hallucinated entailments in low-resource languages25. Second, LLM outputs are highly sensitive to prompt phrasing, breaking the deterministic stability expected in a governed architecture. Third, evaluating the uncertainty of an LLM requires expensive multi-sample techniques like Semantic Entropy or Word-Sequence Entropy24, which vastly inflate latency and operational costs. Therefore, an LLM should only be utilized as an offline annotator for expanding the exact expression support sets in the Production Authority, or as a final tie-breaker for highly ambiguous inputs routed to human review, but never as an unconstrained inline semantic identity assigner.

22. Major scientific risks

The architecture faces several major scientific risks that must be actively mitigated. The first is Multilingual Representation Collapse. This occurs when training predominantly on English datasets forces the embedding space of non-English languages to contort into unnatural topologies, effectively destroying semantic resolution for native speakers of those languages16. The second major risk is False Semantic Merges via OOD Contamination. Without rigorous open-set recognition utilizing Good-Turing estimators and energy scores, the system will inevitably map entirely novel concepts into the nearest existing ConceptCode, permanently polluting the governed ontology20. The third risk is Over-Abstention, or Coverage Collapse. By implementing overly draconian conformal thresholds to prevent false merges, the system may abstain on 99% of valid inputs to maintain a 1% risk level, rendering the experimental generalization track functionally useless and overwhelming human reviewers10. Finally, Threshold Leakage poses a critical threat to the mathematical validity of the system. If calibration thresholds are dynamically adjusted based on test set feedback rather than a rigorously sequestered calibration set, the statistical guarantees of conformal prediction mathematically evaporate.

23. Failure examples

To conceptualize the failure modes of the generalization subsystem, three specific scenarios highlight the limitations of naive semantic matching. The Negation Failure demonstrates the fundamental flaw of bi-encoders. An unseen query states, "Patient does not exhibit signs of hypertension." The candidate retrieved is ConceptCode: HYPERTENSION\_OBSERVED. The dense embedding focuses heavily on the semantic weight of the token "hypertension" and ignores the logical negation "not." Consequently, the system silently assigns the wrong code instead of abstaining, creating a critical error. The Cross-Lingual Calibration Failure illustrates the danger of uniform thresholding. An unseen Arabic query asks, "متى يغلق المسبح؟" (When does the pool close?). The candidate retrieved is ConceptCode: POOL\_OPENING\_HOURS. The Arabic representation is poorly calibrated, outputting a 0.98 confidence score despite a direct semantic mismatch between closing and opening. The system bypasses the abstention gate due to false overconfidence, whereas an English equivalent would have been correctly flagged as low-confidence. The Open-Set Forced Merge highlights the lack of novelty detection. An unseen query requests, "Requesting a permit for a commercial drone flight." The candidate retrieved is ConceptCode: COMMERCIAL\_AIRCRAFT\_PERMIT. Commercial drones are not yet defined in the governed ontology. The system, lacking a Good-Turing OOD estimator, maps the drone query to the nearest neighbor (commercial aircraft), forcing a merge that creates a severe downstream legal and operational failure21.

24. Recommended next three experiments

To operationalize the findings of this research, the engineering teams must immediately execute three foundational experiments. First, execute a Baseline MIPS with SCRC Calibration. Implement a multi-vector model like BGE-M3 to retrieve the top candidates, then pass the similarities through a Selective Conformal Risk Control (SCRC) calibration layer on a held-out dataset. This will determine the baseline empirical tradeoff curve between coverage and risk. Second, conduct a Multilingual Temperature Scaling Grid Search. Execute a strict partition of the calibration set by language family. Train independent temperature scaling variables for each language and measure the reduction in Cross-Lingual Expected Calibration Error (ECE) to quantify the mitigation of overconfidence in non-English inputs25. Third, execute an OOD Conformal p-value Injection. Introduce synthetic out-of-domain queries into the evaluation pipeline and implement the Good-Turing estimator conformal p-value to dynamically append an "Abstain" candidate to every prediction set. Measure the AUROC of the system's ability to reject these OOD queries without dropping coverage on in-domain queries21.

25. Exact evidence that should be required before production activation

Before any experimental generalization subsystem is permitted to automatically promote expressions to the Production Authority, the algorithmic evidence must be mathematically incontrovertible. The required evidence consists of three distinct pillars. First, Unambiguous Set Cardinality is required. The conformal prediction set [Figure omitted from source export] computed at the highly conservative [Figure omitted from source export] confidence level (where [Figure omitted from source export]) must contain exactly one element, proving that the model is completely unconflicted regarding the identity of the concept. Second, OOD Margin Verification must be satisfied. The distance between the query and the selected ConceptCode prototype must be statistically proven to reside within the in-distribution manifold. This is validated by an OOD energy score falling strictly below the 95th percentile of the calibration distribution18. Finally, Cross-Encoder Entailment must be confirmed. A distinct cross-encoder NLI model must evaluate the pair consisting of the unseen expression and the ConceptCode canonical description, returning a strict entailment probability [Figure omitted from source export]. This ensures that the dense retrieval vector match is logically and compositionally consistent10. If and only if all three conditions are mathematically satisfied, the evidence is deemed sufficient to bypass abstention and execute a safe semantic generalization.

26. Literature and Source Documentation Analysis

To satisfy the requirement for bibliographic tracking while maintaining the structure of a continuous research report, the following analysis details the primary technical sources, model documentation, and peer-reviewed literature utilized to construct this architecture. The integration of conformal prediction for open-set and selective classification relies heavily on the foundational distribution-free uncertainty quantification frameworks developed by Angelopoulos and Bates, specifically the 2021 preprint (arXiv:2107.07511) and the 2022 Conformal Risk Control extension (arXiv:2208.02814)27. The transition to open-set classification is documented in the 2025 work on conformal prediction in highly imbalanced and open-set settings (arXiv:2510.13037), which introduces the critical Good-Turing estimator methodologies21. For multilingual representation, the architectural guidelines draw upon the official documentation for Jina Embeddings v3 (September 2024, jina.ai) and IBM's Granite Embedding Multilingual models (2025, Hugging Face)13. The imperative for true multilingual parity and language-specific temperature scaling is supported by the 2026 ZeroEntropy documentation (zeroentropy.dev) and the recent analysis of multilingual calibration drift in LLM-as-a-judge frameworks (Preprints 202603.1324)16. Furthermore, the conceptual separation of selective classification and semantic abstention is grounded in the pioneering work by Geifman and El-Yaniv (2017) and modernized by the 2025 Selective Conformal Risk Control models (arXiv:2512.12844)7. These foundational sources collectively validate the assertion that a safe semantic generalization system must rely on mathematically bounded uncertainty control rather than static heuristic thresholds.

Works cited

1. Semantic Layer — The Knowledge Graph Guys, https://www.knowledge-graph-guys.com/blog/the-semantic-layer

2. ConceptUtil (Fish CORE-26.1.4-SNAPSHOT API) \- Casetivity Docs, https://docs.casetivity.com/CORE-26.1.4-SNAPSHOT/com/ssgllc/fish/service/util/registered/ConceptUtil.html

3. Semantic Web \- Wikipedia, https://en.wikipedia.org/wiki/Semantic\_Web

4. Advances in Semantic Web \- Scirp.org., https://www.scirp.org/book/detailedinforofabook?bookid=2648

5. exploring the link between out-of-distribution detection and conformal prediction with illustrations \- OpenReview, https://openreview.net/pdf?id=GQhlM0Mavg

6. Learning Selective Deferral Policies for Reliable Medical Text Classification \- MDPI, https://www.mdpi.com/2227-7080/14/6/359

7. \[2512.12844\] Selective Conformal Risk Control \- arXiv, https://arxiv.org/abs/2512.12844

8. 8 Embedding Models Compared for Production RAG \[2026 Benchmark\] \- Tensoria, https://tensoria.fr/en/blog/embedding-models-2026-guide

9. Emu: Enhancing Multilingual Sentence Embeddings with Semantic Similarity | by Megagon Labs, https://megagonlabs.medium.com/emu-enhancing-multilingual-sentence-embeddings-with-semantic-similarity-33b9094b92a3

10. Selective “Selective Prediction”: Reducing Unnecessary Abstention in Vision-Language Reasoning \- ACL Anthology, https://aclanthology.org/2024.findings-acl.767.pdf

11. The Best Open-Source Embedding Models in 2026 \- BentoML, https://www.bentoml.com/blog/a-guide-to-open-source-embedding-models

12. Best Embedding Models for RAG in 2026: A Comparison Guide \- StackAI, https://www.stackai.com/insights/best-embedding-models-for-rag-in-2026-a-comparison-guide

13. jina-embeddings-v3 \- Search Foundation Models, https://jina.ai/models/jina-embeddings-v3/

14. Jina Embeddings v3 on Gemini Enterprise Agent Platform Model Garden \- Elastic, https://www.elastic.co/search-labs/blog/jina-embeddings-v3-gemini-enterprise-model-garden

15. Granite Embedding Multilingual R2 Models \- arXiv, https://arxiv.org/html/2605.13521v1

16. The Best Multilingual Embedding Model in 2026: zembed-1 Was Built for the World, https://zeroentropy.dev/articles/best-multilingual-embedding/

17. (PDF) Recent Advances in Open Set Recognition: A Survey (2021, https://scispace.com/papers/recent-advances-in-open-set-recognition-a-survey-51lubm7g0l

18. Out-of-Distribution Detection Should Use Conformal Prediction (and Vice-versa?) \- arXiv, https://arxiv.org/html/2403.11532v1

19. GitHub \- huytransformer/Awesome-Out-Of-Distribution-Detection: Out-of-distribution detection, robustness, and generalization resources. The repository contains a curated list of papers, tutorials, books, videos, articles and open-source libraries etc, https://github.com/huytransformer/Awesome-Out-Of-Distribution-Detection

20. Orthogonal Multimodal Sensing and AI Fusion for the Recognition of Unknown Chemical Threats: A Critical Review \- MDPI, https://www.mdpi.com/2227-9040/14/9/189

21. \[2510.13037\] Conformal Inference for Open-Set and Imbalanced Classification \- arXiv, https://arxiv.org/abs/2510.13037

22. Conformal Inference for Open-Set and Imbalanced Classification \- arXiv, https://arxiv.org/html/2510.13037v1

23. Conformal Inference for Open-Set and Imbalanced Classification \- arXiv, https://arxiv.org/pdf/2510.13037

24. Calibrating LLMs for Selective Prediction: Balancing Coverage and Risk \- OpenReview, https://openreview.net/pdf?id=ZVZGjtP5VB

25. CalibJudge: Calibrated LLM-as-a-Judge for Multilingual RAG with Uncertainty-Aware Scoring \- Preprints.org, https://www.preprints.org/manuscript/202603.1324

26. Translation Equivalence Is Not Linguistic Equity: English–Arabic–Chinese Calibration and Cultural-Item Performance Gaps in a Compact Multilingual Language Model \- Stout Journals, https://stoutjournals.org/index.php/SLLL/article/view/85

27. A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification \- arXiv, https://arxiv.org/html/2107.07511v6

28. Conformal Prediction Adaptive to Unknown Subpopulation Shifts \- arXiv, https://arxiv.org/html/2506.05583v2

29. \[2508.11345\] Conformal Prediction Meets Long-tail Classification \- arXiv, https://arxiv.org/abs/2508.11345

30. \[2507.06867\] Conformal Prediction for Long-Tailed Classification \- arXiv, https://arxiv.org/abs/2507.06867

31. CAPS: A Cross-Lingual Methodology for Detecting Misinformation in Estonian Health News | Request PDF \- ResearchGate, https://www.researchgate.net/publication/400791093\_CAPS\_A\_Cross-Lingual\_Methodology\_for\_Detecting\_Misinformation\_in\_Estonian\_Health\_News

32. Top 10 Multilingual Embedding Models for RAG \- AIMultiple, https://aimultiple.com/multilingual-embedding-models

33. Asymptotic Risk Calibration for Selective Question Answering \- arXiv, https://arxiv.org/html/2608.12008

34. \[2208.02814\] Conformal Risk Control \- arXiv, https://arxiv.org/abs/2208.02814