Executive Summary

Full report

On this page

The problem is essentially multilingual open-set concept classification: given a registry of governed ConceptCodes (each with reviewed synonyms/definitions in multiple languages) and an unseen query phrase, the system must either map it to the correct concept or abstain if the evidence is insufficient. The Embedded Semantics design cleanly separates concept identity (a stable, human-curated registry record) from model evidence (embedding-based retrieval). In practice, exact production resolution uses only reviewed expressions (returning unknown_expression on any novel phrasing), while embedding-based generalization is experimental.

Our analysis recommends a two-stage pipeline: (1) a multilingual retrieval stage (using a bi-encoder to embed the query and retrieve candidate concept prototypes) followed by (2) a thresholded assignment stage (optionally with cross-encoder reranking or entailment verification) that either selects the top concept or abstains. Critically, this must be treated as an open-set problem – new or ambiguous meanings must trigger abstention, not forced classification. We find that the state-of-art multilingual embeddings (e.g. LaBSE, LASER, XLM-R, mBERT-based models) provide good retrieval anchors, especially when trained on parallel or contrastive data. However, raw embeddings are not fully reliable as semantic truth: they serve to generate candidates, while concept identity ultimately depends on validated evidence.

To achieve “safe semantic generalization,” the system must be heavily calibrated and conservative. We review open-set recognition and selective classification techniques to establish rigorous abort/accept criteria. Metrics like recall@k, MRR, language-wise accuracy, and abstention precision/recall will measure candidate quality, while risk-coverage curves evaluate abstention behavior. We also propose benchmark splits that truly hold out unseen languages and paraphrases, and sample test scenarios for cross-script, ambiguous, and out-of-domain queries.

In summary, the strongest path is a retrieval-then-decision architecture using multilingual bi-encoder embeddings and concept prototypes, with explicit confidence thresholds (possibly calibrated per concept or language) to trigger abstention. We then outline prototype design (learned vs centroid), hard-negative mining for training, calibration strategies (e.g. temperature scaling or conformal methods), and even the selective use of LLMs. The report ends with concrete benchmarks, sample thresholds, prioritized experiments, and strict criteria (“promotion gates”) needed before any embedding-driven generalization is safe for production.

Interpretation of the Public Embedded Semantics Architecture

Embedded Semantics is built around a registry of multilingual ConceptCodes (stable identifiers) each with reviewed expressions, definitions, and provenance. A key design principle is: “Separate the probability from the meaning”. In other words, concept meaning lives in the static registry (human-vetted synonyms, definitions, metadata), whereas probabilities and embeddings are ephemeral model outputs. The registry is authoritative: even if the embedding model is replaced, the ConceptCode for a meaning should not change.

In current production, only exact matches to reviewed expressions are resolved. If a query exactly equals a curated expression (in any language), it maps deterministically to that ConceptCode. If it does not, the system “explicitly abstains on unseen phrases with unknown_expression”. (If a reviewed phrase legitimately belongs to multiple concepts, the resolver returns an ambiguous_expression marker.) In contrast, the experimental retrieval subsystem uses embedding models to handle arbitrary queries. This is clearly labeled “experimental”: embeddings may “provide retrieval evidence, but they never become the semantic source of truth”.

Put simply, the architecture is:

• Governing concept registry: Stable ConceptCodes with provened multilingual synonyms and definitions (human-curated).
• Production resolver: If query matches a reviewed expression, output the ConceptCode; otherwise output unknown_expression.
• Research lane (semantic retrieval): Embed the query, retrieve candidate concepts (via nearest-neighbor search over concept prototypes), then apply a calibrated decision (assign one concept or abstain). This lane is currently not production-active and is under intensive study.

This interpretation emphasizes that semantic identity is hand-crafted, while embeddings only supply supplementary signals. Our task is to make the retrieval & decision lane as robust as possible without altering the registry.

Formal Problem Definition

Let $\mathcal{C}$ be the set of known concepts (ConceptCodes) in the registry, each with a set of reviewed textual expressions (potentially in many languages) and possibly a definition. We treat this as a classification task with an additional “abstain/unknown” class. Formally, given an input expression $x$ (in any language), the goal is to compute a function $$ f(x) \;=\; \begin{cases} c \in \mathcal{C}, & \text{if } x \text{ expresses concept }c,\\ \text{abstain}, & \text{if it does not clearly match any known concept.} \end{cases} $$ The critical requirement is high precision on $c \neq \text{abstain}$ – falsely assigning a wrong concept is highly undesirable. Thus the system should prefer abstention (unknown) when uncertain.

This is an open-set, selective classification problem. During testing, $x$ may be an expression of a known concept (a “closed-set” case) or something entirely outside the registry (“unknown”). We can formally view it as follows:

• Known-known classes (KKCs): The registry concepts $\mathcal{C}$ we can assign.
• Unknown-unknown classes (UUCs): Unseen meanings not in $\mathcal{C}$, to be rejected.

This matches the open-set recognition scenario: at test time, queries may come from new classes not present in training. The classifier’s job is not only to correctly label KKCs, but also to detect and reject UUCs.

We further allow that the input $x$ may be in any language or script. Therefore, $x$ is a sequence of Unicode characters with a language tag, and the system must bridge language differences. We define a score function $s(c|x)$ for each known concept $c$, derived from model evidence (e.g. cosine similarity of embeddings). The decision rule will be: if $\max_{c\in\mathcal{C}} s(c|x)$ exceeds a threshold $\tau$, assign the $\arg\max$ concept; otherwise abstain. (In practice we may use more sophisticated decision logic, e.g. comparing the top two scores.)

Key formal points:

• Retrieval vs assignment: Retrieval yields a ranked set of candidate concepts $c_1,c_2,\dots$ with scores. The final assignment must consider calibration: we only pick $c_1$ if $s(c_1|x)$ is sufficiently higher than alternatives and above threshold.
• Open-set nature: The thresholding embodies the reject option. The system must maintain a low risk (error on KKCs) while allowing some coverage (making predictions on a subset of inputs).

Candidate Retrieval vs. Semantic Identity Assignment

In our pipeline, candidate retrieval is the step that uses vector embeddings to propose possible concepts for the query, whereas semantic identity assignment is the final decision of which concept (if any) to output. These serve different roles:

• Retrieval (Bi-encoder): We encode the query and each concept prototype into a shared embedding space (using a pretrained multilingual bi-encoder). We then perform a nearest-neighbor search (e.g. cosine similarity) to find the top-$K$ candidate concepts whose prototypes are closest to the query. This stage does not commit to a label; it merely gathers evidence. For example, retrieving concept prototypes allows us to consider synonyms in other languages or paraphrases, focusing on semantic closeness rather than exact string match.

• Identity Assignment (Decision/Reranking): Given the top-$K$ candidates, we refine the decision. This may involve a cross-encoder or entailment model that takes the query and each candidate’s definitions together, and produces a refined score. The system then compares the highest score(s) to calibrated thresholds. If the top candidate’s score is significantly above others and above a confidence cutoff, we output that concept code; otherwise we output unknown_expression (abstain) or possibly an ambiguous_expression if two concepts are equally good. In this stage, semantic identity is strictly governed by evidence, not by raw embedding closeness.

Importantly, embedding retrieval is only evidence – as Embedded Semantics states, “the registry is the semantic source of truth, embeddings [are] an experimental candidate-retrieval lane, never as semantic identity”. Therefore, while the retrieval stage suggests candidates, the final identity must be reconciled with the registry’s meaning. For example, one might verify that the query entails a concept’s definition (via NLI) before assignment. If the evidence is weak or conflicting (e.g. top-2 scores are close), the system should abstain or flag ambiguity. In summary, retrieval and identity are decoupled: we retrieve candidates broadly, then make a calibrated yes/no decision per concept.

Review of Current Multilingual Representation Approaches

Modern multilingual embeddings can align meaning across languages, but their objectives vary. Key approaches include:

• LASER (Artetxe & Schwenk, 2019): A BiLSTM encoder with shared BPE vocabulary, trained on parallel corpora for 93 languages. LASER supports 28 different scripts and achieves strong zero-shot transfer (XNLI, MLDoc) by treating sentence embedding alignment as the training goal. It set a high bar for supporting massive multilinguality, but suffers from older architectures.

• mUSE (Multilingual Universal Sentence Encoder): A variant of Google’s USE extended via multilingual training (e.g. Yang et al., 2019). mUSE has been a popular baseline for multilingual STS.

• XLM-R / mBERT: Multilingual BERT models provide contextual embeddings, often pooled to sentence vectors. Without fine-tuning, they cover many languages (XLM-R covers 100+) but require task-specific training to yield good sentence embeddings. They tend to be weaker for direct semantic retrieval out-of-the-box than specialized models.

• **LaBSE (Feng et al., 2022)**: A dual-encoder (Siamese) model using a pretrained multilingual BERT backbone plus a contrastive translation-ranking objective. LaBSE was trained on hundreds of millions of sentence pairs (translations) with an additive margin softmax. It achieved state-of-the-art cross-lingual retrieval: 83.7% accuracy on the Tatoeba parallel sentence retrieval over 112 languages, far above LASER’s 65.5%. This shows LaBSE excels at capturing exact equivalence across languages. (On monolingual fine-grained STS tasks, however, specialized English SBERT models still slightly outperform LaBSE.)

• Sentence-BERT (mBERT- or XLM-R-based): Extensions of SBERT exist for multilingual tasks. For instance, one can fine-tune multilingual BERT with a contrastive loss on parallel data. The HuggingFace sentence-transformers library provides models like stsb-xlm-r-multilingual, which are trained on semantic similarity data. These often underperform LaBSE on strict translation alignment but can be fine-tuned for a specific concept task.

• SimCSE / Contrastive Methods: While most SimCSE work is monolingual (English), a few recent papers explore multilingual SimCSE (e.g. using translation pairs as positives). These are promising for cross-lingual alignment, but not as thoroughly benchmarked as LaBSE.

• Knowledge Distillation and Lexicon-based: Some approaches align embeddings via dictionaries or pseudo-labels. For example, x2x distillation (“Making Monolingual Embeddings Multilingual by KD”) or the use of BabelNet to tie languages at word level.

Key Takeaways: The strongest general-purpose model for zero-shot multilingual retrieval is currently LaBSE (and the emerging ALIGN, MPNet, etc., though not yet in our citations). LASER remains competitive for many languages. For our use, we should evaluate multiple: LaBSE, an XLM-R/SBERT variant, and maybe a newer sparse-mixture-of-experts embedding (recent work [48] claims further gains). Ultimately, the choice should be empirically driven. Training with contrastive objectives (translation or paraphrase pairs, additive margin softmax) and hard negatives (next section) is common.

Review of Open-Set Recognition Approaches

The core challenge is recognizing when an input does not belong to any known class. This is the open-set recognition (OSR) problem: classifiers are trained on “known known” classes (KKCs) but must handle “unknown unknown” classes (UUCs) at test. In OSR, the model must “not only accurately classify the seen classes, but also effectively deal with unseen ones”.

Common OSR techniques include:

• Thresholding Confidence: The simplest idea is to use the model’s confidence (e.g. max softmax score) and reject inputs below a threshold. For deep nets, this often means using the softmax response (SR) – the highest softmax probability – and declaring unknown if SR is below τ. This is what Geifman & El-Yaniv (2017) call Softmax Response; they show one can guarantee a user-specified error rate by selecting τ appropriately. However, raw softmax is notoriously overconfident, so calibration is needed (see next section).

• OpenMax (Bendale & Boult 2016): A classical method replaces the softmax layer with an “OpenMax” layer. It fits an extreme-value (Weibull) distribution to distances between training examples and class centroids, to assign some probability mass to an “unknown” output. In effect, it learns a decision boundary in feature space for each class. OpenMax was shown effective on vision tasks, though it requires tuning and isn’t widely used in NLP yet.

• Distance-based / Prototype Models: Models like Nearest Class Mean (NCM) naturally lend themselves to OSR by examining distance to the nearest centroid. Extended methods (e.g. Non-Outlier NNO) incorporate an explicit reject threshold: if the query is too far from all class centroids, classify as unknown. In the vector embedding space, one can similarly flag examples whose nearest-neighbor distance exceeds a learned radius. For example, one might compute the mean embedding (centroid) $\mu_c$ for each concept; at test, compute $d=\min_c \| e(x)-\mu_c\|$. If $d$ is above a threshold, reject.

• One-vs-Rest / 1-vs-Set Classifiers: Instead of a single softmax, use sigmoid outputs per class (as in a one-vs-all scheme). Each class learns an “acceptance region”; a sample can be rejected if all sigmoids are below threshold. This is equivalent to training an OpenMax-like boundary.

• Reconstruction-based (e.g. autoencoders): Some methods (e.g. CROSR) train the network to both classify and reconstruct. If reconstruction error is high, the input may be out-of-distribution. This is less common in NLP and complex to implement.

• Conformal Prediction: A more recent paradigm offers provable coverage guarantees. By holding out a calibration set, one can adjust scores so that the probability of assigning the wrong label is bounded (see Selective Classification and Conformal Prediction literature). In essence, we set thresholds so that, empirically on held-out data, the error is below a target.

In summary, open-set solutions typically build explicit rejection rules on top of a base classifier. For our task, promising strategies include thresholding embedding-based confidences or entailment scores, calibrating them, and if necessary learning a class-conditioned accept threshold. The system should reserve “open space” between concepts, ensuring areas where $\max_c s(c|x)$ is low correspond to “unknown”. Detecting hard false merges or overlapped concepts (false semantic merges) may require manual review or more sophisticated modeling (discussed below).

Review of Calibration Methods

Calibration ensures that a model’s confidence scores correspond to actual correctness probabilities. In deep networks, the raw scores (softmax probabilities or embedding similarities) are typically miscalibrated: modern larger models are often overconfident. A poorly calibrated confidence can make thresholds meaningless.

Key calibration techniques:

• Temperature Scaling: A simple post-hoc method (Guo et al., 2017) that divides all logits by a learned scalar $T>0$ before softmax. On a validation set, one finds $T$ that minimizes negative log likelihood, effectively “softening” confidences. Pleiss et al. note this can “almost perfectly restore network calibration” without affecting accuracy. We can apply temperature scaling to the softmax output of a cross-encoder or to the cosine similarity scores from embeddings.

• Platt Scaling / Isotonic Regression: For binary confidence (one-vs-rest), logistic regression (Platt) or non-parametric isotonic regression can calibrate probabilities. These require a separate calibration set.

• Vector or Dirichlet Scaling: For multi-class, one can generalize temperature to a diagonal (or full) matrix (vector scaling) or use Dirichlet calibration (Kull et al., 2019).

• Conformal Prediction: Conformal methods provide distribution-free guarantees. In classification, one can compute p-values for each candidate class using a held-out calibration set, then set thresholds so that, with high probability, the true label is included in the prediction set. For our selective task, we could use conformal risk-controlling prediction to determine the confidence threshold that yields a desired maximum error (mirroring Geifman’s risk guarantee).

• Selective Classification Calibration: Geifman & El-Yaniv (2017) specifically calibrate the decision threshold to achieve a user-set risk level. They show that by thresholding the softmax (SR) you can (with high probability) guarantee that the error of the accepted predictions is below a target. This could be applied concept-by-concept or globally.

• Label Smoothing: The recent multilingual LLM calibration study by Huang et al. (2026) found that instruction-tuning on high-resource languages can increase model confidence on low-resource languages without accuracy gains, thus miscalibrating the model. They observe that label smoothing during fine-tuning helps keep confidence lower and more uniform across languages. This suggests smoothing or regularization may help calibration for languages with scarce data.

• Calibration Metrics: In evaluation, we should measure expected calibration error (ECE) or reliability diagrams for confidence vs. accuracy. The Survey on OOD in NLP emphasizes the importance of well-calibrated models for safety. We should also evaluate calibration per language, since confidence may drift between languages (see also [43] on multilingual calibration effects).

In practice, we will hold out a calibration set (distinct from both training and final test) to tune threshold(s). This set should reflect the variety of languages and paraphrases. We might learn a global temperature $T$ on all languages (if calibration is consistent) or separate temperatures per language or even per concept if needed. We must prevent leakage: the calibration set must contain no examples from the final test queries and no overlapping expressions with train data.

Review of Abstention/Selective-Prediction Methods

Abstention (the reject option) is fundamentally the goal in this system: prefer “I don’t know” over a wrong concept. This aligns with the selective prediction framework. The literature emphasizes two components: a classifier and a confidence scoring function (CSF). For us, the classifier proposes a concept, and the CSF (e.g. max-similarity or entailment score) measures confidence.

We draw on:

• Geifman & El-Yaniv (2017, 2019): As noted, they formalize selective classification by thresholding a confidence measure to achieve guaranteed error control. The key idea is setting a threshold on the softmax response (SR). This yields an operating curve of coverage vs. risk. We will similarly define a threshold so that only queries with $s(c_1|x)\ge\tau$ (or margin to runner-up ≥ δ) are accepted.

• Chow’s Rule: The classic reject rule (Chow 1957) is to reject any classification below a threshold on maximum class probability. This is the same intuition.

• Deep Open Classifiers: For neural nets, some works (Shu et al., 2017) replace softmax with a set of sigmoid outputs and model the “open space” explicitly. However, a simpler approach is viable given our strong embeddings: we can rely on calibrated scores.

• Adaptive Thresholding: We should consider if a single global threshold is sufficient, or if we need class-conditional thresholds. Some recent works (Nakhaeizadeh & Khandani, 2022) explore per-class thresholds for imbalanced data. In our domain, different concepts may have different ambiguity. One idea is to calibrate a threshold $\tau_c$ for each concept $c$, based on its validation score distribution. For example, if Concept A is very tight (low variance), we might require a higher confidence, whereas a broad concept might allow lower. This would require concept-level calibration data.

• Multilingual Specifics: We might also consider language-specific thresholds. If a model is systematically more confident on, say, English than Turkish, a shared threshold could be unfair. The multilingual calibration study suggests model confidence can vary dramatically by language after tuning. Thus one could either calibrate per language or normalize scores (e.g. z-score per language) before thresholding.

Takeaway: Abstention in our system will be achieved by comparing confidence scores (from retrieval or reranking) to thresholds. We will likely provide coverage-risk curves or AUROC-like metrics (see [57]) during evaluation. The selective classification literature suggests we should not pick arbitrary fixed coverage; rather, we examine performance across thresholds or use area-under-curve (risk-coverage).

Prototype Representation Strategies

A central question is how to represent each concept as one or more vectors in the embedding space. Several strategies:

• Single Centroid (Mean) Vector: Take all reviewed expressions of a concept (across languages), encode them with the bi-encoder, and average to get a class centroid. This is simple and interpretable: at test, we compare the query embedding to each concept centroid. However, a single centroid may be too coarse if the concept covers diverse expressions (e.g. “bank” in different contexts).

• Weighted or Language-Balanced Centroid: If the concept has many more expressions in one language than others, a raw mean will be biased. One could weight expressions inversely by language frequency or ensure each language contributes equally. For instance, compute the centroid of per-language centroids (averaged from each language) to avoid large languages dominating.

• Multi-Prototype: If a concept is known to have sub-groups or multiple senses (even within the same concept code), we can allow multiple prototype vectors. For example, if “apple (fruit)” has very different phrasing in Latin-script vs. logographic languages, it may occupy a curved manifold; representing it by two centroids (one for each cluster) could improve recall. This could be done by clustering the reviewed expressions’ embeddings into k groups and using each cluster center as a prototype.

• Learnable Prototypes: Instead of a fixed mean, one can learn a vector (or distribution) for each class as part of model training. The EmergentMind overview notes that “learnable prototypes” can capture intra-class diversity. For example, in a metric learning setup (Prototypical Networks), one can treat the prototype as a parameter to be optimized. This might require fine-tuning the bi-encoder on concept distinction tasks.

• Stochastic/Distributional Prototypes: A more advanced idea is to model each concept’s embedding as a Gaussian (mean + covariance). Dhamija et al.’s "Stochastic Prototype Embeddings" treat both embeddings and prototypes as Gaussians, and compute class membership by marginalizing over uncertainty. Practically, we could track the variance of embeddings for each concept; if a query lies well outside the learned covariance ellipse of all classes, we reject. This naturally ties into open-set detection by modeling density.

• Embedding Interpolation: If the registry includes formal definitions or glosses, we might incorporate those into the prototype. For instance, use the embedding of the definition (in each language) as an additional anchor.

Recommendation: A good baseline is the simple centroid (mean of reviewed expressions) per concept. We should ensure language balance when computing this mean. If performance is poor, we can experiment with multi-prototype (e.g. one per language or via clustering) and with learned prototypes. Using distributional prototypes (variance) is attractive for outlier detection, but more complex. It could be implemented by measuring the Mahalanobis distance to class centroid (with learned covariance) and using that for thresholding.

Hard-Negative Mining Strategy

Hard negatives are crucial for training an embedding model that discriminates fine-grained semantics. A hard negative is an example that is close in meaning (and thus in embedding space) to the positive class but actually belongs to a different concept. Including such negatives in training prevents the model from lumping distinct concepts together.

Embedded Semantics explicitly uses hard negatives: “Hard negatives are closely related but incorrect concepts used to test whether retrieval can distinguish semantic neighbors rather than merely grouping content from the same topic”. For example, for the concept “car (vehicle)”, a hard negative might be “bus” or “truck” – topically similar but semantically distinct.

In practice, one can mine hard negatives by:

• In-Batch Nearest: During training on contrastive or triplet loss, treat other examples in the same batch as negatives. The hardest in-batch negative for a given query is the other concept embedding with highest similarity.

• Info Retrieval Mining: Run the current model (or a baseline like BM25) on a large corpus of concept phrases. For each query instance, retrieve the nearest neighbors across the dataset; take the top few that are not true positives as negatives. This often yields very difficult negatives (the ones the model currently confuses).

• Dynamic Hard-Negatives: Iteratively, one can train the model, then use it to find new hard negatives (the highest-scoring wrong concepts), add them to the training data, and repeat. NV-Retriever [32] discusses such mining, though it notes the danger of false negatives.

• Avoiding False Negatives: A key caveat is not to label an actual positive as a negative by mistake. Studies have found that up to 70% of “top retrieved” might actually be unlabeled positives. A remedy is to filter hard negatives by a stronger model (e.g. a cross-encoder or even a human check) to avoid corrupting training. For our research, we should curate hard negatives carefully, possibly using the strong production registry to verify if a candidate really is a different concept.

Including hard negatives in training will push the embedding model to spread out similar concepts and tighten clusters around true synonyms. This is critical in a fine-grained registry: we don’t want “dog” and “wolf” or “Python (language)” and “Python (snake)” to have overlapping embeddings.

Multilingual Evaluation Methodology

To evaluate the system comprehensively, we need multilingual, multi-task benchmarks:

• Recall-based Retrieval Metrics: Since retrieval of the correct concept is a first goal, compute Recall@K and MRR. For each test query (with known ground-truth concept or none), check if the true concept is among the top-$K$ retrieved candidates. Report Recall@1, Recall@5, and Mean Reciprocal Rank. Higher is better.

• Cross-Language Agreement: For concepts expressed in multiple languages, we can test consistency. For example, take a query in language A and translate it to B; ideally both map to the same concept. We can measure the rate of agreement. The EmbeddedSemantics methodology explicitly mentions cross-language concept agreement as a metric.

• False-Neighbor (False-Positive) Rate: If a query’s true concept is not in the top-$K$, but a semantically similar but wrong concept appears high, this is a false semantic merge. The provided metrics include a false-neighbor rate. We can compute: for each query, is the top retrieved concept actually wrong (i.e. a false positive)? We then compute the fraction of queries with such errors. This is like 1 – (precision at 1) for known queries.

• Prototype Spread / Cluster Quality: For each concept, we can measure the variance of embeddings of its reviewed expressions (centroid spread), and possibly the separation between concepts. A heuristic: compute the average distance between the concept centroid and its closest different-concept centroid (topological margin).

• Score Margin: The difference between the top score and second score is informative. A small margin means ambiguity. We can track the distribution of top-1 vs top-2 score differences as a metric. Higher margins are safer.

• Selective Classification Metrics: Define risk = error rate on queries the system does not abstain on, and coverage = fraction of queries answered. Plot Risk vs. Coverage (the selective risk curve) or compute area under this curve (analogous to AUC). We want low risk with reasonable coverage. Another standard is the AUGRC (area under generalized RC) metric which averages risk over coverages. Additionally, compute abstention precision (of those we abstained, how many truly were unknown) and abstention recall (of truly unknowns, what fraction we abstained on).

• Open-Set (OOD) Metrics: For held-out unknown queries, use OOD detection metrics: e.g. AUROC of distinguishing known vs unknown by confidence score, FPR@TPR=0.95, AUPR, etc. The NLP OOD survey recommends these standard metrics.

• Multilingual/Linguistic Parity: Compute all the above metrics per language or language family. We should report, for example, Recall@1 on English vs Chinese vs Arabic, and measure disparity (max–min difference). A gap indicates bias or data imbalance. Also compare script families: tests might reveal that Latin-script queries get higher accuracy than e.g. Thai or Amharic, highlighting uneven coverage.

• Error Analysis Metrics: To understand mistakes, annotate some failure cases by type (mistranslation, ambiguous phrase, etc.) and report their frequencies.

In practice, we will create a test corpus of queries covering (a) multiple languages (ideally including low-resource and distinct scripts), (b) concept synonyms not seen in training, (c) ambiguous queries, and (d) OOD sentences. We will then compute the above metrics on this benchmark. The bibliography (questions 11–14) suggests separate sections for cross-script, ambiguity, out-of-domain tests (below).

Cross-Script Evaluation

Different writing systems can pose unique challenges. We should ensure languages from diverse scripts are included (e.g. Latin, Cyrillic, Arabic, Devanagari, Chinese characters, etc.). LASER’s success was partly due to covering 28 scripts, but not all scripts have equal data or model performance. For cross-script evaluation:

• Test Queries in Various Scripts: Collect or generate queries in as many scripts as feasible. For example, if a concept is primarily annotated in Latin-script languages, test its equivalents in Cyrillic or logographic scripts to see if the embedding model truly aligns them.

• Script-Specific Metrics: Report metrics separately for each script or script group. For instance, Recall@1 on Latin, Recall@1 on non-Latin, etc. If certain scripts lag, it may indicate the embedding model is not well aligned for them.

• Script-specific Calibration: Compare the distribution of similarity scores by script. We may need different thresholds for different scripts if, say, Chinese embeddings systematically have lower cosine similarity with their concept prototypes than English ones.

• Addressing Gaps: If performance is poor on some scripts, consider script-specific interventions: e.g. add script-identified prototypes (if concept has entries in that script, make sure they exist), or use transliteration as a bridge. The benchmark will reveal whether such steps are needed.

Ambiguity Evaluation

Some queries are inherently ambiguous (map to multiple concepts). The system should detect this. To evaluate:

• Ambiguous Query Set: Create a set of ambiguous expressions. For example, words like “bank”, “bat”, “plan” which have multiple senses. Ensure each has at least two plausible ConceptCodes in the registry.

• Response Check: For each ambiguous query, check if the system (a) abstains, (b) returns an ambiguous_expression indicator (like in exact match), or (c) returns one concept incorrectly. Ideally, the system should either abort or list multiple candidates. We might adapt the output format to allow multiple concepts when the top scores are within a small margin.

• Score Gap Analysis: Compute the distribution of top-1 vs top-2 scores on ambiguous queries. In ambiguous cases, this gap should be small. We can set an “ambiguity threshold”: if gap < ε, mark as ambiguous. We should tune ε on held-out ambiguous examples.

• Human Labeling: For some ambiguous queries, have human judges determine which (if any) concept the query better matches. Use this to validate if the system’s ambiguity flagging is correct (precision/recall of ambiguity detection).

The goal is that ambiguous cases are not forced into a single concept. The FAQ indicates the current exact-match resolver labels these as ambiguous_expression. We must extend this logic: if the evidence for two (or more) concepts is comparable, we should similarly avoid a single assignment and instead output multiple possibilities or abstention. In the final report, we’ll include examples where the model correctly identifies ambiguity vs where it fails.

Out-of-Domain Evaluation

Out-of-domain (OOD) refers to inputs that belong to no known concept. For OOD testing:

• OOD Query Set: Compile queries unrelated to any concept in $\mathcal{C}$. These can be random sentences on novel topics, nonsensical phrases, or from domains entirely outside the registry. For example, if our concepts are technical terms, use literary or colloquial sentences; or include mix-language code-switched gibberish.

• Desired Behavior: The system should abstain on most or all OOD queries. We measure OOD detection metrics: fraction of OOD queries flagged as unknown (abstention recall), and fraction of ID queries correctly not flagged (abstention precision). Equivalently, treat it as a binary classification (ID vs OOD) and compute AUROC, AU-PR, FPR@95% TPR, etc.

• Related vs Novel OOD: To stress-test, include two types of OOD:

1. Semantic shift OOD: phrases that look superficially related but are actually new concepts (e.g. if registry covers animals, test with machines).
2. Completely unrelated (non-semantic) OOD: gibberish or queries in an unrelated domain (e.g. math formula if no math in concepts).

We must report the system’s error (misclassification rate) on ID vs OOD separately. The OOD survey emphasizes that a model should not predict any known class for truly out-of-distribution samples. Our target is near-100% abstention on OOD, subject to coverage trade-offs on ID.

Proposed Benchmark Architecture

We propose a pipeline benchmark that mimics the intended retrieval/classification flow:

1. Training Phase: Train one or more multilingual bi-encoder models (see experiments in Section 19). Use the registry’s reviewed expressions as training data: positives are equivalent multilingual phrases for the same concept, negatives include random other concepts plus curated hard negatives (Section 10).

1. Prototype Construction: From the training data, compute each ConceptCode’s prototype(s) (e.g. averaged embeddings). If using learned prototypes, they are part of model.

1. Candidate Retrieval (Test): For each test query (held-out expressions), compute its embedding and retrieve top-$K$ concept prototypes via nearest-neighbor search.

1. Re-Ranking (Optional): Apply a cross-encoder or entailment model: input (query, definition of candidate concept) into a BERT-based binary classifier or scoring model. Re-compute a refined similarity/confidence score for the top candidates.

1. Threshold Decision: Compute a confidence score for the top candidate (e.g. sigmoid of cross-encoder, or softmax difference). Compare to threshold(s) $\tau$. If above threshold and sufficiently above runner-up, accept; otherwise output unknown or multiple labels.

1. Abstention Handling: If the system decides to abstain or ambiguous, record that as “no concept” or “multiple”.

1. Evaluation: Compute the metrics described above on the test set.

This architecture should be modular so that we can swap the embedding model, add a reranker, or change threshold rules. It reflects the distinction the site makes between governed identity (the prototype/threshold logic) and probability evidence.

Proposed Train/Calibration/Test Separation

A rigorous data split is crucial to avoid leakage. We propose:

• Training Set: Use a subset of the reviewed expressions for each concept. For example, if concept C has 5 known expressions (across languages), we might use 3 for training.

• Calibration (Validation) Set: Reserve another subset (e.g. 1 expression per concept) for tuning hyperparameters and thresholds. This set is not used to train embeddings or prototypes. It is used to calibrate any temperature scaling, set the confidence threshold $\tau$, and perhaps tune per-concept thresholds. It should cover all languages present in the training set.

• Test Set: The remaining expressions for each concept are held completely out. Critically, we may want to simulate new languages by holding out all expressions of certain languages for a concept. For example, if a concept has English, Spanish, German, hold out German as unseen in both training and calibration; then test the model’s zero-shot German retrieval. Alternatively, do k-fold rotation: train on $k-1$ languages, test on the $k$th.

• Out-of-Concept (OOD) Test Set: In addition, have a set of expressions that correspond to no concept (e.g. random sentences). These are only used for final evaluation of abstention; they should not appear at all during train/calibration.

• Disjoint Vocabulary Check: Ensure that no exact test phrase appears in training (by design of held-out sets). Also, if using phrase-level contrastive learning, avoid leakage of near-identical phrases.

• Hard Negative Independence: If using mined hard negatives from large corpora, ensure that any example used for calibration/testing was not used to mine negatives that were then included in training.

In summary, this is a three-way split: training (for embeddings/prototypes), calibration (for thresholds), and test (for final metrics). We should possibly use cross-validation over concepts or languages to maximize data usage, but always keep the folds strict.

Recommended Metrics

Based on the above discussions, key metrics include:

• Retrieval Metrics:
• Recall@1 / Recall@5: Fraction of queries whose true concept appears in the top 1 (or 5) retrieved candidates. Higher is better.
• Mean Reciprocal Rank (MRR): The average of $1/\text{rank of true concept}$. Rewards putting the correct concept at top.

• Ranking Quality:
• Top-1 vs Top-2 Score Margin: The average and distribution of $s_1 - s_2$, where $s_1$ and $s_2$ are the top two scores for a query. Larger margins indicate clearer decisions. We can report e.g. % of queries with margin > 0.1, 0.2, etc. This was proposed in EmbeddedSemantics metrics.

• Cross-Language Agreement: Compute Recall@1 (or accuracy) per language, then report the worst-case and average. Also compute some disparity measure (e.g. standard deviation of recalls across languages). If concept definitions exist in all languages, we expect high agreement; if not, gaps indicate bias.

• Open-Set/Abstention Metrics:
• Selective Risk vs Coverage: For each threshold $\tau$, calculate risk (error on answers made) and coverage (fraction answered). Plot the Risk-Coverage curve. Also compute Area Under this Curve (AURC) or the proposed AUGRC metric, which summarizes overall performance across thresholds.
• Coverage at Fixed Risk: e.g. what is the coverage when we require risk ≤ 1%? This analogously to “prec@k” with k such that we accept 99% confidence.
• Abstention Precision and Recall: As in the site’s methodology, compute the precision (of abstentions: what fraction were truly unknown) and recall (of unknowns: what fraction were abstained). This assesses how well the system avoids false rejections/acceptances.

• Out-of-Domain Detection: Treat identification of unknowns as a binary decision. Report metrics like:
• FPR@95% TPR: False positive rate (accepting a concept for OOD) when 95% of in-distribution (ID) are correctly accepted.
• AUROC / AUPR: For separating ID vs OOD by confidence score.

• Calibration Metrics:
• Expected Calibration Error (ECE): The difference between predicted confidence and actual accuracy (can bin by score) for in-distribution queries.
• Calibration per Language: Compute ECE for each language.

• Concept Coverage: How many concepts have sufficient support? For each concept, count the number of reviewed phrases / languages. Report distribution of support. This is not an accuracy metric, but a data quality check.

All these metrics should be reported with citations to their definitions: for example, EmbeddedSemantics suggests Recall@k, MRR, cross-language agreement, false-neighbor, score margins, abstention P/R. Selective classification literature emphasizes risk/coverage. We will use tables and charts to present these.

Below is an example of how we might tabulate system performance (hypothetical numbers):

Table: Sample metrics for various model configurations. Thresholds in bottom row are hypothetical pass criteria (recommendations, not observed results).

Proposed Promotion Gates

Before allowing embedding-based resolution into production, we recommend strict gating thresholds. For example, one might require (recommendation):

• Recall@1 ≥ 0.90 (i.e. at least 90% of held-out in-domain queries retrieve the correct concept first).
• Recall@5 ≥ 0.98 (almost all have the true concept in the top 5).
• Mean Reciprocal Rank ≥ 0.92.
• Top1–Top2 Score Margin ≥ 0.10 on average (to ensure clear winners).
• Cross-Language Agreement ≥ 0.90 (agreement between languages above 90%).
• False-Neighbor Rate ≤ 0.05 (less than 5% of queries are mis-assigned).
• Abstention Precision ≥ 0.95 (when system abstains, at least 95% of those cases truly had no appropriate concept).
• Coverage ≥ 90% at Risk ≤ 1% (i.e., answering 90% of queries with ≤1% error).
• OOD AUROC ≥ 0.99 (almost perfect separation of known vs unknown).
• Calibration ECE ≤ 0.05 (well-calibrated confidences).

These thresholds are illustrative. In practice, we would set them based on risk tolerance and development results. The bottom row of the sample table above labels them “hypothetical pass criteria.” We would only “promote” the model to production if it comfortably exceeds these bars on an extensive evaluation. In summary, gates combine retrieval quality (Recall, MRR), discriminability (margin, false positives), and safety (abstention precision, calibration).

Recommended Model Experiments (Ranked by Expected Information Gain)

To explore the best design, we suggest the following experiments in order of priority:

1. Embedding Model Comparison: Train and evaluate multiple multilingual bi-encoder models on the registry data: e.g. (a) LaBSE (frozen or fine-tuned), (b) XLM-R based SBERT (fine-tuned on our data), (c) mUSE, (d) any recent dense retriever (e.g. (placeholder) “Ultra”?). Measure Recall@k and calibration on held-out languages. This will tell us the baseline representational power. (Citations: LaBSE results, LASER coverage.)

1. Hard-Negative Impact: Starting from the best model above, add hard negative mining to training. For example, after an initial model run, mine top-5 nearest neighbors of each positive and add them as negatives. Then retrain or fine-tune. Compare performance against using only random negatives. This is known to greatly improve discrimination. Evaluate the drop in false-positive rate and the change in margin/distribution.

1. Prototype Construction Variants: Test different prototype strategies: single centroid vs language-weighted centroid vs multi-prototype. For multi-prototype, perhaps use k-means on each concept’s embeddings or assign separate prototypes for each language. Evaluate whether multi-prototype improves Recall@1 (especially if concept has heterogeneous phrases).

1. Cross-Encoder Reranking: Introduce a reranker. Take the top-K candidates from the bi-encoder (say K=10) and use a cross-encoder model (e.g. a multilingual BERT) to score (query, concept definition) pairs. Compare results with/without reranker. This will gauge whether the additional cost is justified. The sentence-transformers guide suggests this approach is often beneficial.

1. Threshold Calibration: Experiment with different thresholding schemes: (a) a single global threshold on score, (b) separate thresholds per concept or per language. Use the calibration set to set these. Evaluate which approach yields better risk-coverage tradeoff and balanced language performance.

1. Generative LLM Assistance: As an exploratory step, evaluate if a large LLM (e.g. GPT) can aid decision: e.g. use it in zero-shot mode to rank candidates or verify meaning (“Does ‘X’ mean concept C?”). Compare its outputs’ accuracy and confidence to the bi-encoder baseline. This will inform whether an LLM adds value or spurious errors (see next section).

1. Ablations: Test the effect of removing multilingual evidence. For example, simulate “monolingual mode” by restricting training and retrieval to one language at a time, then measuring drop. This reveals how much cross-lingual training helps.

Each experiment should measure our suite of metrics (Sec. 17) and report by language and concept. Together, they inform which model and strategy yields the most robust semantic generalization.

Reranker Decision Criteria

A reranker (cross-encoder or other joint model) is justified when the bi-encoder’s top results still need disambiguation. The Sentence-BERT guide notes that cross-encoders achieve higher scoring accuracy but are slower; a typical solution is bi-encoder retrieve → cross-encoder re-rank top-N. In our case:

• Use the reranker if K (bi-encoder candidates) is not too large (e.g. 5–10). If bi-encoder performance is already very high at top-1, a reranker may add little. But if there are many close candidates (small margins), the reranker can leverage full-text interaction (query+definition) to improve the top ranking.

• Criteria: We can learn from metrics. If for many queries the correct concept is ranked 2nd by bi-encoder (recall@1 low but recall@5 high), reranking could boost it. A rule of thumb: if bi-encoder Recall@1 is significantly below Recall@5, a reranker might recover those cases. We should explicitly test (Exp. 4 above) to quantify gain.

• If using an LLM for reranking, we need to ensure it’s grounded (e.g. using the candidate definitions or contexts to avoid hallucination).

Finally, the reranker’s output should feed into the same calibration mechanism: even after reranking, we apply the confidence threshold rule. The cross-encoder score might directly serve as $s(c|x)$ for thresholding.

LLM-Based Adjudication Analysis

Generative LLMs (e.g. GPT-4) can be considered as adjudicators or augmenters for concept resolution, but they carry risks. Potential uses include:

• Paraphrase Generation / Data Augmentation: Use an LLM to generate additional synonyms or translations of each concept to enrich prototypes. This is relatively low-risk if we manually vet them, but blindly trusting GPT’s outputs can introduce errors.

• Zero-Shot Reranking: One could prompt an LLM with a query and a candidate concept description, and ask if they match. For example: “Given the phrase X, which concept among [list] best matches?”. In principle, GPT might know many languages and concepts.

However, LLMs are known to hallucinate plausible but false information. That undermines trust: if an LLM incorrectly “hallucinates” that a query matches a concept, it can cause a silent error. As Alansari & Luqman (2025) note, hallucinations “undermine the reliability and trustworthiness” of LLM outputs. In a high-stakes registry, this is worrisome. We should use LLM outputs with skepticism:

• Any LLM judgment should be treated as low-confidence evidence or for human-in-the-loop review, not as a hard decision rule.
• For example, if the bi-encoder has an ambiguous case, an LLM could generate a justification or paraphrase, but we should not blindly accept it. We might limit LLM use to detection of possible errors (e.g. flag “concept not found”).
• If privacy or cost is a concern, relying on LLM APIs in production may be impractical.

Conclusion: An LLM might assist in generating additional training data (paraphrases or definitions) or as an advisory reranker. But it should not be the final arbiter of meaning without verifiable evidence. Especially, an LLM that has seen no registerions can’t guarantee to honor the “stable registry” requirement; it might invent new relations not in the registry. Therefore, we caution that integrating LLMs for final decisions could decrease trust, unless carefully controlled.

Major Scientific Risks

Key risks and pitfalls include:

• False Semantic Merge: Two distinct concepts might become indistinguishable by the embedding model. If concept prototypes overlap, an unseen query may be wrongly assigned. The system should detect this risk (see Section 10). The site itself warns against “silently treating [distinct meanings] as equivalent”. We mitigate this by hard negatives and by preserving ambiguity flags, but it remains a risk if, for example, two sense-distinct concepts share nearly identical definitions.

• Insufficient Language Coverage: If some languages or scripts have sparse training data, the model may be poorly calibrated for them. The Huang et al. study shows confidence can become miscalibrated for low-resource languages. An unseen language phrase might thus be accepted or rejected at the wrong rate. We risk geographic or linguistic bias: e.g. if Arabic phrases tend to get lower similarity scores than English equivalents, a global threshold might over-reject Arabic queries. We must monitor language parity metrics to catch this.

• Over-reliance on Embeddings: Embeddings capture general semantics but may not encode subtle registry-specific distinctions (e.g. cultural connotations, pragmatic usage). The model could systematically rank one concept above another for reasons outside the intended definition. For instance, bias in pretraining can cause certain culture-specific terms to align poorly.

• Data Leakage and Overfitting: If the calibration or test sets accidentally overlap training data (or synonyms of it), performance estimates will be overly optimistic. Rigor in data splitting is crucial.

• Threshold Leakage / Tuning on Test: If we use the same held-out set to both tune thresholds and report metrics, we leak information. We must have a separate calibration set and final test.

• Adversarial or Spurious Queries: Certain queries might “game” the embedding model (e.g. adversarial examples or idioms). The system could be tricked into high confidence on nonsense. This risk is common to all ML, but with abstention and OOD checks we can partly guard against it.

• Model Drift and Concept Evolution: Since concepts are stable, but language evolves, new slang or usage might not fit. The system might gradually become outdated unless retrained with new synonyms. Without updates, coverage decreases over time.

• LLM Hallucinations: If we integrate generative models for augmentation or reranking, we risk their factual errors. As discussed, this could reduce trust.

Overall, the scientific risk is building a system that seems accurate on benchmarks but fails safely on novel or skewed real-world inputs. That is why we emphasize robust evaluation (Sec. 11-14) and strict abstention thresholds.

Failure Examples

Below are illustrative failure cases the system might encounter:

• Ambiguous Phrase Misassignment: Query: “Tōkyō” (in Japanese script) meaning the capital city. Suppose we have concepts “Tokyo (city)” and “Tokyo Tower (landmark)”. The query is ambiguous (could refer to the city or the tower). If the model confidently chooses “Tokyo Tower” (maybe because context clues favored that concept), it fails. The system should ideally abstain or flag ambiguity.

• Close Concepts Confusion: Concepts: C1 = “USA (country)”, C2 = “United States Dollar (currency)”. Query: “US”. The bi-encoder may find both concepts similar. If the system assigns one (say country) with moderate confidence, it is wrong for currency contexts. This is a false positive (confusing neighbors).

• Low-Resource Language: Concept: “Car”. Reviewed phrases only exist in English and Spanish. Query: “voiture” (French for car). The embedding model may map “voiture” near “car”, but with lower score. If the threshold is too high, it abstains (false negative). If too low, it might wrongly assign “car” to a different concept if that score was slightly higher. Alternatively, if a French phrase like “bateau” (boat) appears, the model might incorrectly map it to “car” due to surface similarity (“bat”).

• Script Mismatch: Concept: “Steganography (art of hidden writing)”. Reviewed in Latin script. Query: its Japanese translation in Kanji. If the model’s embedding poorly aligns Japanese with English for this niche term, it may fail to retrieve it, causing abstention.

• Out-of-domain: Query: “Quantum entanglement” but registry only covers horticulture. The system should say unknown. A failure is if it maps to some concept like “plant fusion” because it found “entanglement” “fusion” keywords. The correct action is abstain.

• Misleading Similarity: Concept: “Jaguar (animal)” and “Jaguar (automobile)”. Query: “eats deer”. The embedding might pick up “jaguar” + “eats deer” and lean toward the animal concept. If we intended the car concept, this is a failure. Without disambiguating context, the system shouldn’t guess. The safe failure mode is to abstain.

Each of these examples underscores why we need a conservative threshold and ambiguity detection: an incorrect concept is worse than “don’t know.” They also highlight evaluating borderline cases (Sec. 11–14).

Recommended Next Three Experiments

Based on the above, our immediate next steps should be:

1. Evaluate Threshold Strategies: Using the chosen embedding model, systematically test global vs. concept-specific vs. language-specific confidence thresholds. For each scheme, measure risk/coverage and per-language accuracy. This will directly inform how we calibrate cross-lingual parity and whether per-concept thresholds are needed (Question 5–7).

1. Reranker Utility Test: Integrate a cross-encoder reranker for the top-5 candidates and compare performance. This quantifies if reranking significantly improves top-1 precision or disambiguation (Question 15). If effective, it may become a permanent part of the architecture.

1. LLM Augmentation Probe: Conduct a controlled experiment where an LLM generates paraphrases for a subset of concepts, expanding their prototypes, or where an LLM re-scores a few ambiguous cases. Check if this improves recall without degrading precision. Also test directly if using an LLM to answer “Which concept?” yields consistent results. This will gauge LLM benefit vs. hallucination risk (Questions 16–17).

Each experiment should be run on the multilingual evaluation bench with calibration. The outcomes will guide further design (e.g. whether to prune LLM components or focus on thresholds).

Exact Evidence Required Before Production Activation

For a concept to be considered “production-ready” (i.e., eligible for embedding-driven resolution), it should have robust supporting evidence. Based on the design, we recommend requiring at least:

• Multilingual Reviewed Expressions: At least two independent reviews (e.g. one per language) equating to that concept. Ideally, those reviews should come from different language families. For example, if a concept only has English synonyms, we would hold off adding it to the registry until a second language (or a definition) is provided. The site emphasizes that expressions attach with “locale, review status, and equivalence”, so each concept should have verified equivalents.

• Authoritative Definition or Context: A formal definition or description of the concept (in any language) from a trusted source (dictionary, technical spec, etc.). This can help later if we use entailment checks.

• Prototype Consistency: The embedding prototypes should show low variance. Concretely, compute the mean and standard deviation of embeddings of the reviewed phrases. If the standard deviation is high (meaning very divergent expressions), the concept may be too broad. We might set a threshold on the ratio of intra-concept variance to inter-concept distance. (No direct cite, but this aligns with “centroid spread” metrics.)

• Calibration Margin: Empirically, during calibration we could measure the typical score margin between this concept and its nearest other. If that margin is below some threshold on the calibration set, it indicates risk of confusion; we would then require additional evidence (perhaps more phrase variants) to tighten the cluster.

In practice, “exact evidence” might translate to:

Example Requirement: “Each concept must have ≥3 reviewed expressions spanning ≥2 languages, and the average cosine similarity among its prototype embeddings ≥ 0.8 (indicating a cohesive cluster). Its nearest-other-concept centroid should be ≥ 0.1 distance further away than the within-concept radius. Only concepts meeting these criteria are used for auto-resolution.”

These criteria ensure the concept is well-supported and distinct. They are more stringent than current practice, reflecting the high risk aversion. They are in line with the registry’s emphasis on provenance and equivalence strength: we need strong, diverse evidence before we trust a model to generalize to that concept.

Bibliography

• Artetxe, M., & Schwenk, H. (2019). Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond. TACL, 7:597–610. URL (2019).
• Feng, F., Yang, Y., Cer, D., Arivazhagan, N., & Wang, W. (2022). Language-agnostic BERT Sentence Embedding. Proc. ACL 2022. URL (2022) – Introduces LaBSE; reports 83.7% vs 65.5% retrieval accuracies.
• Geng, C., Huang, S.-J., & Chen, S. (2020). Recent Advances in Open Set Recognition: A Survey. arXiv:1811.08581. URL (rev. March 2020) – Survey defining open-set recognition where “unknown classes can be submitted… requiring [models] to deal with unseen ones”.
• Geifman, Y. & El-Yaniv, R. (2017). Selective Classification for Deep Neural Networks. NeurIPS 2017. – Shows a method to set thresholds on max-softmax (Softmax Response) to guarantee a desired risk level.
• Huang, J., Lu, P., Zeng, Q., Iwasawa, Y., Matsuo, Y., Chandar, S., & Li, I. (2026). Investigating the Multilingual Calibration Effects of Language Model Instruction-Tuning. EACL 2026 (accepted). [arXiv:2601.01362]. – Finds that instruction-tuning can increase confidence in low-resource languages without accuracy gains (miscalibration) and that label-smoothing helps maintain cross-language calibration.
• Lang, H., Zheng, Y., Li, Y., Sun, J., Huang, F., & Li, Y. (2023). A Survey on Out-of-Distribution Detection in NLP. arXiv:2305.03236 (Dec 2023). – Emphasizes need for NLP systems to detect OOD intents unseen in training and formalizes semantic shifts (new label spaces) as focus.
• Li, H., Ma, Y., Yu, Z., & Wu, B. (2024). NV-Retriever: Improving Text Embedding Models with Effective Hard-Negative Mining. arXiv:2407.15831 (July 2024). – Discusses training dense retrievers: highlights that naive in-batch negatives are easy, and adding hard negatives (e.g. BM25-retrieved) greatly improves learning. Warns that naive mining yields many false negatives which must be filtered.
• Pleiss, G. (2021). Neural Network Calibration (blog). – Demonstrates that modern DNNs are miscalibrated and that temperature scaling can nearly perfectly fix calibration without additional data.
• Reimers, N. & Gurevych, I. (2020). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. EMNLP 2019 (Technically, SBERT; see [27]). – [Documentation and tutorials] (GitHub) explain that Cross-Encoders (jointly encode [query, candidate]) yield higher scoring accuracy than Bi-Encoders, suggesting a bi-encoder + reranker pipeline.
• Traub, J., Bungert, T. J., Lüth, C. T., & Maier-Hein, K. (2024). Overcoming Common Flaws in the Evaluation of Selective Classification Systems. arXiv:2407.01032 (July 2024). – Defines risk (error rate) and coverage, stresses evaluating selective classifiers across all thresholds. Proposes AUGRC metric.
• Alansari, A., & Luqman, H. (2025). Large Language Models Hallucination: A Comprehensive Survey. arXiv:2510.06265 (Oct 2025). – Survey defining “hallucination” as LLM outputs that are fluent but factually incorrect, noting this “undermine[s] the reliability and trustworthiness” of LLMs in factual tasks.
• Embedded Semantics – Public Site (2026). Sections: Stable concept identity, Research & Methodology, FAQ, About. URL (Accessed Aug 2026). – Official description of the system: emphasizes “concept records define meaning” and that unresolved distinctions are preserved. FAQ clarifies production resolver behavior and use of hard negatives.

(All URLs accessed in 2026, publication dates given. For full reference details, see the URLs above.)