Language-Agnostic Embeddings by Averaging Mutable Translations

Full report

On this page

Executive summary

Averaging embeddings across translations can produce a useful language-agnostic representation, but it is not a magic operation. It works best when three conditions hold simultaneously: the translations are genuinely semantically equivalent for the task at hand, the underlying embedding space is already aligned or has been aligned into a common geometry, and the vectors are normalized so that language-specific anisotropy or norm differences do not dominate the average. Under those conditions, averaging behaves like prototype construction: it suppresses language-specific noise and retains the shared semantic signal. This logic is strongly supported indirectly by cross-lingual word-mapping work, multilingual sentence encoders trained to pull translations together, and meta-embedding results showing that simple means can be surprisingly competitive once geometry is handled properly.

The strongest practical recommendation is therefore conservative: if the “items” are words or short labels, use aligned word spaces such as MUSE or VecMap-style mappings; if the items are phrases, titles, or sentences, use a multilingual sentence encoder such as LASER, LaBSE, a multilingual Sentence-Transformer, multilingual-E5, or BGE-M3, then normalize each translation embedding, average them, and normalize again. For static spaces, add mean-centering, length normalization, orthogonal alignment, and CSLS-based retrieval. For raw pretrained language-model sentence vectors, consider mean-centering or whitening before aggregation because anisotropy is a documented problem.

Simple mean is the baseline to beat. Weighted means are often better when translations vary in informativeness or noise level; classic examples include IDF/SIF-style weighting for bag-of-words sentence representations. By contrast, direct primary-source evidence for coordinate-wise medians, geometric medians, or other robust estimators specifically over multilingual translation sets is sparse. They are reasonable engineering options when some translations are noisy, idiomatic, non-literal, or sense-mismatched, but the literature surveyed here provides much stronger direct support for aligned means, weighted means, and learned cross-lingual encoders than for robust estimators over translation sets.

The main failure modes are not subtle. Averaging can blur polysemy, collapse literal and idiomatic readings, wash out word-order-sensitive contrasts, and over-smooth lexical or morphological distinctions that matter for the application. These risks are especially severe for single words without sense disambiguation, for distant language pairs where linear isomorphism is weak, and for “translations” that are really summaries, explanations, or localizations rather than meaning-preserving renderings. In short: averaging is most defensible when the translation set is sense-specific, clean, aligned, and task-matched.

Definitions and scope

For this report, a language-agnostic embedding is a vector representation in which semantically equivalent items from different languages are intended to lie close enough that downstream comparison can be performed directly in embedding space, without translating everything into one pivot language at inference time. That is the design goal of multilingual word-alignment systems such as MUSE and VecMap, multilingual sentence encoders such as LASER and LaBSE, and more recent multilingual retrieval models such as multilingual-E5 and BGE-M3.

A mutable translation is best understood operationally, not philosophically. It is a surface realization in another language that may vary in wording, morphology, register, script, or local syntax while preserving the task-relevant content of an item. In this sense, “mutable” includes literal translations, close paraphrastic translations, alternative polite or informal renderings, and inflectional or orthographic variants, but it excludes cases where the rendering adds or drops propositions, resolves ambiguity in a way not present in the source, transforms an idiom into a non-literal explanation, or otherwise changes what a benchmark like STS, NLI, or bitext mining would treat as semantically equivalent. This task dependence is consistent with how XNLI operationalizes translation equivalence for inference pairs, how multilingual STS operationalizes graded similarity, and how paraphrase and idiom studies show that surface variation is not automatically semantics-preserving.

The phrase same underlying meaning should therefore be treated as a task-conditioned equivalence relation. At the strongest level, it means preservation of truth-conditional or entailment-relevant content. At a weaker level, it can mean preservation of retrieval intent or referential identity. Those are not identical requirements. For example, two product titles in different languages may be “the same item” for catalog retrieval even if one adds brand qualifiers, but two sentences in an NLI benchmark are not equivalent if one softens a quantifier or changes negation. This distinction matters because averaging across translations of different semantic granularity will often create a vector that represents none of them well.

A practical implication follows. If the item is a word, especially a polysemous one, averaging across its possible translations is dangerous unless the sense is first fixed. If the item is a sentence or short text unit that already has a stable intended reading, averaging is much more defensible because modern multilingual sentence encoders were explicitly trained to make translations neighbors in a common space.

Why averaging can work and when it breaks

The cleanest theoretical story is the prototype view. Suppose each translation embedding \(z_i\) can be decomposed into a shared semantic component \(s\) plus language-specific and lexical noise \(\epsilon_i\). If the embeddings already live in a common semantic space and the noise terms are roughly zero-mean, the arithmetic mean estimates \(s\) while reducing variance. This is the same broad intuition behind translation-ranking sentence encoders, cross-lingual distillation methods that force translations to occupy the same region, and meta-embedding work in which simple averaging becomes competitive once spaces are made comparable.

The literature provides three strong pieces of evidence for this prototype view. First, word-mapping work shows that orthogonal alignment plus careful similarity scaling can make bilingual spaces comparable enough for translation retrieval; Conneau et al. explicitly refine an adversarially initialized mapping with Procrustes and then retrieve with CSLS, and they report strong gains for both word translation and sentence translation retrieval using IDF-weighted sentence averages. Second, LaBSE learns cross-lingual sentence embeddings by combining MLM, TLM, dual-encoder translation ranking, and additive-margin softmax, and reports a large Tatoeba advantage over LASER, which is exactly what one would expect from a model trained to collapse translation variants toward a shared semantic point. Third, Coates and Bollegala show that even across distinct source embeddings, arithmetic averaging can be surprisingly effective, while Jawanpuria et al. show that averaging is more amenable after learned orthogonal rotations and a common Mahalanobis metric scaling.

What breaks this story is geometry. Pretrained language-model sentence representations are often anisotropic: certain high-variance directions or language-specific means dominate the space. Li et al. show that vanilla BERT sentence embeddings form a non-smooth anisotropic semantic space, and Su et al. show that whitening can improve isotropy and semantics. Chang et al. further show, using XLM-R, that different languages occupy similar linear subspaces after mean-centering, while language-specific means encode language-sensitive information. That means naive averaging can accidentally average language identity, frequency bias, and other nuisance structure unless normalization is done first.

What also breaks the story is non-compositionality and ambiguity. Contextual embedding analysis using paraphrases shows that BERT handles polysemous words better than static embeddings, but also gives synonyms surprisingly different representations in many cases and remains sensitive to word order. Idiomatic expressions are worse: their meaning is not inferable from constituent words, and dedicated idiom work such as MICE exists precisely because ordinary compositional assumptions fail. Consequently, averaging literal and non-literal translations, or mixing translations that resolve ambiguity differently, can produce a vector that is semantically blurred or even misleading.

Finally, averaging breaks when alignment assumptions are too weak. Mapping-based bilingual lexicon induction has long relied on approximate geometric isomorphism between spaces. That assumption is good enough for many related languages, but it degrades for distant pairs and for scenarios with significant domain or corpus mismatch. This is why later methods emphasize orthogonal constraints, whitening, self-learning, latent metric spaces, or nonlinear mappings, and why mBERT’s cross-lingual transfer is documented to work better for typologically similar languages and similar word orders.

Methods landscape

The multilingual-embedding literature falls into a few clear families. The table below compares the most relevant ones for translation-set averaging.

Several conclusions follow from this landscape. If you want to average vectors for word senses or short lexical items, mapping-based static spaces are still attractive because they make alignment and postprocessing explicit. If you want to average vectors for phrases, titles, or sentences, sentence-level multilingual encoders are the better starting point because they internalize the translation-alignment problem during training instead of leaving it to post hoc Procrustes.

A second conclusion is that the phrase “language-agnostic” hides an important distinction. LASER and LaBSE are built to place translations close together at the sentence level. mBERT and XLM-R are powerful multilingual encoders, but they are not, by themselves, optimized as sentence-embedding models; the literature on BERT-flow and whitening exists because naive sentence extraction from pretrained language models often underuses semantic information. That distinction matters directly when one plans to average translation embeddings.

Evidence on averaging, alignment, and evaluation

The most direct empirical evidence for translation-set averaging comes from old but still informative cross-lingual sentence retrieval work. In the MUSE paper, sentence translation retrieval is performed with IDF-weighted averages of aligned word embeddings over Europarl sentences, and the paper reports meaningful gains when the word spaces are aligned well and retrieved with CSLS rather than plain nearest neighbors. This is a genuine cross-lingual averaging result: the sentence vector is literally an average over token vectors in an aligned multilingual space.

Weighted averaging has independent support from sentence-embedding work outside the multilingual setting. Arora, Liang, and Ma’s SIF baseline represents a sentence as a weighted average of word vectors followed by principal-component removal and reports strong STS performance gains over unweighted baselines. Although that paper is monolingual, its lesson transfers cleanly to multilingual settings: if function words or highly frequent lexical items differ across translations, weighting can make the aggregated vector more semantic and less frequency-driven.

Simple mean is harder to dismiss than intuition suggests. Coates and Bollegala show that arithmetic averaging can yield competitive meta-embeddings even across source spaces that are not trivially comparable, and they explicitly discuss why the result is less paradoxical than it appears. Jawanpuria et al. strengthen that message by showing that averages become more meaningful after orthogonal rotations into a common latent space and a shared metric scaling. For the present problem, the implication is straightforward: once multilingual translations are embedded in a sufficiently common and well-conditioned space, a simple mean should be a serious baseline rather than a throwaway heuristic.

By contrast, strong direct benchmark evidence for coordinate-wise median, geometric median, trimmed mean, or similar robust estimators over multilingual translation sets is limited in the primary literature I found. The survey and methodological papers around meta-embeddings and pooling are dominated by means, weighted means, concatenation, learned attention, and latent-space alignment; robust estimators appear much more as a theoretically plausible engineering choice than as a well-established multilingual benchmark winner. The safest analytic conclusion is therefore that robust estimators are promising for contamination-heavy translation pools, but not yet canonized by the same level of evidence as aligned means and weighted means.

Alignment and postprocessing are not optional details; they are central to whether averaging is meaningful at all. For static spaces, the core pipeline in the literature is now fairly standard: length-normalize, mean-center or otherwise normalize, solve an orthogonal mapping via Procrustes or self-learning, optionally whiten/reweight, and use CSLS at retrieval time to mitigate hubness. Artetxe et al. make the orthogonal transform the core of a broader linear-transformation framework, while Conneau et al. show that CSLS provides a strong and robust gain and does not need delicate hyperparameter tuning in their setup.

For contextual or sentence-level multilingual encoders, external word-space alignment is usually unnecessary, but normalization still matters. BERT-flow, whitening, and multilingual-geometry results all point in the same direction: shared semantics are mixed with nuisance geometry such as anisotropy and language means. If you plan to average multiple LaBSE, LASER, XLM-R, or multilingual-E5 sentence vectors into a prototype, the evidence favors at least L2 normalization per vector and, when using raw or mixed model outputs, development-set checks for mean-centering or whitening.

The main public evaluation suites for the problem are also clear.

Implementation and experimental design

The implementation decision that matters most is the unit of representation. If the item is truly lexical, use a word-level pipeline built on aligned static embeddings, ideally with subword-aware models such as fastText so that morphology and OOV forms are handled better. fastText represents words as bags of character n-grams, which is especially useful for morphologically rich languages and rare forms. If the item is a sentence, title, or phrase, prefer a sentence encoder; SBERT-style bi-encoders exist precisely because pairwise cross-encoders are too expensive for large-scale semantic search, and multilingual extensions and retrieval-focused models build on that same computational logic.

Tokenization should be left to the model’s own tokenizer. LASER uses shared BPE, XLM-R uses SentencePiece-based subword segmentation, and SentencePiece itself was designed to avoid language-specific pre- and postprocessing. In practice, that means one should not manually normalize scripts or segment words unless the model documentation explicitly expects it; do keep original case, punctuation, and script as the checkpoint expects.

A sensible aggregation pipeline is shown below. It is consistent with the alignment literature for static spaces and with the sentence-embedding postprocessing literature for contextual spaces.

flowchart LR
    A[Canonical item or sense] --> B[Collect multiple translations]
    B --> C[Filter for task-level semantic equivalence]
    C --> D[Embed each translation]
    D --> E{Embedding family}
    E -->|Static word spaces| F[Length normalize and mean-center]
    F --> G[Orthogonal alignment and optional whitening]
    G --> H[Aggregate translations]
    E -->|Joint multilingual sentence model| I[L2 normalize and optional mean-centering or whitening]
    I --> H[Aggregate translations]
    H --> J[Renormalize prototype]
    J --> K[Evaluate with retrieval, STS, or transfer task]

For the aggregation operator itself, the best report-ready baseline is:

\[ p = \text{norm}\!\left(\frac{1}{n}\sum_{i=1}^n \text{norm}(z_i)\right), \]

where norm is L2 normalization. If translations are of visibly uneven quality, use

\[ p_w = \text{norm}\!\left(\frac{\sum_i w_i \,\text{norm}(z_i)}{\sum_i w_i}\right), \]

with \(w_i\) set by IDF/SIF-like informativeness, translation confidence, or a learned gating model. For robust alternatives, compare coordinate-wise median, geometric median, a translation medoid (the translation whose embedding is closest to the others), and a trimmed-mean variant that discards the most outlying translation before averaging. The literature gives you stronger prior confidence in the mean and weighted mean than in the others, so the robust estimators should be treated as ablations rather than defaults.

A rigorous experiment suite should include at least the following conditions. Use MUSE/VecMap-aligned fastText as the word-level baseline; LASER and LaBSE as sentence-level bitext baselines; mean-pooled XLM-R as a strong but imperfect contextual baseline; a distilled multilingual Sentence-Transformer; and at least one recent retrieval model such as multilingual-E5 or BGE-M3. Then evaluate single-translation embeddings, unweighted means, weighted means, medoids, and robust estimators under three preprocessing settings: raw, L2-normalized, and mean-centered-plus-L2-normalized. For static spaces, add a no-alignment control and an orthogonal-alignment-plus-CSLS condition.

The metrics should be chosen by task rather than by convenience. Use cosine similarity for nearest-neighbor retrieval and clustering once vectors are normalized; use P@1, P@5, and P@10 on MUSE and Tatoeba-style retrieval; use F1 on BUCC; use Spearman on multilingual STS; use accuracy on XNLI; and use MAP or nDCG on multilingual retrieval suites such as MIRACL if you want a harder retrieval stress test. Do not let a single metric stand in for “language agnosticism.” A method can look excellent on sentence retrieval and still fail on entailment-sensitive tasks.

The expected outcome, based on the literature, is not that averaging will revolutionize already strong multilingual sentence encoders. Rather, I would expect modest, reliable gains in robustness when the translation set is clean and meaning-preserving, larger gains for static or weakly aligned baselines, and the clearest benefits under noisy retrieval or cross-domain variation. I would also expect gains to saturate after a small number of translations, because models such as LaBSE, multilingual Sentence-Transformers, and multilingual-E5 are already trained to minimize cross-lingual dispersion among equivalent texts.

Recommendations and research agenda

For practitioners, the best default is simple. If the item is longer than a word, start with a multilingual sentence encoder, not a word-alignment pipeline. Generate one embedding per translation, L2-normalize each vector, average them, and L2-normalize again. Use the arithmetic mean as the default baseline and add a weighted mean only if you have a principled reliability or informativeness signal. For sentence embeddings extracted from raw pretrained multilingual LMs rather than dedicated embedding models, test mean-centering or whitening on a held-out development set before finalizing the pipeline.

If the item is a word or short label, sense disambiguation is the first job, not averaging. A multilingual prototype built from the translations of an ambiguous English word such as bank will often be worse than the best single sense-specific translation. In that setting, aligned static embeddings remain useful because they expose the alignment process and permit dictionary-based evaluation, but they should be paired with strict sense inventory control or contextual disambiguation.

Avoid averaging across translations that are not genuinely commensurate. Exclude summaries, explanations, culturally adapted localizations, literal renderings of idioms that differ in meaning from idiomatic paraphrases, and any translation that changes quantification, polarity, or named-entity granularity. The more semantically heterogeneous the translation pool, the more the average becomes a compromise vector rather than a semantic prototype.

From an evaluation standpoint, do not declare success on one benchmark. A strong result should hold simultaneously on at least one exact-equivalence retrieval benchmark such as Tatoeba or BUCC, one graded-similarity benchmark such as multilingual STS, and one more semantics-sensitive transfer benchmark such as XNLI. If the method is intended for retrieval systems, also check a realistic multilingual retrieval benchmark such as MIRACL or a broader suite like MMTEB.

The main open problem is that the literature still lacks a standard benchmark dedicated specifically to translation-set aggregation. Most evidence is indirect: token averaging inside cross-lingual retrieval systems, sentence encoders trained on parallel data, or meta-embedding studies across source spaces. A dedicated benchmark would vary the number of translations per item, translation quality, idiomaticity, polysemy, and typological distance, and would compare simple means, confidence-weighted means, medoids, robust estimators, and learned set encoders under a unified protocol. The current literature strongly motivates that benchmark, but does not yet fully supply it.

A second open problem is geometry-aware aggregation. Multilingual representation work increasingly suggests that embeddings occupy structured, sometimes highly anisotropic spaces with language-sensitive and language-neutral directions. That raises the possibility that Euclidean averaging is not always the right operator, especially after hyperspherical normalization. Future work should test spherical barycenters, learned subspace projection, and sense-aware manifold aggregation instead of assuming that the arithmetic mean is always the correct prototype operator. The recommendation here is therefore pragmatic rather than absolute: use the mean first, but treat it as a baseline conditioned on geometry, not as a theorem of multilingual semantics.

A final limitation of this review is worth stating explicitly. Primary-source evidence is rich for multilingual alignment, multilingual sentence encoders, averaging of token embeddings into sentence vectors, and simple averaging in meta-embedding. It is much thinner for direct head-to-head studies of robust estimators over groups of semantically equivalent translations. That gap should be read as a research opportunity, not as evidence that robust estimators are useless. But today, the highest-confidence practitioner guidance remains: align first, normalize carefully, define equivalence strictly, and treat the normalized simple mean as the baseline to beat.