The Digital Linguistic Divide: A Comprehensive Analysis of AI Training Corpora, Tokenization Economics, and Strategies for Equitable Representation

Full report

On this page

The exponential scaling of large language models (LLMs) has catalyzed a revolution in computational linguistics, human-computer interaction, and global information retrieval. However, the architectural foundation of these artificial intelligence systems relies upon a deeply skewed representation of human linguistics. While the physical world is characterized by an immense diversity of thousands of living languages, the digital ecosystem—and the generative models trained upon it—exhibit a profound linguistic hegemony. This comprehensive report investigates the systemic disparities in language representation across global digital demographics, online publishing volumes, large-scale web scraping architectures, quality filtering heuristics, subword tokenization mechanics, and downstream multilingual benchmarks. By analyzing the compounding effects of data scarcity, algorithmic exclusion, and the financial penalties imposed on morphologically rich languages, this analysis demonstrates how current computational paradigms inherently marginalize low-resource languages (LRLs). Furthermore, the investigation outlines an ethical, community-led data stewardship strategy designed to rectify these epistemic exclusions and cultivate culturally authentic training corpora.

Global Linguistic Demographics and the Threat of Extinction

To properly contextualize the linguistic disparities embedded within artificial intelligence, it is necessary to baseline the true linguistic diversity of the global population against its digital footprint. The global population of approximately 8 billion individuals speaks 7,170 living languages, distributed across 242 countries1. However, this diversity is highly concentrated and structurally fragile. Ethnologue demographic data indicates that 90% of these languages are spoken by populations of 100,000 or fewer3. Furthermore, the linguistic vitality of the globe is under severe threat. Current indigenous language vitality assessments classify 482 languages as institutional, 3,462 as stable, 3,226 as endangered, and 448 as extinct2. The United Nations Educational, Scientific and Cultural Organization (UNESCO) Atlas of the World's Languages in Danger provides critical insight into this fragility. The third edition of the Atlas expanded its monitoring from 600 to 2,500 endangered languages, utilizing interactive mapping and vitality parameters to track linguistic decline4. The classification system ranges from "critically endangered"—where the youngest speakers are grandparents who speak the language only partially and infrequently—to completely "extinct"5. Predictive linguistic models estimate that up to 90% of the world's languages may face extinction by the year 2100, which equates to a language dying approximately every two weeks5. The preservation of these languages is frequently hindered by a lack of institutional support; as observed with the Arbereshi (80,000 speakers) and Arvanitika (50,000 speakers) varieties of Albanian in Italy and Greece, linguistic nourishment is fundamentally an institutional duty rather than an individual burden5. When analyzing the distribution of the most widely spoken languages, a critical distinction emerges between first-language (L1) native speakers and second-language (L2) adoption. This distinction is vital for understanding digital representation, as total speaker counts often mask the reality of native use, cultural depth, and the volume of original literary output.

Table 1: The world's largest languages by speaker count, demonstrating the divergence between native (L1) and second-language (L2) adoption6. As illustrated, English dominates the total speaker count entirely due to its status as a global lingua franca, despite Mandarin Chinese, Spanish, and Hindi possessing vastly larger native-speaker populations1. Linguistic parameters further complicate these demographics; for instance, if Hindi and Urdu were counted as a single mutually intelligible language (Hindustani), it would rank ahead of English in native speaker volume1. However, when transitioning from demographic reality to digital representation, this distribution is radically inverted, creating the foundational inequality of the digital age.

The Hegemony of the Web: Content Languages and Digital Infrastructure

An analysis of global web content language statistics reveals a digital environment wherein English exerts an absolute, self-perpetuating monopoly. Data tracking the content languages of the world's websites demonstrates that nearly half of all indexable digital text is published in English7. This dominance is not merely a reflection of user preference but is deeply intertwined with the underlying technologies that power the internet.

Table 2: Usage statistics of content languages for globally tracked websites contrasted with approximate demographic representation6. The disparity is particularly glaring when juxtaposing demographic weight with digital footprint. Hindi, spoken by over 600 million people, represents less than 0.1% of indexable web content6. Bengali, with over 270 million speakers, similarly accounts for only 0.1% of the web6. Conversely, German, representing a significantly smaller demographic base, commands nearly 6% of web publications6. This divergence is historically rooted in the unequal distribution of digital infrastructure, economic imbalances, and the architecture of web technologies. The digital ecosystem is heavily biased toward Latin-script, high-resource environments. For example, Content Management Systems (CMS) like WordPress power over 40% of the web, and server-side programming languages such as PHP are utilized by nearly 70% of websites9. The documentation, support forums, and default configurations for these underlying technologies are predominantly English-centric9. Consequently, the barrier to entry for developing web infrastructure in non-Latin scripts or low-resource languages is artificially elevated, suppressing the organic volume of online publishing from these communities.

The Pre-Training Bottleneck: Data Extraction and Common Crawl

Generative LLMs operate on the principle of the scaling law, which dictates that model performance correlates directly with the volume and quality of the pre-training data11. To satisfy this insatiable demand for text, AI developers rely heavily on large-scale scraping operations. The Common Crawl foundation provides the most prominent open-source web corpus, representing petabytes of text and serving as the foundational substrate for almost all major foundational models12. However, because this raw material is derived directly from the skewed web ecosystem, the extraction process naturally inherits, and often amplifies, the underlying representational biases. Statistical distributions generated from recent Common Crawl archives emphasize this centralization. The percentage of primary language documents retrieved via the Compact Language Detector 2 (CLD2)—which is capable of identifying 160 different languages—heavily favors Western European languages and Russian14.

Table 3: Common Crawl primary language distributions over successive archival dumps, measured as a percentage of total ingested HTML pages14. The architectural reliance on these massive dumps creates a mathematically unavoidable data scarcity problem. English dominates approximately 42% of the extracted documents, vastly over-indexing its share of global speakers14. Multilingual datasets hosted on platforms like the Hugging Face Hub that incorporate English add another 30% to this dominance, further skewing the representation16. Languages spoken by hundreds of millions of people across the African continent and Southeast Asia barely register in the fractions of a percent14. As a result, the pre-training objective of next-token prediction lacks the requisite statistical density to form coherent semantic representations for the vast majority of the world's languages.

Algorithmic Exclusion: The Epistemological Violence of Quality Filtering

The challenge of pre-training LLMs extends beyond merely gathering petabytes of data; the raw material must be aggressively cleansed. Web scrapes contain substantial volumes of spam, boilerplate code, adult content, and unintelligible text17. To sanitize this data, researchers deploy automated quality filtering pipelines. However, these quality heuristics frequently function as unintentional agents of linguistic exclusion, disproportionately purging data from low-resource languages, regional dialects, and informal vernaculars.

The Perplexity Penalty and KenLM Mechanics

One of the most ubiquitous methods for text quality filtering utilizes a lightweight n-gram language model, almost exclusively KenLM, to calculate the perplexity of scraped documents11. A KenLM model computes the probability of a sentence based on the probabilities of its constituent words19. The perplexity score is derived mathematically as the inverse probability of the test set, normalized by the number of words. The model is typically trained on a trusted "high-quality" reference corpus, such as a language-specific Wikipedia dump18. During the filtering phase, the model evaluates new documents; if a text exhibits a high perplexity score—meaning the statistical distribution of its word sequences differs significantly from the pristine reference corpus—it is flagged as low-quality, out-of-domain, or noisy, and subsequently discarded11. This mechanism establishes a severe, systemic bias against non-standardized languages. For many marginalized communities, Wikipedia is either non-existent, severely underdeveloped, or written in a highly academic, stilted register that fails to capture natural, everyday usage. When an n-gram model trained on formal, standardized text encounters regional dialects, community forum discussions, code-switching, or oral traditions transcribed into text, it assigns an artificially high perplexity score due to the presence of out-of-vocabulary words and novel syntactic structures11. Consequently, the filter misidentifies authentic cultural expression as spam. Furthermore, KenLM relies heavily on Kneser-Ney smoothing to estimate the probability of unseen n-grams19. In highly agglutinative or morphologically rich languages—where single root words can accept hundreds of valid prefixes and suffixes to convey complex meaning—the frequency of specific n-grams is naturally sparse19. The n-gram model interprets this morphological richness as a statistical anomaly, leading to elevated perplexity scores and the algorithmic deletion of valid linguistic data. Research indicates that utilizing perplexity thresholds derived from high-resource languages and arbitrarily applying them across a multilingual corpus systematically decimates the survival rate of text from less commonly written languages19. Recent theoretical advancements propose mitigation strategies, such as an ensemble approach utilizing both a "Good KenLM" trained on high-quality text and a "Bad KenLM" trained deliberately on noisy data (e.g., spam, hate speech)11. By employing Z-score standardization to balance the scores, this ensemble method isolates true noise from benign dialectal variations, significantly improving the recall of high-quality data in resource-constrained environments without the need for computationally expensive neural classifiers11. However, even advanced filtering cannot materialize data that does not exist.

Innovations and Limitations in Multilingual Pipelines: FineWeb and FineWeb2

Recognizing the catastrophic data loss caused by monolithic, English-centric filtering rules, recent engineering efforts have attempted to refine multilingual data processing. The FineWeb2 dataset, released through the Hugging Face ecosystem, introduces a highly sophisticated pipeline specifically engineered to adapt pre-training data processing to individual languages22. FineWeb2 curates high-quality documents spanning 20 primary languages, moving away from pure n-gram perplexity by utilizing an XLM-RoBERTa embedding-based deep learning classifier to score documents for structure and knowledge density23.

Table 4: High-quality selection (Top 10%) of the FineWeb2-HQ dataset across varying linguistic subsets, demonstrating the resulting volume of curated pre-training data23. The critical innovation of the FineWeb2 pipeline is its dynamic adaptation24. Rather than enforcing universal constants for heuristic filters, the system automatically tailors thresholds—such as the ratio of symbols to words, punctuation fractions, and n-gram repetition concentration—based on distribution-derived rules specific to each language24. For instance, FineWeb2 avoids applying a fixed English-translated stopword list. Instead, it defines stopwords dynamically based on relative frequency thresholds within reference datasets, accommodating the fact that a language like German distributes the syntactic function of the English "the" across "der," "die," and "das"24. Crucially, FineWeb2 integrates targeted "precision filtering" specifically aimed at low-resource languages to combat the high levels of cross-lingual contamination common in web scrapes24. This mechanism builds an affinity list of words common in a target language but uncommon elsewhere, filtering out documents that fail this affinity test24. The pipeline also leverages MinHash-based clustering for per-language deduplication, removing highly similar documents to increase training efficiency by a factor of six23. Furthermore, it uses language family information to intelligently propagate tokenizers; if a language lacks a native tokenizer, it assigns one from a closely related language within the same family and script24. While these pipeline improvements significantly enhance the quality of the retained data and set a new standard for open-source multilingual curation, they possess inherent limitations. A highly optimized filtering pipeline applied to an empty reservoir yields nothing; if native speakers are not actively publishing online, the data cannot be curated. Additionally, aggressive MinHash deduplication, while mathematically efficient, risks removing unique cultural context if similar narrative templates are repeatedly used to document disparate oral histories or regional lore18.

The Tokenizer Tax: Structural Inequity at the Subword Level

Even when text from low-resource languages successfully survives the extraction and filtering pipelines, it is subjected to an architectural penalty during model ingestion known as the "tokenizer tax" or "subword fertility disparity"26. Modern LLMs do not process raw text directly; they ingest discrete mathematical representations called tokens. Tokenizers, frequently utilizing algorithms like Byte-Pair Encoding (BPE) or WordPiece, are trained on the pre-training corpus to identify the most statistically frequent subword units, merging bytes into larger tokens to maximize computational efficiency27. Because English and high-resource Western languages overwhelmingly dominate the training data, the tokenizer learns to map entire English words (e.g., "understanding") into single, efficient tokens27. Conversely, when the tokenizer encounters a low-resource or morphologically complex language, it fails to recognize the statistical patterns of the words. Instead, it is forced to fragment the text into tiny, semantically meaningless subword units, or fallback entirely to individual character bytes27. The metric utilized to quantify this inefficiency is "tokenizer fertility"—defined as the average number of tokens required to represent a single word27.

Table 5: Subword fertility across European languages based on SIB-200 parallel text evaluation, indicating the tokenization penalty imposed by script and morphological complexity27. The consequences of high subword fertility are severe, structural, and multifaceted:

1. Financial Penalty: Commercial LLM API services are universally billed per token27. Because a morphologically complex language like Ukrainian requires 2.66 tokens per word compared to English's 1.23, a user querying an AI in Ukrainian pays substantially more for the exact same semantic payload, creating an immediate economic barrier to access27.

2. Context Window Constriction: Generative models operate with fixed context windows (e.g., 8,000 to 128,000 tokens)31. If a low-resource language requires two to three times as many tokens to express a thought, the effective memory and reasoning horizon of the AI is drastically slashed, preventing complex reasoning over long native-language documents27.

3. Performance Degradation: BPE algorithms applied to low-resource datasets often split words at arbitrary byte boundaries rather than logical morpheme boundaries27. For instance, instead of splitting a root word from its plural suffix, the tokenizer may arbitrarily split the root word itself. This structural disruption fundamentally impairs the model's ability to learn structural grammar, leading to degraded syntactic comprehension and depressed cross-lingual few-shot performance27. The tokenizer tax is an invisible, yet profound, structural disadvantage.

Empirical Asymmetries: Model Performance on Multilingual Benchmarks

The downstream effect of data scarcity, aggressive filtering, and tokenizer inefficiency culminates in stark performance asymmetries across languages. The evaluation of commercial, proprietary models and open-weight models using rigorous, multi-faceted benchmarks highlights the failure of massive parameter counts alone to bridge the linguistic divide.

The Belebele Reading Comprehension Benchmark

The Belebele benchmark represents a critical evaluation instrument for testing native-language reasoning. It is a multiple-choice machine reading comprehension dataset spanning 122 language variants32. Crucially, unlike earlier benchmarks that relied on automated machine translation—which introduces "translationese" and flattens cultural variance—Belebele relies on parallel data generated entirely by human experts fluent in both English and the target language33. Testing across this benchmark reveals a heavily bifurcated AI ecosystem. State-of-the-art models demonstrate near-human competence in high-resource environments but suffer catastrophic degradation in low-resource settings32.

Table 6: Average zero-shot reading comprehension accuracy on the Belebele benchmark across diverse linguistic tiers, illustrating the precipitous drop in capability for low-resource languages32. For high-resource targets, the models easily exceed 85% accuracy. However, for African languages such as Yoruba, Sesotho, and Shona, performance frequently drops into the 30% range—a result that is mathematically indistinguishable from random chance in a standard multiple-choice format32. Furthermore, functional evaluations expose even larger within-language variability than static data benchmarks36. The analysis also points to a profound divergence in evaluation reliability when utilizing LLMs as automated judges. When LLM judges are tasked with evaluating outputs in low-resource settings, inter-annotator agreement plummets35. While empirical studies indicate GPT-4o possesses stronger labeling capabilities for grammatical and technical constraints, LLaMA 3 70B demonstrates a slight advantage in correctness and cultural common sense evaluation; nevertheless, both architectures fail to consistently align with human raters on low-resource targets35. The paradox of high observed agreement often stems from skewed ratings rather than true evaluative comprehension35.

FLORES-200 and the Translation Gap

Evaluation of translation capabilities utilizing the FLORES-200 benchmark further underscores this structural deficit. FLORES-200 encompasses 204 language varieties and serves as a standard benchmark for evaluating Massively Multilingual Neural Machine Translation (NMT) and generative LLMs37. When analyzing translation accuracy, a recurring phenomenon is the significant discrepancy between standard BLEU and spBLEU (SentencePiece BLEU) scores37. Standard BLEU relies on exact word matches and whitespace tokenization, which severely penalizes morphologically rich languages that combine multiple concepts into unbroken character strings38. spBLEU attempts to mitigate this by tokenizing at the subword level38. However, for extremely low-resource languages, such as Waray (an Austronesian language of the Central Philippine subgroup spoken by three million people), the gap between the two metrics exposes the model's struggle with orthographic variance and synonym flexibility38. Regional, instruction-tuned LLMs—such as the Gemma-SEA-LION-v4-27B-IT model—show surprising capability to perform translation, but dedicated NMT models like NLLB-200 often maintain a distinct edge in syntactic fidelity for long-tail languages38. Crucially, both architectures frequently produce lexically valid and contextually accurate outputs that surface-level automatic metrics misclassify as errors due to the single-reference nature of the benchmark, rendering the evaluation of low-resource generation highly suspect38. Efforts to explicitly address these performance gaps, such as the Aya Expanse and Aya 101 models, have focused heavily on balancing representation through targeted multilingual generative instruction tuning across 101 languages40. While these highly performant open-source models succeed in closing the performance delta slightly by improving cross-lingual transfer, the foundational limitations imposed by pre-training data volume dictate that low-resource languages still lag significantly behind their high-resource counterparts42.

Cultural Misalignment and the Erasure of Oral Traditions

The disparity in digital linguistic representation is not merely an engineering bottleneck or an economic penalty; it constitutes an epistemological crisis. Language is the primary vehicle for cultural transmission, organizing indigenous knowledge systems, ecological observations, historical narratives, and complex social protocols. When automated scraping pipelines pull predominantly from standard written sources—news sites, government portals, and digitized encyclopedias—they systematically exclude the vernaculars and oral traditions that characterize how the majority of the global population naturally communicates5. Many indigenous and low-resource languages lack a deeply entrenched, globally recognized standardized orthography5. Consequently, native speakers operating in digital spaces may utilize fluid spelling conventions, organically mix dialects, or leverage complex code-switching in their communication. Data-quality heuristics strictly penalize this fluidity. The algorithmic filters demand a rigid statistical predictability that oral histories transcribed into online forums simply do not possess20. The resulting AI models internalize an impoverished, sterile version of the language, stripped of its idioms, culturally specific ways of expressing knowledge, and nuanced worldview. This leads directly to cultural misalignment. When tested on cultural appropriateness, models trained strictly on westernized, translated texts generate responses that may be grammatically sound but are pragmatically absurd or culturally offensive to the native community32. Experimental interventions on the Winogrande benchmark reveal the magnitude of this issue: when GPT-4o was evaluated out-of-the-box on subsets of data specifically split by human annotators for cultural appropriateness, the model achieved an average performance lift of \+3.3% across all languages (and up to \+12.6% for Shona) on the culturally appropriate subsets32. This indicates that the models possess underlying reasoning capabilities that are actively suppressed when forced to operate on culturally misaligned or translated prompts32. True proficiency requires context, not just vocabulary.

Toward Ethical Stewardship: The CARE Principles and Community Publishing

Rectifying the digital language deficit requires a fundamental pivot away from the colonial paradigm of extraction—where large technology entities scrape whatever residual data exists on the web—toward an ethical paradigm of intentional, community-led data cultivation. Improving the cultural authenticity of training and retrieval materials dictates that native speakers must be empowered and financially supported to author original articles, community documentation, and digital stories in their own languages.

The CARE Principles for Indigenous Data Governance

To prevent the exploitation of marginalized languages by corporate actors, data generation initiatives must strictly adhere to the CARE Principles for Indigenous Data Governance: Collective Benefit, Authority to Control, Responsibility, and Ethics45. Unlike the standard FAIR principles (Findable, Accessible, Interoperable, Reusable), which focus strictly on data utility and machine readability, the CARE principles emphasize the human, historical, and cultural rights of the data creators45. Native speaker leadership ensures the implementation of:

1. Informed Consent and Authority: Language communities maintain absolute sovereignty over how their traditional knowledge, oral histories, and sacred texts are digitized and utilized by commercial LLMs.

2. Cultural Stewardship: Taxonomies, semantic nuances, and culturally sensitive materials are curated by those who possess the lived context to understand them, rather than outsourced to automated perplexity classifiers or underpaid gig workers.

3. Fair-Share Mechanics: Any commercial, academic, or technological benefit derived from the integration of these linguistic datasets into foundational models must result in material or non-material reciprocation to the originating community45.

A Strategy for Culturally Authentic Material Generation

The most effective, sustainable method to circumvent the KenLM perplexity penalty and bypass the tokenization tax is to flood the digital ecosystem with high-quality, high-density native text. Relying on machine translations of English data is a failed paradigm; it introduces systemic "translationese," degrades syntactic variety, and flattens cultural variance, ultimately failing to improve the underlying model reasoning34. Instead, language communities should be incentivized to produce original digital literature. By systematically publishing technical explanations, localized Wikipedia equivalents, agricultural techniques, legal records, and community lore online, native speakers organically construct the reference corpora required to train accurate, culturally aligned language models. This localized publishing strategy provides the massive data volume required for subword tokenizers to eventually encounter sufficient frequency data, allowing them to map complete morphological roots rather than arbitrary bytes, thereby drastically reducing the fertility penalty27. Furthermore, original text provides the grammatical diversity necessary to train more forgiving and precise deep-learning quality filters—akin to those deployed in FineWeb2—ensuring that the AI architectures of the next decade can parse, respect, and accurately reproduce the full spectrum of human communication23.

Conclusion

The architecture of modern artificial intelligence currently operates on a digitally homogenized substrate that profoundly distorts global linguistic reality. While the physical world relies on thousands of diverse languages, digital scraping mechanisms favor a highly localized fraction of Latin-script dialects. This disparity is mechanically enforced at every stage of the AI development pipeline: from the raw extraction of Common Crawl data, to the perplexity filtering heuristics that misinterpret cultural dialects as statistical noise, culminating in a subword tokenizer tax that financially, computationally, and syntactically penalizes low-resource users. The empirical manifestations of this architecture—demonstrated through the collapse of reasoning accuracy in benchmarks like Belebele and translation fidelity in FLORES-200—indicate that scaling parameters alone will not resolve the deficit. A sustainable, equitable future for natural language processing requires structural intervention. It demands the continuous refinement of adaptive, language-specific filtering algorithms, the modernization of tokenization mechanics to respect morphological boundaries, and, most crucially, the prioritization of ethical data stewardship. By empowering native speakers to generate original, culturally authentic material under the CARE principles of data governance, the artificial intelligence community can begin to construct foundational models that reflect the true multiplicity of human thought.

Works cited

1. Languages by number of native speakers | List, Top, & Most Spoken, https\://www\.britannica.com/topic/languages-by-number-of-native-speakers-2228882

2. Ethnologue | Languages of the world, https\://www\.ethnologue.com/

3. Global Language Statistics: By Country \[2025\], https\://www\.simonandsimon.co.uk/blog/global-language-statistics-by-country/

4. UNESCO \- Atlas of The World's Languages in Danger | PDF \- Scribd, https\://www\.scribd.com/document/682140707/UNESCO-Atlas-of-the-World-s-Languages-in-Danger

5. UNESCO's Atlas on Endangered Languages and the Local Context, https\://www\.researchgate.net/publication/357786961\_UNESCO's\_Atlas\_on\_Endangered\_Languages\_and\_the\_Local\_Context

6. List of languages by total number of speakers \- Wikipedia, https\://en.wikipedia.org/wiki/List\_of\_languages\_by\_total\_number\_of\_speakers

7. Usage statistics of content languages for websites \- W3Techs, https\://w3techs.com/technologies/overview/content\_language

8. Top Site Usage Statistics of Content Languages \- W3Techs, https\://w3techs.com/technologies/topsite/content\_language

9. W3Techs \- extensive and reliable web technology surveys, https\://w3techs.com/

10. Distribution of Content Management Systems among websites that, https\://w3techs.com/technologies/segmentation/cl-ja-/content\_management

11. Rethinking KenLM: Good and Bad Model Ensembles for Efficient, https\://arxiv.org/html/2409.09613v1

12. A Sampling of 2025 Research Referencing Common Crawl, https\://commoncrawl.org/blog/a-sampling-of-2025-research-referencing-common-crawl

13. Expanding the Language and Cultural Coverage of Common Crawl, https\://commoncrawl.org/blog/expanding-the-language-and-cultural-coverage-of-common-crawl

14. Statistics of Common Crawl Monthly Archives by commoncrawl, https\://commoncrawl.github.io/cc-crawl-statistics/plots/languages

15. commoncrawl/statistics · Datasets at Hugging Face, https\://huggingface.co/datasets/commoncrawl/statistics

16. Language distribution across Hugging Face (HF) datasets, Common, https\://www\.researchgate.net/figure/Language-distribution-across-Hugging-Face-HF-datasets-Common-Crawl-CC-pages-and\_tbl2\_400370924

17. arXiv:2409.09613v1 \[cs.CL\] 15 Sep 2024, https\://arxiv.org/pdf/2409.09613

18. arXiv:2311.12537v1 \[cs.CL\] 21 Nov 2023, https\://arxiv.org/pdf/2311.12537

19. Efficient Pre-Training of a Spanish Language Model using Perplexity, https\://arxiv.org/pdf/2207.06814

20. A Survey on Data Selection for Language Models \- arXiv, https\://arxiv.org/html/2402.16827v1

21. arXiv:2205.04810v1 \[cs.CL\] 10 May 2022, https\://arxiv.org/pdf/2205.04810

22. FineWeb2 \- a HuggingFaceFW Collection, https\://huggingface.co/collections/HuggingFaceFW/fineweb2

23. epfml/FineWeb2-HQ · Datasets at Hugging Face, https\://huggingface.co/datasets/epfml/FineWeb2-HQ

24. A Guide to the FineWeb2 Dataset: How It's Built, Filtered, and Used, https\://kili-technology.com/blog/fineweb2-dataset-guide

25. FineWeb2: One Pipeline to Scale Them All — Adapting Pre-Training, https\://openreview.net/pdf?id=jnRBe6zatP

26. The Token Tax: Systematic Bias in Multilingual Tokenization \- arXiv, https\://arxiv.org/html/2509.05486v1

27. The Tokenizer Tax Across 25 European Languages \- arXiv, https\://arxiv.org/pdf/2605.24718

28. The Tokenizer Tax: Quantifying and Explaining the Cross-Lingual, https\://arxiv.org/pdf/2607.24276

29. Token Premiums of French and Regional Languages in 2026 LLM, https\://arxiv.org/abs/2609.39001

30. \[2605.24718\] The Tokenizer Tax Across 25 European Languages, https\://arxiv.org/abs/2605.24718

31. Token Premiums of French and Regional Languages in 2026 ... \- arXiv, https\://arxiv.org/html/2609.39001v1

32. Enhancing LLM Performance for Low-Resource African Languages, https\://ojs.aaai.org/index.php/AAAI/article/view/34996/37151

33. CroFinBen: A Multilingual Benchmark of Large Language Models, https\://www\.sciopen.com/article/10.1007/s11390-025-5524-7

34. First Steps in Benchmarking Latvian in Large Language Models, https\://aclanthology.org/2025.resourceful-1.22.pdf

35. Evaluating the Quality of Benchmark Datasets for Low-Resource, https\://aclanthology.org/2025.gem-1.41.pdf

36. Multi-lingual Functional Evaluation for Large Language Models \- arXiv, https\://arxiv.org/html/2506.20793v2

37. A Data-Centric Approach to Low-Resource Machine Translation, https\://arxiv.org/html/2605.27050v1

38. (PDF) Regional LLM versus Dedicated Neural MT for Waray–English, https\://www\.researchgate.net/publication/415181789\_Regional\_LLM\_versus\_Dedicated\_Neural\_MT\_for\_Waray-English\_A\_Bidirectional\_FLORES-200\_Evaluation\_of\_a\_Low-Resource\_Philippine\_Language

39. ChatGPT MT: Competitive for High \- ACL Anthology, https\://aclanthology.org/anthology-files/pdf/wmt/2023.wmt-1.40.pdf

40. Papers Explained 109: Aya 101 \- Medium, https\://ritvik19.medium.com/papers-explained-aya-101-d813ba17b83a

41. Papers Explained 151: Aya 23 \- Ritvik Rastogi \- Medium, https\://ritvik19.medium.com/papers-explained-151-aya-23-d01605c3ee80

42. Shivalika Singh | alphaXiv, https\://www\.alphaxiv.org/@shivalika-singh

43. hf-blog-translation/aya-expanse.md at main · huggingface-cn/hf-blog, https\://github.com/huggingface-cn/hf-blog-translation/blob/main/aya-expanse.md

44. Bridging the Gap: Enhancing LLM Performance for Low-Resource, https\://www\.researchgate.net/publication/387141227\_Bridging\_the\_Gap\_Enhancing\_LLM\_Performance\_for\_Low-Resource\_African\_Languages\_with\_New\_Benchmarks\_Fine-Tuning\_and\_Cultural\_Adjustments

45. The CARE Principles for Indigenous Data Governance., https\://www\.researchgate.net/figure/The-CARE-Principles-for-Indigenous-Data-Governance\_fig2\_346665037