Protocol5 Research Report on a Public Unicode-to-Meaning Embedding System

Full report

On this page

Core judgment

Your idea is technically viable as a semantic retrieval experiment, not as a literal character-by-character “translation” system. The strongest version of the idea is this: treat ISO/IEC 10646 and Unicode as the public symbol substrate, derive public semantic descriptors for those symbols from Unicode data sources, embed those descriptors and English terms into a shared vector space, and then use similarity search to retrieve approximate conceptual neighbors rather than exact conversions. That makes the project well-suited to Protocol5’s publicly experimental positioning, which already presents itself as a parent platform for exact mathematics, machine-publication systems, and machine-readable route contracts rather than a black-box consumer product.

The key qualification is that ISO/IEC 10646 is not itself a semantic ontology. Unicode and ISO/IEC 10646 are synchronized at the level of character codes and encoding forms, but Unicode adds the functional character specifications, character data, and algorithms that implementations actually rely on. For Han ideographs in particular, the Unicode Consortium states that the standard does not formally define “what the ideograph is” semantically; it defines ideographs via mappings and then supplements them with ancillary data in the Unihan database. That means your system can be language-neutral in the sense of shared representation and approximate retrieval, but it cannot honestly claim that the code points themselves are already a complete universal meaning language.

That distinction matters because it leads to the right architecture. The project should not be “raw code point to meaning.” It should be public Unicode symbol metadata to embedding to similarity search, optionally followed by an LLM-assisted reranker or verbalizer. That preserves your “no secret dictionary” principle while still giving the system enough semantic evidence to work.

What ISO and Unicode give you

ISO/IEC 10646 and Unicode give you a globally standardized and synchronized encoding surface for text. Unicode explicitly describes itself as the universal character encoding standard for written characters and text, and the Unicode FAQ states that Unicode and ISO/IEC 10646 remain synchronized as they expand. That makes the standard a sound foundation for a public experiment because it avoids proprietary code pages and hidden symbol agreements.

For general characters, Unicode’s Character Database gives you formal names and properties such as general category, normalization behavior, mappings, scripts, and other attributes. Unicode explicitly says the UCD is an integral part of the standard and catalogs the semantics needed for interoperability and correct behavior in implementations. Those properties are precisely the sort of public evidence you need to build open descriptors for a symbol catalog.

For emoji, CLDR provides locale-specific names and keywords, and the Unicode emoji specification defines emoji characters and sequences, their presentation behavior, modifier handling, and interoperability guidance. That is especially useful for your experiment because emoji are some of the most overtly concept-bearing public Unicode symbols, and CLDR gives you multilingual keyword evidence without creating a private mapping layer.

For Han ideographs, Unihan is the crucial resource. It contains readings, dictionary-like data, semantic-variant relationships, radical-stroke information, and in many cases an English definition through kDefinition. Unicode also notes that fuzzy matching and relationships between ideographs require ancillary data beyond the bare encoding. In practice, that means ideographs can contribute conceptual signal, but only when you use the supporting Unihan fields rather than pretending the bare scalar value is self-defining.

What ISO and Unicode do not give you is a ready-made interlingua where every assigned code point carries directly comparable concept weights. Many code points are orthographic, structural, or formatting elements rather than concept-bearing symbols, and script metadata itself is designed for text processing, not for turning the entire standard into a universal semantic language. Unicode’s Script and Script_Extensions properties are useful for classification, but the standard also warns that characters with Common or Inherited behavior and out-of-context usage require careful handling.

What makes the experiment feasible

The strongest evidence in favor of your concept comes from modern multilingual embedding research. LaBSE demonstrated high-quality language-agnostic sentence embeddings across 100+ languages and reported strong bi-text retrieval performance across 112 languages. SONAR extends the idea further into a fixed-size multilingual and multimodal sentence embedding space covering 200 languages, outperforming earlier multilingual sentence embeddings on multilingual similarity search tasks. Multilingual E5 and M3-Embedding likewise show that open multilingual embedding models can support semantic retrieval across more than 100 languages and across different granularities of text.

That does not mean the model magically understands every Unicode code point as an atomic concept. What it means is that a shared embedding space can be used for approximate semantic proximity, which is exactly the regime you described: not 1 + 1 = 2, but “this symbol cluster is perhaps nearest to these English ideas.” The Large Concept Models work is also relevant philosophically: it explicitly explores modeling in a sentence representation space where concepts are treated as higher-level language-agnostic units, using SONAR as the underlying space. Even if you never adopt that architecture directly, it supports the legitimacy of a concept-first rather than token-first experiment.

Another useful result for your “no AI at runtime” preference is that simple embedding composition can still be meaningful. The “Simple but Tough-to-Beat Baseline” paper showed that weighted averages of word embeddings can be a strong unsupervised sentence representation baseline, especially when labeled data is scarce. That supports a database-only gist mode in which you precompute lexeme and symbol vectors, then compose a query vector from stored parts instead of calling a model live for every request. It will be weaker than an active embedding model, but it is a defensible fallback mode for approximate retrieval.

There is also emerging evidence that semantics are not reducible to a special “magic dictionary” at the embedding layer. A recent TMLR paper showed that transformer models with frozen visual Unicode-based embeddings can still learn useful high-level semantics, arguing that semantics are an emergent property of model composition and data rather than something stored only in the input embedding matrix. That does not prove your exact design, but it does support your instinct that there is value in operating below traditional word-token boundaries.

Where the idea breaks if implemented too literally

The first failure mode is treating code points as the wrong unit. Unicode text segmentation rules define extended grapheme clusters as the default “user-perceived characters,” and Unicode emoji are often multi-code-point sequences rather than single scalars. If you assign embeddings only to single code points, you will break many emoji, combining-mark sequences, and other real text units. For this project, the meaningful atomic unit is often a grapheme cluster or named emoji sequence, not a scalar value.

The second failure mode is assuming all public Unicode symbols are concept-bearing. Some are. Many are not. Controls, format characters, variation selectors, combining marks, and many ordinary script letters mainly serve orthographic or rendering functions. Han ideographs and emoji can carry conceptual signal, but even there the signal is uneven and context-sensitive. The right move is not “embed all Unicode equally,” but rather downweight or exclude low-semantic categories and enrich the higher-value subsets with public metadata.

The third failure mode is private-use characters. You specifically want to avoid a “versioned private-use profile,” and the Unicode Standard strongly supports that instinct. Unicode states that private-use characters have interpretations determined by private agreement, can conflict across systems, and have no standard-defined interpretation. In other words, a private-use profile would directly undermine your goal of building a public, language-neutral experiment.

The fourth failure mode is pretending the system can translate without any semantic evidence layer. You can avoid a hidden proprietary dictionary, but you cannot avoid some public evidence source. For emoji that source can be CLDR annotations. For Han it can be Unihan definitions, readings, variants, and dictionary-like fields. For general Unicode symbols it can be UCD names and properties. For English it can be a public lexicon or a corpus-derived embedding inventory. The right standard is therefore not “no dictionary at all,” but “no secret closed mapping.”

The fifth failure mode is security and trust. Unicode explicitly documents mixed-script and whole-script confusables, and those issues become more important when users can submit arbitrary symbols as semantic queries. A public Protocol5 demo should normalize inputs, filter dangerous categories, and detect confusables before any retrieval step.

Recommended architecture for the JustAnIota Converter

The architecture that best fits your brief is a facade-centered ASP.NET Core system with a strict separation between ingestion, retrieval, and optional AI assistance. The public API surface should be thin and stable, while the experiment logic remains replaceable behind interfaces. This is consistent with Protocol5’s current public stance of exposing machine-readable routes, package mirrors, and explicit authority boundaries rather than burying behavior in opaque pages.

A good enterprise shape is shown below.

This design is not just stylistic. SQL Server 2025 now has first-class vector support, exact and approximate vector search, AI model registration, and ADO.NET vector transport through SqlVector<T>. Microsoft’s docs also show that .NET applications can remain loosely coupled through abstractions such as IEmbeddingGenerator, which is useful even if you keep ADO.NET as the canonical persistence path.

The most important interface is the facade. Other Protocol5 projects should never need to know whether a result came from a live embedding model, a precomputed centroid, a database-only nearest-neighbor graph, or an LLM reranker. A minimal facade surface can expose four families of operations: convert English to IOTA-1 candidates, convert IOTA-1 or raw Unicode back to English candidates, explain why a result was returned, and perform round-trip testing. That gives you clean testability and keeps the experiment consumable by sibling demos or future UAIX-related tooling. The supporting evidence for this modularity comes from Microsoft’s AI abstractions and SQL’s native vector access, but the actual interface contract is the architectural synthesis I would recommend here.

Data model and retrieval pipeline

The most important design choice is to store descriptors, not just symbols. Each row in your Unicode-side catalog should represent either a single concept-bearing character or an approved grapheme sequence, plus a canonical public descriptor assembled from Unicode sources. For a Han ideograph, that descriptor can include the code point, character name, kDefinition, selected readings, semantic variants, radical/stroke data, and script metadata. For emoji, it can include CLDR short names and keywords. For general symbols, it can include the official Unicode name, block, category, and script properties. The embedding is computed from that descriptor text, not from a naked scalar value.

A practical SQL schema would include at least UnicodeSymbol, EnglishTerm, MeaningEdge, QueryCache, and ExperimentRun. The first two store public descriptors and vectors. MeaningEdge stores precomputed nearest-neighbor relations between the two spaces. QueryCache stores common composed vectors for no-AI mode. ExperimentRun stores reproducibility metadata such as Unicode version, CLDR version, embedding model name, dimensions, and scoring settings. Versioning matters because Unicode properties and script-extension assignments can change over time, and the standard explicitly warns implementers to expect such changes as data improves.

SQL Server 2025 is a strong fit for this part of the system. Microsoft documents a native vector type, exact similarity with VECTOR_DISTANCE, approximate search with VECTOR_SEARCH, and DiskANN-based vector indexes. The docs also note that exact distance calculations ignore vector indexes and that approximate search is the scalable option when you need ANN behavior. Exact search is still recommended for smaller candidate sets; Microsoft gives a rule-of-thumb that exact search is appropriate when fewer than roughly 50,000 vectors must actually be compared.

That leads to a useful dual-mode design:

The assisted mode should embed the user’s English text or Unicode descriptor on demand using a local multilingual embedding model, then search the opposite catalog. The database gist mode should normalize and segment the input, look up stored vectors for known lexemes or graphemes, and compose a weighted centroid using a simple unsupervised scheme. The persistence model supports both. The research literature supports multilingual retrieval in a shared space and also supports weighted-average composition as a meaningful no-label baseline.

For ADO.NET specifically, Microsoft now documents native use of vectors through Microsoft.Data.SqlClient, including the SqlVector<float> type, optimized binary transport, stored procedures, and bulk copy. That makes a repository-centered implementation entirely reasonable even if you later add EF Core or higher-level abstractions elsewhere.

SQL Server and LM Studio integration

SQL Server 2025 gives you two different AI integration stories. The first is the database-native path: CREATE EXTERNAL MODEL, AI_GENERATE_EMBEDDINGS, and vector functions. The second is the application-managed path: call an embedding service from .NET, then store the result in SQL. Both are valid, but for your specific stack I would make the C# service the primary LM Studio client and use SQL as the vector store and search engine.

The reason is subtle but important. Microsoft’s CREATE EXTERNAL MODEL documentation requires HTTPS/TLS for remote inference endpoints and lists accepted embedding endpoint formats as Azure OpenAI, OpenAI, Ollama, and ONNX Runtime. LM Studio does expose OpenAI-compatible /v1/embeddings endpoints and its own native /api/v1/* API, but its docs and examples are centered on localhost HTTP usage with optional API-token auth rather than on a database-facing HTTPS deployment. Inference from those facts suggests that direct SQL-to-LM-Studio integration is possible only if you put LM Studio behind a local HTTPS reverse proxy and make it behave as an OpenAI-compatible endpoint. Without that extra plumbing, the cleanest architecture is: LM Studio behind the app, SQL behind ADO.NET.

If you want embeddings generated inside SQL itself, Microsoft’s ONNX Runtime path is the stronger fit. The SQL docs explicitly show local ONNX embedding generation with API_FORMAT = 'ONNX Runtime', a local runtime path, and AI_GENERATE_EMBEDDINGS, though Microsoft also warns that third-party models and libraries are your responsibility from a security and compliance standpoint. That is an attractive option if you want a fully self-contained Windows SQL box for batch ingestion.

For LM Studio model choice, the safest operational guidance is to use a multilingual embedding model for embedding generation and reserve a separate local instruct model for explanation or reranking. LM Studio’s own docs show a dedicated embeddings endpoint and model-loading workflow, and its model catalog includes EmbeddingGemma, which is described as a lightweight open embedding model trained on 100+ spoken languages and intended for retrieval, classification, clustering, and semantic similarity search. That is not the only possible model, but it is a concrete, current, local example that fits this experiment far better than English-only embeddings.

Risks, evaluation, and open questions

The main scientific risk is overclaiming universality. Cross-lingual embedding spaces can support approximate retrieval, but they do not make every Unicode symbol into a stable, context-free concept. The system should be marketed exactly as you described it in the strongest honest wording: a mathematical and semantic approximation demo for “gist” and conceptual neighborhood, not a canonical translation system. That framing is consistent with both the research literature and Protocol5’s current experimental character.

The main engineering risk is noise from weak symbols. The cure is aggressive curation and weighting: use grapheme clusters rather than scalars where appropriate, exclude private-use characters and most non-semantic categories, normalize inputs, detect confusables, and favor symbols backed by richer public metadata. Unicode normalization, segmentation, emoji-sequence structure, and confusable detection are all explicitly documented and should be part of the ingestion and query pipeline from the first prototype.

The main product risk is query-time disappointment in database-only mode. A no-AI runtime can work only insofar as the database already contains enough lexeme and symbol coverage to compose a useful vector from known pieces. That mode is best treated as a deterministic fallback and benchmarking mode, not the flagship experience. The flagship experience should use live local embeddings and optionally a local reranker, while clearly exposing the evidence trail so the demo feels inspectable rather than mystical.

The most important evaluation metrics should be semantic, not translational: top-k concept hit rate, mean reciprocal rank, round-trip drift, human-rated gist quality, and latency for exact versus ANN search. BLEU-style exact translation metrics are a poor fit for a system whose declared purpose is approximation. SQL Server’s exact and approximate search support makes it straightforward to compare precision and performance experimentally at the database layer.

The main open questions are narrower and tactical. I could not line-by-line inspect your uploaded markdown drafts in this environment, so this report synthesizes your prompt with public technical sources rather than reconciling every detail of those internal notes. The remaining design decisions you will still need to lock down are the public symbol profile for “IOTA-1,” the exact embedding model and dimensionality you want to standardize on, whether LM Studio remains application-side or is reverse-proxied for direct SQL use, and how much of the Unicode repertoire you want to expose publicly versus keep as background evidence only. Those choices do not change the high-confidence conclusion: the project is feasible if you treat Unicode as a public symbolic substrate plus metadata layer, not as a pre-existing universal meaning language.