Best Practices for a Bidirectional English ⇄ ɩ≃1 Converter on Protocol5.com

Full report

On this page

Executive summary

Protocol5 already has the beginnings of the right product surface. The current public site exposes an Iota Protocol section and a converter route, and the converter page already shows a draft interaction model with a direction toggle, database-only / hybrid / LLM-assisted modes, ranked candidates, trace evidence, semantic similarity, category search, and round-trip actions. It also states the core architectural commitments that matter most: the system is intentionally approximate rather than exact, it is public-symbol based rather than a private codebook, it uses the same-server SQL category corpus first, it prefers Category.Categories before Category.Words, it rejects private-use characters, it uses NFC normalization plus System.Text.Rune and grapheme grouping in the logic layer, and it keeps embedding population off the public web host. In other words, the engineering task is not to invent the product from scratch; it is to harden, formalize, and scale a draft that already points in the right direction.

The strongest implementation choice is to treat English ⇄ ɩ≃1 as an evidence-first approximate semantic conversion system, not as a lossless translator. That framing is also consistent with your uploaded internal research, which repeatedly treats IOTA-1 / JustAnIota as an approximate public-symbol semantic layer over assigned Unicode / ISO 10646 characters and standard sequences, rather than a hidden bilingual mapping or private-use encoding. Unicode itself supports this framing: the Unicode Character Database is the public property layer for characters, Unihan is the public enrichment layer for Han ideographs, CLDR annotations provide names and keywords for many emoji and symbols, and UTS #51 defines emoji characters and sequences. None of those standards provide a universal meaning language by themselves, but together they provide enough public evidence to build an inspectable converter.

For Protocol5 specifically, the best operating model is a WordPress UI plus ASP.NET Core facade plus C# logic layer plus SQL-backed symbol atlas, with optional local model inference for embedding generation and reranking. On the front end, use a server-registered WordPress block with the Interactivity API and same-origin REST calls. On the backend, expose a narrow, testable IJustAnIotaConverterFacade over normalization, segmentation, matching, ranking, and provenance generation. In storage, keep canonical symbol/category/word records and embeddings together so that the same system can explain why a candidate was chosen. Where possible, follow the current Protocol5 page’s stated rule: search Category.Categories first, then Category.Words, then ISO 10646 / glyph fallback.

The most important technical decision is the text pipeline. The converter should ingest UTF-8 text, preserve the raw original, normalize to NFC for the working form, iterate over Unicode scalar values with Rune, treat grapheme clusters and emoji sequences as user-visible units, split paragraphs into sentences with Unicode-aware boundary analysis, and then derive semantic segments of roughly one to three words using rule-based segmentation plus optional dependency parsing. That segmentation should preserve fixed expressions, phrasal verbs, named entities, emoji sequences, and negation-bearing mini-phrases before falling back to individual words. Only after those segments are created should the matcher apply the ordered lookup policy: Category.Categories → Category.Words → Category.ISO10646/Glyphs.

The advice on retrieval is equally clear. Use a hybrid ranking stack: exact and alias matches first, lexical fuzzy expansion second, embedding retrieval third, reranking last. For open models, multilingual-e5-large-instruct is the safest balanced recommendation for a first high-quality semantic system; BGE-M3 is the stronger option if you want dense, sparse, and multi-vector retrieval in one model; LaBSE remains a good baseline; and OpenAI text-embedding-3-small is the best managed-service option if cloud use is acceptable. Because SQL Server’s native vector type currently supports at most 1,998 dimensions, text-embedding-3-small fits at its default 1,536 dimensions, but text-embedding-3-large at 3,072 dimensions only fits if you explicitly shorten the output with the dimensions parameter.

Finally, the page should visibly communicate the epistemic status of every result. The converter should always show approximate result, ranked alternatives, confidence, segment trace, source provenance, and round-trip preview. Users should never have to guess whether a given output was an exact mapping, a category-level abstraction, a word fallback, or a glyph-level last resort. That transparency is not just good UX; it is the product’s core credibility mechanism. Protocol5’s current converter draft already points in this direction, so the best next step is to make those ideas systematic and production-grade.

Current state and target operating model

The current Protocol5 home page already positions the site as a “public interface” for exact mathematics, dense symbolic media, and protocol implementation, and it exposes an “Iota Protocol” cluster containing IOTA-1, Converter, Protocol Rules, and Public Symbols. The current converter page then narrows that concept into a more specific claim: English is converted to public-symbol candidates, ɩ≃1 symbols back to English candidates, and English can be run through a round trip; results remain intentionally approximate; and the SQL category corpus is searched first. This means the recommended converter page is best understood as the productionization of an existing Protocol5 demo surface, not a net-new product area.

That matters because the page’s public wording is already unusually good. It explicitly says the system is “best gist, not exact translation,” says private-use areas are prohibited, says the visible symbol inventory must remain public and inspectable, and says the pipeline should build a symbol atlas from UCD, CLDR, Unihan, and emoji data rather than from bare code point numbers. Those are exactly the right operating constraints for a public-facing Unicode-to-meaning experiment and they align with the uploaded internal analyses.

The recommended target architecture is shown below.

flowchart LR
    A[WordPress page or block] --> B[WP REST route]
    B --> C[WP server-side proxy]
    C --> D[ASP.NET Core Facade API]
    D --> E[C# Logic Layer]
    E --> F[(SQL symbol atlas and vectors)]
    E --> G[Optional local embedding / rerank service]
    H[Unicode ingestion worker] --> F
    I[UCD + Unihan + CLDR + emoji data] --> H

The reason for this split is simple. WordPress is good at page composition, same-origin interaction, editorial integration, and low-friction content updates. ASP.NET Core is better for Unicode-safe text processing, structured APIs, dependency injection, typed contracts, health checks, and production observability. WordPress also already recommends block.json as the canonical registration format for blocks, the Interactivity API for richer front-end behavior, register_rest_route() for REST routes, JSON Schema-backed request validation/sanitization, and nonce-backed cookie authentication for same-origin browser calls. On the .NET side, ASP.NET Core provides first-class OpenAPI generation, endpoint filters, ProblemDetails, rate limiting, health checks, and reverse-proxy-friendly hosting.

A good practical rule is this: the browser should talk to WordPress, and WordPress should talk to the C# backend. This avoids unnecessary browser-to-backend CORS complexity, lets the public page use same-origin WordPress cookies and nonces, and makes it easier to insert anti-abuse controls before the conversion engine runs. If WordPress must forward requests to a configurable backend endpoint, use wp_safe_remote_post() rather than a raw outbound request helper, because WordPress explicitly validates URLs and redirects there to reduce SSRF risk.

A concise working division of responsibilities is below.

The comparison above synthesizes the current Protocol5 converter page with official WordPress and ASP.NET Core guidance.

Text normalization, segmentation, and ordered matching

The text pipeline should begin with a strict distinction between raw input, canonical working form, and search keys. Preserve the raw user string exactly as entered for display and auditing. Normalize a working copy to NFC so canonically equivalent strings land on a consistent representation. Only use more aggressive compatibility folding such as NFKC-style search keys for secondary approximate lookup, never as the visible canonical value, because compatibility normalization can collapse distinctions that may still matter for user-facing output. Unicode calls normalization a conformance-critical operation, and the current Protocol5 page already says the logic layer owns NFC normalization.

After normalization, never process text as “one C# char equals one character.” In .NET, Rune represents Unicode scalar values, while StringInfo.GetTextElementEnumerator() iterates text elements; the .NET documentation explicitly notes that rune count is not the same thing as user-perceived display characters. This distinction is essential for supplementary-plane characters, combining marks, complex emoji, and any text containing ZWJ sequences. The current Protocol5 page also already commits the logic layer to System.Text.Rune scalar handling and grapheme grouping, so this is one area where the public design and Unicode best practice completely agree.

Emoji and other public symbol sequences should be treated as atomic matching candidates whenever Unicode says they form standard sequences. UTS #51 defines emoji characters and sequences and provides the structure needed to recognize emoji presentation selectors, skin-tone modifiers, and ZWJ sequences. For converter behavior, that means a family emoji, a gendered occupation emoji, or a skin-tone-modified person emoji should not be split into separate matching atoms in the semantic matcher just because the underlying Unicode representation has multiple code points.

For paragraph and sentence boundaries, the most defensible baseline is Unicode-aware boundary analysis rather than regex-only splitting. UAX #29 defines default grapheme, word, and sentence boundaries, and ICU’s BreakIterator family is specifically designed to locate character, word, line, and sentence boundaries across scripts. In practice, that means paragraph splitting can stay simple, but sentence splitting should use ICU-style boundary rules rather than only punctuation heuristics. That recommendation is also supported by recent retrieval and chunking research, which shows that segmentation quality matters, while expensive semantic chunking does not always justify its cost over simpler boundary-aware methods.

The distinctive requirement in your request is the next layer: breaking each sentence into semantic segments of roughly one to three words for Category.Categories. The best rule set is not “blind 3-grams.” It is a tiered syntactic segmentation strategy:

First, preserve units that should stay intact: emoji sequences, named entities, fixed expressions, phrasal verbs, and tightly bound compounds. Universal Dependencies gives you a portable grammar vocabulary for many of these units, including fixed, flat, compound, and the phrasal-particle form of compound. Stanza can jointly handle tokenization and sentence segmentation and can add dependency parses; OpenNLP supports sentence detection, tokenization, chunking, and parsing if you prefer that ecosystem.

Second, derive short semantic chunks from syntax rather than only proximity. For English, the highest-yield segment classes are short noun phrases, verb + particle constructions, verb + essential object pairs, and adjective/adverb phrases that express stance or state. Keep negation bound to the head when the result changes meaning materially, so “not safe” and “do not enter” are treated as higher-priority segments than their positive heads alone. Where the parser produces larger phrases, clip them to one to three words by preference order: three-word idiom or phrasal unit, then two-word phrase, then single word.

Third, use fallback windows only after grammar-aware extraction. Once idioms, entities, and small phrases are removed, walk the unmatched residue with a longest-match-first window of 3 → 2 → 1 tokens. This directly supports the matching policy you requested: try Category.Categories with the best short semantic spans first, then try Category.Words on leftover words, and only then fall back to glyph-level or ISO 10646 candidate retrieval. That order also matches the current Protocol5 public page, which says the SQL category corpus is searched first and that Category.Categories is preferred before Category.Words.

A concrete grammar policy for the first release is below.

This policy is a synthesis of Unicode segmentation standards, UD syntax conventions, available parser capabilities, and the current Protocol5 converter rules.

A “How are you today?” example shows why this matters. Under the recommended policy, the engine should first attempt the idiomatic three-word segment “how are you” as a greeting-state category, and only then treat “today” as a temporal modifier. If the greeting category is absent, the fallback should be a two-word pair such as “how are” plus the leftover pronoun, not a naive word-by-word search from the outset. That is the kind of grammar-aware segmentation needed to make Category.Categories genuinely useful instead of decorative. This is a design recommendation grounded in the matching order and segmentation principles above.

The library choices should reflect these layers.

The comparison above is based on official documentation and repositories for .NET, ICU, Stanza, OpenNLP, and Microsoft’s tokenizer package.

Ranking, fidelity, and data architecture

The ranking system should be hybrid, not purely vector-only. Dense semantic retrieval is strong, but public-symbol conversion has an unusually heavy need for lexical and structural evidence, because categories and words are short, curated, and often alias-rich. The best pipeline is therefore: exact/alias candidate generation, then fuzzy lexical expansion, then embedding retrieval, then optional reranking. That matches both recent retrieval results and the BGE-M3 authors’ own recommendation to use hybrid retrieval plus reranking. BEIR also found that BM25 remains a robust baseline and that reranking / late interaction can outperform simple dense retrieval, albeit at higher compute cost.

For open models, there are three especially plausible choices. multilingual-e5-large-instruct is the safest “balanced” semantic engine for a first production-grade release because it is multilingual, available under MIT, has 1,024-dimensional embeddings, supports ONNX export, and is explicitly trained to use query instructions. BGE-M3 is more ambitious: it supports dense, sparse, and multi-vector retrieval, works across 100+ languages, and supports long inputs up to 8,192 tokens, which makes it especially attractive if you want one model to power both semantic similarity and lexical-like weighting. LaBSE remains a strong stable baseline for a shared multilingual vector space. If you want a managed cloud option, text-embedding-3-small is cost-effective and fits SQL Server’s 1,998-dimension cap at its default 1,536 dimensions; by contrast, text-embedding-3-large defaults to 3,072 dimensions and therefore must be dimension-reduced before storage in SQL Server.

A practical scoring policy is below.

final_score =
  0.45 * semantic_similarity
+ 0.20 * lexical_or_alias_match
+ 0.15 * category_priority
+ 0.10 * structural_fit
+ 0.10 * round_trip_consistency

This weighting is not a standard; it is a recommended starting point. The essential idea is that semantics should dominate, but not erase the product’s explicit rule hierarchy. If a segment has a strong Category.Categories match and a weaker Category.Words match, the category candidate should usually win unless the semantic gap is large. Confidence should be calibrated, not guessed, using held-out evaluation sets and calibration diagnostics such as reliability diagrams and Brier-style probability assessment.

The provenance model should be first-class. Every returned candidate should include at least: matched segment, match level (Category, Word, Glyph), semantic score, lexical score, supporting descriptor sources (for example UCD name, CLDR annotation, Unihan kDefinition or variant relation), model version, vector dimension, index mode (exact, ANN, reranked), and round-trip back-translation. That is the difference between a transparent converter and an inscrutable symbol generator. Protocol5’s current converter page already points in this direction with ranked candidates, trace evidence, approximate result labeling, mode indicators, and semantic comparison panes.

Round-trip fidelity must also be framed honestly. The goal is not exact bijection; it is controlled, explainable lossiness. A good UX and evaluation scheme should classify outputs into stable, acceptable paraphrase, lossy but useful, and failed. The best metric bundle is: retrieval quality metrics such as Recall@k, MRR, and nDCG for candidate ranking; semantic fidelity metrics such as BERTScore or embedding-space similarity for English round-trips; and confidence-calibration metrics for the displayed confidence estimate. MTEB and BEIR are useful references for retrieval-style embedding evaluation, while BERTScore is a good fit for paraphrase-sensitive round-trip checking.

For persistence, the current Protocol5 page’s instincts are right: keep the symbol atlas in SQL and store both semantic records and vectors together. A good relational design is:

This schema keeps the match hierarchy explicit and lets you explain every result in user-facing provenance. The current public Protocol5 page already describes a symbol-atlas build from UCD, CLDR, Unihan, and emoji data, vector persistence with model and provenance metadata, and the ability to return mode, scores, ranked candidates, and provenance.

Because Protocol5’s public page already assumes SQL Server, the default recommendation is to stay with SQL Server as the first persistence layer, but to be realistic about present limitations. SQL Server 2025’s vector type stores vectors in optimized binary form, supports up to 1,998 dimensions, supports float16 in preview, exposes native SqlVector<T> support in Microsoft.Data.SqlClient, supports exact nearest-neighbor search through VECTOR_DISTANCE, and offers preview ANN support with VECTOR_SEARCH and CREATE VECTOR INDEX. Microsoft explicitly recommends exact search when the filtered candidate set is roughly below 50,000 vectors, and its ANN index is preview-only with several versioning caveats. That implies a conservative first release should rely on filtered exact search first, then adopt native ANN only if you accept preview risk, or use a dedicated vector engine if your scale grows faster than SQL’s preview feature maturity.

If you stay with SQL Server, there is one especially important operational nuance: deduplicate embeddings before indexing, and plan to rebuild or refresh indexes after large re-embedding campaigns. Microsoft’s vector-index guidance says duplicates hurt result quality and that large-scale embedding replacement can degrade recall and ranking even if queries still return valid results.

A candidate comparison for embeddings and storage is below.

The comparison above synthesizes official model cards, OpenAI documentation, Microsoft SQL documentation, and official project repositories or docs for the vector backends.

API, UX, security, and operations

The public API contract should be narrow and explicit. The minimal surface should be a facade with four routes: convert, round-trip, compare, and status/health. Use ASP.NET Core OpenAPI to document the routes, endpoint filters for validation/logging shortcuts, and ProblemDetails for machine-readable errors. Because the current converter page is already built around conversion, round-trip, comparison, status, and category search concepts, these endpoint families map cleanly to the user-facing design.

A recommended route set is:

An example convert request shape is:

{
  "input": "How are you today?",
  "direction": "english_to_iota",
  "mode": "database_only",
  "maxCandidates": 5,
  "includeTrace": true,
  "includeRoundTripPreview": true
}

And a corresponding response contract should include:

{
  "output": "…",
  "approximate": true,
  "segments": [
    {
      "text": "how are you",
      "matchLevel": "Category",
      "confidence": 0.83,
      "provenance": ["CLDR", "UCD", "Category.Categories"]
    },
    {
      "text": "today",
      "matchLevel": "Word",
      "confidence": 0.74,
      "provenance": ["Category.Words"]
    }
  ],
  "candidates": [],
  "roundTripPreview": {
    "english": "How are you today?",
    "lossiness": "acceptable_paraphrase"
  }
}

Those route and payload shapes are recommendations, but they are designed to fit ASP.NET Core’s OpenAPI / ProblemDetails model and the current Protocol5 UI concepts.

The UX should make approximation legible. The current converter page already has the right skeleton: input area, direction selector, mode selector, result pane, ranked candidates, trace evidence, semantic similarity, category search, and round-trip action. The production page should keep those elements but add three things that materially improve trust: a segment strip that shows how the sentence was broken into 1–3 word semantic units, a confidence/provenance drawer that expands per segment, and a round-trip diff view that highlights what stayed stable versus what drifted. For this product, “show your work” is not optional; it is the interface.

For the WordPress implementation, the most maintainable UI approach is a server-registered custom block using block.json and the Interactivity API, backed by a WordPress REST route. The front-end can render a large textarea, a direction toggle, a mode picker, and an expandable evidence panel; the actual conversion call should go to a WordPress REST route registered on rest_api_init, which in turn calls the ASP.NET backend. The WordPress route should define a request schema so WordPress can validate and sanitize structured request fields. For plugin settings or admin textareas, sanitize_textarea_field() is appropriate; for actual conversion inputs, prefer validation plus output escaping over destructive sanitization, because over-sanitizing can damage the very Unicode content the converter exists to process.

Security should follow the text and data model instead of being bolted on later. On the WordPress edge, use cookie authentication plus nonces for same-origin browser interactions. On the WordPress-to-ASP.NET hop, use a server credential, HMAC, or JWT bearer token; ASP.NET Core’s JWT bearer middleware is the cleanest standard option if you want explicit token validation. On the .NET side, add rate limiting immediately, return structured ProblemDetails, and keep health endpoints separate from public conversion endpoints. For arbitrary Unicode input, also run mixed-script/confusable checks based on Unicode security guidance so the system can warn on suspicious symbol combinations even if it still processes them.

Privacy and logging deserve separate mention. OWASP ASVS says logs should not include sensitive personal data unless legally required and should themselves be treated as high-value assets. That matters here because a public converter will inevitably process arbitrary free text. The safest default is to log request hashes, timing, selected mode, corpus/matcher path, and aggregate score distributions, but not full raw user text unless the user explicitly opts into diagnostic sharing. If you need short-term debugging, keep any raw-request logging behind a temporary elevated diagnostic flag with automatic expiry.

Deployment should stay conservative. Host the ASP.NET Core service behind a reverse proxy, containerize it for reproducibility, and expose health checks for SQL connectivity, corpus readiness, and optional local model readiness. Microsoft’s hosting guidance supports Dockerized ASP.NET Core and reverse-proxy deployments, and health checks are specifically intended for load balancers and orchestrators. The current Protocol5 converter page also explicitly says population should happen in local tooling rather than on public mutation routes, which is the right operational boundary: build or refresh embeddings off-host, publish read-only artifacts to the public service, and keep the public web surface strictly read-mostly.

One final deployment nuance is local model integration. LM Studio is convenient because it exposes OpenAI-compatible /v1/embeddings and related endpoints on a local server, but SQL Server’s CREATE EXTERNAL MODEL currently requires HTTPS with TLS for supported inference endpoints. Since LM Studio’s standard local development pattern is http://localhost:1234/v1, the cleanest default is C# logic layer → LM Studio, not SQL Server → LM Studio, unless you add an internal TLS-terminating gateway or move to an ONNX-based local inference path that SQL Server can call differently.

Roadmap, testing, and implementation starting point

The rollout should be phased, because the hardest problems are not the textarea or route plumbing; they are symbol-atlas curation, segmentation quality, confidence calibration, and round-trip evaluation. A practical phased plan looks like this:

gantt
    title Protocol5 English ⇄ ɩ≃1 phased rollout
    dateFormat  YYYY-MM-DD
    section Foundation
    WordPress block and same-origin REST shell     :a1, 2026-05-12, 14d
    ASP.NET facade, contracts, ProblemDetails      :a2, 2026-05-12, 21d
    Unicode normalization and boundary pipeline    :a3, 2026-05-19, 21d
    section Corpus and matching
    Category / Word / Glyph schema and importers   :b1, 2026-06-02, 21d
    UCD + CLDR + Unihan + emoji provenance build   :b2, 2026-06-02, 28d
    Rule-based categories-first matcher            :b3, 2026-06-09, 21d
    section Retrieval and scoring
    Embedding pipeline and hybrid ranking          :c1, 2026-06-23, 28d
    Confidence calibration and round-trip scorer   :c2, 2026-07-07, 21d
    section Hardening
    Security review, abuse controls, confusables   :d1, 2026-07-21, 14d
    Load testing, user testing, rollout            :d2, 2026-07-28, 21d

The highest-priority implementation steps are these:

1. Freeze the product contract. Decide that every response must return approximate status, segment trace, ranked candidates, confidence, provenance, and the actual matching path used.
2. Build the Unicode-safe text layer first. Raw preservation, NFC working form, rune iteration, grapheme grouping, sentence segmentation, and semantic-segment extraction should ship before any serious embedding work.
3. Stand up the SQL corpus in plain relational form before ANN optimization. Categories, words, glyphs, aliases, descriptor evidence, and embeddings first; performance tuning second.
4. Implement the ordered matcher exactly as policy, not as an emergent side effect. Categories first, then words, then glyphs, with longest semantic segments before shorter ones.
5. Add hybrid semantic ranking only after deterministic rules work. Start with exact and lexical candidate generation; then add embeddings; then optional reranking.
6. Ship round-trip preview and trace evidence in the first public release. Without them, the page will be much harder to trust and debug.

Those are product recommendations, but they are tightly grounded in the current Protocol5 page’s design and the supporting platform guidance above.

A minimal façade interface can look like this:

public interface IJustAnIotaConverterFacade
{
    Task<ConvertResponse> ConvertAsync(ConvertRequest request, CancellationToken ct = default);
    Task<RoundTripResponse> RoundTripAsync(RoundTripRequest request, CancellationToken ct = default);
    Task<CompareResponse> CompareAsync(CompareRequest request, CancellationToken ct = default);
    Task<ServiceStatusResponse> StatusAsync(CancellationToken ct = default);
}

And the core flow should look like this:

public async Task<ConvertResponse> ConvertAsync(ConvertRequest request, CancellationToken ct)
{
    string raw = request.Input;
    string nfc = raw.Normalize(NormalizationForm.FormC);

    var paragraphs = ParagraphSplitter.Split(nfc);
    var sentenceUnits = paragraphs.SelectMany(p => _sentenceSegmenter.Split(p));

    var results = new List<SegmentResult>();

    foreach (var sentence in sentenceUnits)
    {
        // Protect user-visible atoms first.
        var atoms = _graphemeAndEmojiProtector.Protect(sentence);

        // Build semantic segments: idioms/entities/phrasal verbs/noun phrases first.
        var semanticSegments = await _semanticSegmenter.ExtractAsync(atoms, ct);

        foreach (var segment in semanticSegments)
        {
            CandidateSet candidates =
                await _categoryMatcher.MatchAsync(segment, ct) // Category.Categories
                ?? await _wordMatcher.MatchAsync(segment, ct)  // Category.Words
                ?? await _glyphMatcher.MatchAsync(segment, ct); // Category.ISO10646/Glyphs

            var ranked = await _ranker.ScoreAsync(segment, candidates, ct);
            results.Add(ranked.Best);
        }
    }

    var output = _renderer.Render(results, request.Direction);
    var roundTrip = request.IncludeRoundTripPreview
        ? await _roundTripAnalyzer.PreviewAsync(raw, output, ct)
        : null;

    return new ConvertResponse
    {
        Input = raw,
        Output = output,
        Approximate = true,
        Segments = results,
        RoundTripPreview = roundTrip
    };
}

This code is illustrative, but it encodes the actual architectural policy recommended in this report and reflects the current Protocol5 rule structure.

If you choose an open embedding model and want to run it from C# without a Python sidecar, ONNX Runtime is the cleanest path. Microsoft provides a C# API for ONNX Runtime, and Sentence Transformers’ own documentation notes that if you export to ONNX outside of the Sentence Transformers runtime, you must apply pooling and normalization yourself to turn token embeddings into sentence embeddings. That detail is easy to miss and is one of the strongest arguments for wrapping model inference behind a dedicated adapter instead of letting application code call raw ONNX sessions everywhere.

Testing should cover correctness, retrieval quality, calibration, and user comprehension. A rigorous matrix is below.

The metric bundle above is grounded in standard retrieval and semantic-generation evaluation literature and calibration guidance.

A focused first-release test set should include cases like these:

Open questions and limitations

A few design choices still require product decisions rather than pure engineering. The largest is the exact definition of Category.Categories: whether it is primarily a curated short-phrase ontology, a phrase-frequency lexicon, or a concept taxonomy with alias layers. The right segmentation and ranking rules depend on that answer. The current public page and your uploaded materials strongly imply a curated concept layer, but they do not fully specify its editorial governance.

There is also an infrastructure choice to settle early: whether Protocol5 wants to remain with SQL Server’s native vector stack even while ANN features are still partly preview, or whether it wants a dedicated vector backend for the semantic stage while keeping SQL as the system of record. The current public page clearly favors SQL-first and that is architecturally coherent, but the preview status of parts of the ANN surface means there is a real production tradeoff between platform simplicity and search maturity.

Finally, some of the uploaded internal files were only available here as partial snippets rather than fully queryable documents, so the recommendations above align with the visible internal design direction but may not reflect every lower-level implementation decision already made in your private drafts. Where that happened, this report favors the highest-confidence overlap between the current public Protocol5 converter page, the available internal research snippets, official platform documentation, and recent primary research.