Protocol5 English and Iota-1 Conversion Page Research Report

Full report

On this page

Executive summary

As of May 5, 2026, the top-level Protocol5 site exposes a generic “Converter” surface in its homepage navigation, but the clicked converter path returned a 404 during this research. A separate, live Protocol5 page exists at the JustAnIota converter route under the .NET hub, but that page explicitly describes itself as an approximate bidirectional semantic converter that reports the “best gist,” not exact translation, and it is framed as an implementation experiment rather than a normative UAI standard. That means the current public Protocol5 surface does not yet provide the kind of stable, user-facing English ⇄ Iota-1 page the request calls for.

The strongest public design guidance comes from JustAnIota. Its public records consistently say that semantic meaning should come from registries, schemas, canonicalization, examples, and validator behavior, not from Unicode code points, glyph appearance, embeddings, hashes, or screenshots. Its converter, concept bridge, registry explorer, examples, and validator together already outline a robust product pattern: natural-language source text, locale and normalization metadata, registry-backed candidate matching, an inspectable envelope, warnings, and reverse English glosses.

For the specific page requested on Protocol5, the best-fit implementation is a rules-first, deterministic, segment-first converter. In practical terms, that means: normalize text, split into sentences, extract clause/segment candidates, compare short semantic spans against Category.Categories before falling back to Category.Words, then map matched concepts to code points or visible tokens, with a canonical JSON envelope and a validation report. Fuzzy and embedding-based retrieval should be available only as an optional assistive layer, not as the authority of record, because the public JustAnIota records explicitly reject embeddings and glyph appearance as semantic authority. An uploaded integration note aligns with this direction and explicitly asks for an English ⇄ Iota-1 page plus segment-first matching before word-level matching.

The main blockers are specification gaps, not UI engineering. The public materials describe converter stages, registry requirements, validation behavior, and example fixtures, but they do not publish a complete authoritative English ↔ Iota-1 lexical ontology, a full Category.Categories / Category.Words / Category.ISO10646 registry export, or a final, frozen code-point policy. There is also a real policy tension between the Protocol5 .NET converter page, which prohibits private-use characters, and the JustAnIota public spec/tooling, which allows PUA preview forms in controlled profiles. Those unresolved points should be settled before the Protocol5 page goes live.

Findings from prioritized sites

The user asked that research begin with protocol5.com and justaniota.com, and that all prioritized sites be listed in the report. Those two prioritized domains are:

Within those domains, the most decision-relevant pages were these:

Two especially important inconsistencies surfaced.

First, the code-point policy is not yet unified. The Protocol5 .NET converter page says public symbols only and prohibits PUA. By contrast, the JustAnIota IOTA-1 profile and tooling explicitly discuss demo PUA or ASCII-safe compact candidates, controlled PUA mapping fields, and Plane 15 / Plane 16 demo namespaces. These are not compatible release positions; Protocol5 should pick one public policy and document it clearly.

Second, the envelope shape is still draft-like. One public example uses fields like profile, uai_version, locale, direction, normalization, registry, and payload. Another uses spec_version, uai_target, source_language, source_direction, registry_digest, mode, and units. That strongly suggests a still-evolving draft family rather than a single frozen transport contract. Protocol5 should ship one canonical envelope schema, version it, and keep older drafts explicitly deprecated.

Functional requirements and page design

The conversion page should be a dedicated Protocol5 route such as /iota-1/converter or /tools/iota-1, linked from the homepage and not hidden behind the .NET hub. The page should follow the JustAnIota tool pattern closely: human-readable source text, locale/normalization controls, visible candidate output, canonical envelope JSON, reverse gloss, ranked candidates, trace evidence, and validation warnings. The existing JustAnIota public workbench model also supports a browser path with REST-service fallback, which is a good deployment pattern for Protocol5.

A good public page should satisfy these minimum requirements:

This requirements set is supported by the public converter stages, concept bridge, validator, examples, and registry requirements already published by JustAnIota, plus the existing Protocol5 experimental converter surface.

The page should also separate semantic authority from presentation. The human should always be able to inspect the underlying concept IDs and registry records behind a visible output string, because the JustAnIota public materials repeatedly state that Unicode and glyph appearance are not the semantic promise; the registry and validation system are.

flowchart TD
    A[User input] --> B[Normalize text]
    B --> C[Sentence split]
    C --> D[Clause and segment extraction]
    D --> E[Generate 1 to 3 token category spans]
    E --> F[Match Category.Categories first]
    F -->|matched| G[Map concept IDs to code points or visible tokens]
    F -->|not matched| H[Fallback to Category.Words]
    H -->|matched| G
    H -->|not matched| I[Mark unresolved or literal passthrough]
    G --> J[Assemble canonical envelope]
    I --> J
    J --> K[Validate]
    K --> L[Visible Iota-1]
    K --> M[Decoded English gloss]
    K --> N[Ranked candidates and trace]

The public JustAnIota converter, validator, and concept bridge already implement most of the stages in this diagram in prototype form; the main Protocol5 task is to package them into a stable public page with stricter deterministic behavior.

For UI/UX, the best layout is a two-column workbench with a lower evidence panel: left side for input and controls, right side for outputs, and a full-width panel below for trace/validator details. That mirrors the public “analysis / visible / envelope / gloss / warnings” structure users can already infer from the JustAnIota tool pages.

A practical example helps. For the user phrase “How are you today?”, the page should show the following, clearly labeled as illustrative unless the registry makes it exact:

That caution is necessary because the public JustAnIota examples are registry-versioned fixtures rather than universal equivalence proofs, and the public registry surface remains a demo snapshot rather than a published full lexicon.

Error handling should mirror the public validator posture. In concrete terms, Protocol5 should reject malformed JSON, invalid code-point syntax, unpaired surrogates, unsupported code points, and malformed glyph sequences; warn on PUA, bidi controls, variation selectors, and invisible controls; and distinguish between “valid but unresolved,” “valid and approximate,” and “invalid.” If the page exports JSON, it should use UTF-8 and stay inside the I-JSON constraints; if canonicalized for hashing or signatures, deterministic property sorting is appropriate.

Grammar and segmentation rules

For English input, the segmentation pipeline should be layered rather than flat: normalization → sentence splitting → clause extraction → short semantic segment generation → matching. That is consistent with the public JustAnIota focus on normalization, tokenization, registry candidates, ambiguity review, and export readiness, and it also fits the uploaded integration note’s request to break paragraphs into sentences and sentences into segments before comparing Category.Categories.

Normalization should default to NFC for public input, because the public JustAnIota tools surface NFC and NFKC review modes, and Unicode normalization gives equivalent strings a consistent binary representation. At the same time, Unicode text segmentation rules are specified in terms of normalized text and are designed to respect canonical equivalence without requiring every implementation to physically rewrite all input first.

Sentence splitting should start with a Unicode-conformant boundary engine, not ad hoc regexes. Unicode UAX #29 defines default grapheme, word, and sentence boundaries and explicitly allows tailoring for language or environment. It also notes that default sentence rules will not catch all abbreviation cases such as “Mr. Jones,” which means English production behavior should add abbreviation and honorific tailoring on top of UAX #29. In the browser, Intl.Segmenter now provides locale-sensitive grapheme, word, and sentence segmentation across current browsers; ICU BreakIterator is the server-side analogue and follows UAX #29-style boundary analysis.

Clause extraction should be parser-driven. Universal Dependencies gives a clean operational vocabulary for exactly the structures you care about here: conj / cc for coordination, case for English prepositions in nominal projections, mark for subordinate-clause markers, advcl for adverbial clauses, ccomp and xcomp for clausal complements, acl for clauses modifying nouns, and punct for punctuation. spaCy’s parser and sentence utilities, and Stanza’s tokenization, parsing, and NER pipelines, are both suitable implementation tools for English clause extraction.

The recommended English segmentation rules are below.

This rule set is grounded in Unicode boundary guidance, UD relation definitions, PROPN semantics, and the JustAnIota public emphasis on tokenization, ambiguity handling, and deterministic registry review.

The key heuristic for the user’s requested “segments before words” behavior is this: after clause extraction, generate overlapping phrase candidates of 3, then 2, then 1 token, but do not simply accept the first hit greedily. A better design is dynamic programming over the token sequence, because it can compare overlapping alternatives and maximize a global score such as:

• exact category hit over fuzzy category hit
• longer phrase over shorter phrase
• higher reversibility over lower reversibility
• stable/public registry entry over debug/session entry
• zero unresolved tokens over higher unresolved count

That recommendation is an implementation inference, but it follows directly from the uploaded segment-first note and JustAnIota’s deterministic-registry-first posture.

Matching architecture and data model

The most important schema rule is that Category.ISO10646 should store Unicode scalar values or code-point sequences, not font-local glyph IDs. Unicode’s glossary distinguishes characters, glyphs, and glyph codes: glyph codes are local to a font, while a rendering engine maps character data to glyphs during layout. OpenType’s cmap table is literally a character-to-glyph-index mapping table. In other words, semantics should point to code points or stable tokens, while font glyph indices remain presentation-layer details.

That matters even more if Iota-1 uses supplementary-plane characters. The public JustAnIota profile examples include code points like U+F1012 and U+F18A0, and the public workbench mentions Plane 15 and Plane 16 demo namespaces. OpenType requires a format 12 cmap subtable for supplementary-plane Unicode characters (U+10000–U+10FFFF), so if Protocol5 ships a font for visible Iota-1 code points outside the BMP, the font engineering must reflect that.

A production data model should look something like this:

erDiagram
    REGISTRY_VERSION ||--o{ CATEGORY_ENTRY : contains
    REGISTRY_VERSION ||--o{ WORD_ENTRY : contains
    CATEGORY_ENTRY ||--o{ CATEGORY_LABEL : has
    WORD_ENTRY ||--o{ WORD_LABEL : has
    CATEGORY_ENTRY ||--o{ CODEPOINT_MAPPING : maps_to
    WORD_ENTRY ||--o{ CODEPOINT_MAPPING : maps_to
    CATEGORY_ENTRY ||--o{ EXAMPLE_CASE : illustrated_by
    WORD_ENTRY ||--o{ EXAMPLE_CASE : illustrated_by
    CATEGORY_ENTRY ||--o{ NEGATIVE_CASE : constrained_by
    WORD_ENTRY ||--o{ NEGATIVE_CASE : constrained_by
    CATEGORY_ENTRY ||--o{ VALIDATION_RULE : checked_by
    WORD_ENTRY ||--o{ VALIDATION_RULE : checked_by

    REGISTRY_VERSION {
      string version_id
      string digest
      string status
      datetime published_at
    }

    CATEGORY_ENTRY {
      string category_id
      string default_gloss
      string reversibility
      string scope
      string profile
      int priority
    }

    WORD_ENTRY {
      string word_id
      string lemma
      string default_gloss
      string reversibility
      string profile
    }

    CODEPOINT_MAPPING {
      string mapping_id
      string target_form
      string codepoint_sequence
      string normalization
      string plane_policy
    }

This structure closely reflects the JustAnIota public registry requirements: stable ID, human-readable label, definition, allowed profiles, locale notes, examples, negative cases, reversibility, code-point mapping, and validator behavior.

The forward-matching algorithm should have four tiers:

For exact phrase lookup, use a trie, Aho-Corasick automaton, or spaCy PhraseMatcher-style indexed phrase matcher. pyahocorasick is explicitly designed for fast multi-pattern search and supports building the string index ahead of time for reuse. For fuzzy matching, RapidFuzz provides multiple string metrics in optimized C++-backed implementations; in the browser, Fuse.js is a lightweight zero-dependency fuzzy-search option. For semantic preview, Sentence-BERT / Sentence Transformers provide practical embeddings and similarity search, and Faiss or SQL Server vector search can scale retrieval when registries grow. But the JustAnIota spec explicitly says embeddings are not semantic authority, so embedding search should only propose candidates for deterministic confirmation by the registry layer.

A practical scoring function for overlapping candidates is:

score =
  +100 if exact registry hit
  +25  if Category.Categories instead of Category.Words
  +10  per token of matched span
  +15  if reversibility == "lossless"
  +10  if entry scope == "public"
  -20  if fuzzy rather than exact
  -40  if semantic preview rather than exact/fuzzy
  -50  if ambiguity remains unresolved

Then solve the best segmentation path with dynamic programming or Viterbi-style decoding. Greedy longest-first matching is tempting, but dynamic programming is safer when a high-value 2-token segment plus a high-value 1-token segment beats a weaker 3-token segment.

The reverse algorithm should mirror the forward one: normalize the input, split into grapheme/code-point units, try longest registered code-point sequences first, map them to stable concept IDs, then generate English glosses from the registry. If multiple glosses are possible, rank them by profile, locale, reversibility, and example support, and mark the result as approximate or ambiguous rather than pretending exactness. That aligns with the public Protocol5 and JustAnIota transparency about approximate versus deterministic output.

Performance-wise, the correct split is:

• Exact lookup for primary conversion, because it is cheap and deterministic.
• Fuzzy lookup only on a narrowed shortlist.
• Semantic lookup only for unresolved cases or preview mode.
• Vector search only when the registry is large enough to justify ANN indexing.

Unicode itself notes that rule-based boundary implementations can be compiled into deterministic state machines and combined with trie-like property tables and tailoring tables. That matches the exact kind of architecture needed here: precompiled boundary logic, precompiled phrase lexicons, and optional semantic retrieval behind them.

Implementation patterns, libraries, and tests

At the tooling level, the most useful comparison is not “which NLP library is best in the abstract,” but “which layer should each library own.”

The strongest Protocol5-specific recommendation is: use JS for UX, .NET for the canonical converter API, and optionally Python for offline grammar-evaluation tooling. That directly reuses the existing Protocol5/JustAnIota implementation center of gravity without giving up the better NLP experimentation ecosystem.

A practical public API contract should include:

If JSON leaves the server, it should be UTF-8 I-JSON, and any canonical export or signature flow should use a deterministic canonicalization scheme.

A recommended canonical envelope for Protocol5 would be:

{
  "profile": "protocol5.iota1.converter.v1",
  "registry_version": "2026-05-05-demo",
  "direction": "en_to_iota1",
  "locale": "en-US",
  "normalization": "NFC",
  "source_text": "How are you today?",
  "segments": [
    {
      "span": "how are you",
      "match_type": "category",
      "concept_id": "greeting.health_query",
      "reversibility": "lossless"
    },
    {
      "span": "today",
      "match_type": "word",
      "concept_id": "time.today",
      "reversibility": "lossless"
    }
  ],
  "output": {
    "visible": "<illustrative-visible-form>",
    "code_points": ["<illustrative-code-point-1>", "<illustrative-code-point-2>"]
  },
  "status": "illustrative_until_registry_confirms",
  "warnings": []
}

This is a recommended schema, not a claim about the current public JustAnIota final schema. It is proposed precisely because the public materials currently show more than one draft-style envelope shape.

The JavaScript side can stay simple for an MVP:

function normalizeForIota(text) {
  return text.normalize("NFC").trim();
}

function wordTokens(text, locale = "en-US") {
  const seg = new Intl.Segmenter(locale, { granularity: "word" });
  return [...seg.segment(text)]
    .filter(s => s.isWordLike)
    .map(s => s.segment.toLowerCase());
}

function categoryFirstMatch(tokens, categoryMap, wordMap) {
  const out = [];
  for (let i = 0; i < tokens.length; ) {
    let matched = null;

    for (let len = 3; len >= 1; len--) {
      const phrase = tokens.slice(i, i + len).join(" ");
      if (categoryMap.has(phrase)) {
        matched = {
          span: phrase,
          conceptId: categoryMap.get(phrase),
          kind: "category",
          len
        };
        break;
      }
    }

    if (!matched) {
      const token = tokens[i];
      matched = {
        span: token,
        conceptId: wordMap.get(token) ?? null,
        kind: wordMap.has(token) ? "word" : "unresolved",
        len: 1
      };
    }

    out.push(matched);
    i += matched.len;
  }
  return out;
}

This snippet follows the browser segmentation model exposed by Intl.Segmenter and a category-first matching policy. In production, replace the greedy loop with dynamic programming so overlapping alternatives can be scored globally.

For higher-quality grammar segmentation, a Python sidecar or offline evaluator can use spaCy like this:

from __future__ import annotations
import spacy
from typing import List, Dict, Any

NLP = spacy.load("en_core_web_sm")

CLAUSE_DEPS = {"ROOT", "advcl", "ccomp", "xcomp", "acl", "conj", "parataxis"}

def extract_semantic_segments(text: str) -> List[Dict[str, Any]]:
    doc = NLP(text)
    segments: List[Dict[str, Any]] = []

    for sent in doc.sents:
        for tok in sent:
            if tok.dep_ not in CLAUSE_DEPS:
                continue

            # Start from the clause subtree
            span_tokens = [t for t in tok.subtree if not t.is_space and not t.is_punct]

            # Exclude obvious named entities / proper nouns by default
            filtered = [
                t for t in span_tokens
                if t.pos_ != "PROPN" and t.ent_type_ == ""
            ]

            lemmas = [t.lemma_.lower() for t in filtered if t.is_alpha]
            if not lemmas:
                continue

            # Generate 1-3 token windows for Category.Categories matching
            candidates = []
            for i in range(len(lemmas)):
                for j in range(i + 1, min(i + 4, len(lemmas) + 1)):
                    candidates.append(" ".join(lemmas[i:j]))

            segments.append({
                "sentence": sent.text,
                "clause_head": tok.text,
                "lemmas": lemmas,
                "candidates": candidates
            })

    return segments

This follows the documented strengths of spaCy’s tokenizer, sentence utilities, dependency parser, and entity recognizer, while aligning the clause logic with UD relation families.

The recommended test suite should mix public fixtures, negative cases, and user-centered phrases:

The public JustAnIota examples and validator provide the right model for how fixtures and warnings should be structured: source text, registry candidates, validation report, compact demo, and at least one negative case.

Open questions, edge cases, and recommendations

The biggest open question is simply this: what is the authoritative Iota-1 lexicon and code-point policy? The public pages do not expose a complete registry export for Category.Categories, Category.Words, and Category.ISO10646; the JustAnIota registry pages are explicit that the current registry is a demo snapshot, and the examples are explicit that fixtures are not universal equivalence proofs. Without that authoritative registry, any production page can only be partially faithful.

A second open question is whether public Protocol5 should use only public Unicode assignments or also support PUA preview in controlled profiles. The Protocol5 .NET converter page and the JustAnIota spec/tooling currently point in different directions on that question. Until that is resolved, the safest public release choice is: public Unicode-visible mode for end users, optional controlled preview mode for internal testing, and explicit warnings whenever non-public code points are involved.

A third open question is how strict reversibility must be. The public Protocol5 converter page embraces approximate gist recovery; the JustAnIota registry model emphasizes reversibility levels and validator-backed evidence. A strong public product can reconcile those positions by splitting outputs into three labeled classes: lossless, approximate, and unresolved. That would be both honest and consistent with the public materials.

The most important release recommendations are straightforward:

• Freeze one canonical envelope schema and version it. The public examples currently show more than one draft-like JSON shape.
• Ship deterministic exact matching first, with segment-first matching against Category.Categories, then Category.Words. Add fuzzy and semantic retrieval only as opt-in assistive modes.
• Expose trace evidence in the UI so users can see sentence boundaries, clause boundaries, category hits, word fallbacks, unresolved spans, and warnings. That is already the implicit product pattern across the JustAnIota tool set.
• Treat proper nouns, numbers, URLs, and control characters conservatively. Preserve them for round trip; do not force them into semantic compression unless whitelisted.
• If supplementary-plane characters are used, ensure the shipped font and rendering stack support them correctly with appropriate OpenType mappings.
• Build an evaluation harness around published fixtures, negative cases, user examples, and round-trip checks before the page is linked from the Protocol5 homepage. The public JustAnIota record family already points toward validator-backed proof before support claims.

In short, the research supports building the requested page on Protocol5, and it strongly suggests how to build it: a transparent, registry-backed, sentence-and-segment-aware converter with authoritative exact matching first, visible evidence always, and approximate/semantic helper layers clearly separated from semantic authority. The missing piece is not the web page itself; it is the finalization of the authoritative Iota-1 registry and encoding policy.