Improving the Protocol5 JustAnIota IOTA-1 Converter

Full report

On this page

Executive summary

The current Protocol5 JustAnIota converter is already more sophisticated than a simple dictionary mapper: it performs paragraph and sentence segmentation, tries long stored English segments in Category.Categories before word-level fallback in Category.Words, ranks public Unicode glyph candidates from Category.ISO10646, exposes trace and vector evidence, normalizes text with NFC and grapheme-aware handling, and stores embeddings in SQL Server vector columns behind a C# facade. At the same time, the public surface is explicitly approximate, public-symbol-only, and anti–private-codebook by design. On the public host today, the status endpoint reports only 36 public seed concepts, a configured vector width of 1998, liveAiConfigured: true, and sqlConfigured: true but reachable: false, which means the live site can fall back to a substantially thinner semantic base than the intended SQL corpus.

That architecture is workable for “English gist → nearest public symbol” conversion, but it is not yet a robust glyph-semantic system. The principal weaknesses are structural. First, semantics are still anchored primarily in registry rows and descriptor text rather than in the glyph’s visual form. Second, rendering is tightly coupled to public Unicode rows, which is correct for Protocol5’s governance boundary but makes it hard to interpret novel or composite glyphs whose meaning depends on geometry, containment, stroke arrangement, or relational composition. Third, the matching path is language-centric: the converter starts from English segments and only later ranks visible glyphs, so it is strong on lexical anchoring and weaker on glyph-first interpretation. Fourth, tokenization and segmentation can erase internal symbolic structure; recent work shows that when meaningful substructure and token boundaries misalign, model meaning representations degrade.

The right upgrade is therefore not to replace Protocol5’s evidence-first, public-symbol boundary. It is to add a semantic overlay that separates four layers cleanly: the visible glyph, the vector-native visual structure, the semantic embedding space, and the ontology-constrained canonical IOTA expression. In practice, that means introducing glyph records that store SVG decomposition, primitive graphs, visual embeddings, semantic embeddings, ontology tags, attention metadata, and a converter-specific phase-lock score; then using a fusion pipeline that combines a vision-language encoder with a vector-native SVG encoder and re-ranks nearest neighbors under ontology and composition constraints.

My recommended stack is: SigLIP 2 as the primary vision-language encoder, SVGformer as the primary vector-native structural encoder, and DINOv2 as an optional auxiliary visual branch for robustness on rasterized or imperfect SVG inputs. In storage, the best near-term choice is to stay inside the existing Protocol5/.NET/SQL Server architecture for Phase 1, because SQL Server 2025 already supports native vector columns, exact kNN, and DiskANN-based approximate search, while SqlVector<T> support in Microsoft.Data.SqlClient fits the current ADO.NET architecture. If scale, payload filtering complexity, or multi-vector retrieval requirements outgrow that path, Qdrant is the cleanest secondary target.

The most important design principle is this: Protocol5 should continue to render only public, inspectable symbols, but it should stop pretending that semantic meaning lives only in token rows. Meaning should instead be inferred from a fused evidence bundle—geometry, relative composition, multilingual descriptors, ontology tags, neighborhood consensus, and converter diagnostics—and only then collapsed into an ontology-validated IOTA canonical expression and public-symbol output. That preserves the Protocol5 boundary while materially improving precision, extensibility, and auditability.

Current system diagnosis

Protocol5’s public documentation gives a coherent snapshot of the existing converter. The live pipeline is English-first; it splits text into paragraphs and sentences, attempts the longest matching stored English segments first, then falls back to words, and finally ranks public Unicode glyph rows. The logic layer owns Unicode normalization, rune/scalar handling, grapheme grouping, semantic segmentation, candidate ranking, vector-evidence summaries, approximation labels, and private-use rejection. The repository layer is persistence-agnostic, but the SQL Server implementation stores English anchors and public symbol embeddings in vector columns and uses VECTOR_DISTANCE, VECTOR_SEARCH, and DiskANN when available. Embedding population is intentionally local-only, while the hosted public demo exposes only read-only endpoints.

This yields several concrete failure modes.

Token registry failure appears whenever a glyph or composite symbol is not well covered by the seed registry or reachable SQL corpus. The public status endpoint currently reports only 36 public seed concepts and an unreachable SQL corpus on the public surface. In that condition, the converter necessarily degrades toward a sparse registry-plus-fallback behavior. This is enough for simple demo phrases such as good help → 好救, but it is not enough for nuanced glyph semantics, especially for multi-part or visually novel symbols.

Rendering coupling failure arises because the final visible output is tightly bound to Category.ISO10646 rows. Protocol5 is clear that this is a rule, not an accident: IOTA-1 must use assigned public Unicode characters and standard public sequences; private-use areas and secret semantic maps are disallowed. That is the right governance posture, but it means the current system is optimized for selecting a public symbol candidate, not for representing the internal structure of a glyph whose meaning depends on shape composition. In other words, the visible symbol inventory is inspectable, but the semantic machinery behind it is still too row-centric.

Lexical mapping failure is subtler but more important. The current grammar order privileges English phrase structure and word fallback. That is excellent when the source signal is English prose and the destination is approximate public symbols. It is weaker when the source signal is itself a glyph, especially a composite or unknown one. The public search endpoint also reflects this bias: search is over categories, words, or ISO-10646 rows using either caller-supplied embeddings or input text, rather than over a first-class glyph-object graph with explicit visual decomposition.

There is also a tokenization and structure-loss failure that comes from the broader model ecosystem rather than Protocol5 alone. Haslett shows that misalignment between meaningful radicals and token boundaries systematically corrupts model representations in Chinese and across several European languages, and that collapsing meaningful form into fewer, longer tokens can reduce accuracy. Shih and colleagues similarly show that LLMs and LVLMs struggle with rare scripts not encoded in Unicode, even when given picture-based or description-based support. For an IOTA converter, that means a glyph should never be treated as a black-box token if its internal arrangement carries semantic load.

Finally, there is a semantic observability gap. Protocol5 already returns ranking lanes, scores, provenance, and evidence families, which is the correct direction. But it does not yet expose enough structured diagnostics about why a glyph candidate was selected in geometric, ontological, or compositional terms. For a glyph-semantic converter, raw distance scores are necessary but not sufficient; the system also needs primitive-level explanations, ontology checks, relation inference, and neighborhood-consensus diagnostics.

Proposed semantic architecture

The central architectural change is to treat each glyph as a multilayer semantic object rather than as a single registry row. The proposed object has four separable layers: a surface layer containing the public symbol or SVG representation; a structure layer containing paths, primitives, relations, and composition graphs; an embedding layer containing visual, semantic, and ontology-projected vectors; and a canonical layer containing the ontology-validated IOTA expression that the converter can render or explain. This is consistent with Protocol5’s evidence-first posture and with ontology-backed glyph work such as EASY-AI and BEAM, which separate visual symbols from the machine-readable formalisms that govern how those symbols compose and communicate meaning.

In practical terms, the converter should maintain three retrieval spaces instead of one. The first is a visual space for “what this glyph looks like,” learned from raster renderings and patch-level features. The second is a vector-native structural space for “how this glyph is built,” learned from SVG paths, primitive relations, and geometric attention. The third is a semantic-ontology space for “what this glyph is allowed to mean,” learned from curated descriptors, multilingual anchor text, and ontology tags. Retrieval should happen in all three spaces, with late fusion and explicit constraint checking before the converter emits a canonical expression. CLIP-style models, SigLIP, and DINOv2 provide strong generic visual representations; SVGformer and DeepSVG provide vector-native SVG representations; EASY-AI and BEAM provide the right conceptual precedent for semantic and compositional constraints.

A useful way to think about the proposal is that Protocol5 currently has a good public-symbol renderer and a partial semantic retriever, but it lacks a first-class glyph semantics kernel. That missing kernel should own visual decomposition, primitive graph construction, embedding fusion, ontology validation, and converter diagnostics. The existing IJustAnIotaConverterFacade can remain the stable entry point, but the facade should call this new kernel before ranking or emitting visible symbols. This preserves API stability while materially increasing semantic depth.

flowchart LR
    A[Input text or glyph] --> B[Unicode and SVG canonicalization]
    B --> C[Sentence and grapheme segmentation]
    B --> D[SVG path parsing and primitive extraction]
    C --> E[Text embedding tower]
    D --> F[Visual embedding tower]
    D --> G[SVG structural encoder]
    E --> H[Fusion and query vector set]
    F --> H
    G --> H
    H --> I[ANN retrieval in visual semantic and ontology indexes]
    I --> J[Ontology constraint filter]
    J --> K[Composition parser and reranker]
    K --> L[IOTA canonical expression]
    L --> M[Public symbol rendering]
    K --> N[Evidence trace and diagnostics]

Candidate model and tool choices

The following table compares the highest-value candidates for the representation stack.

The net recommendation is a dual encoder plus structural encoder: SigLIP 2 for image-text alignment, SVGformer for vector-native structure, and DINOv2 as a robustness branch. DeepSVG and IconShop should not be the main semantic retrieval engines, but they are quite valuable for training data expansion, perturbation testing, and active-learning loops.

Pipeline and data model

The proposed pipeline begins by preserving Protocol5’s existing Unicode and public-symbol rules, but inserts an SVG and glyph analysis stage before semantic ranking. SVG is a standardized XML-based vector format whose path data encodes moveto, lineto, curveto, arc, and closepath instructions; SVG viewports and viewBox control coordinate systems, and bounding boxes can be computed in SVG space. Those standards matter because they let the converter derive stable, auditable primitive features from the symbol itself rather than treating the glyph as an opaque image.

The decomposition stage should canonicalize transforms, flatten simple shapes to equivalent paths, normalize scale and orientation, then extract a primitive graph with nodes for strokes, closed shapes, holes, endpoints, junctions, arcs, and symmetry axes. SVGformer is especially relevant here because it shows that continuous SVG input plus geometric self-attention can materially improve representation learning and retrieval over prior SVG baselines; DeepSVG is relevant because it formalizes the path hierarchy and the arbitrary ordering of shapes in SVG documents.

Storage should be multi-vector and versioned. At minimum, each glyph record should carry: a surface representation, a structured decomposition, a global visual embedding, one or more regional or primitive embeddings, a semantic text embedding, an ontology projection, provenance, model-version metadata, and converter diagnostics. Because Protocol5 already lives in C# plus SQL Server, the lowest-risk first move is to keep storage in SQL Server 2025 using native vector columns and exact search for small candidate sets, while the public search endpoints and ADO.NET stack continue to work. SQL Server’s exact search is explicitly recommended when the candidate set is under roughly 50,000 vectors after filtering; DiskANN ANN is the next step when scale requires it.

The main alternative storage choices are shown below.

A glyph record should look more like a semantic asset than a word dictionary entry. One workable schema is:

{
  "glyphId": "iota:glyph:warn-fire:v1",
  "surface": {
    "unicodeSequence": ["U+26A0", "U+706B"],
    "display": "⚠火",
    "svgHash": "sha256:...",
    "rasterPreviewUri": "blob://glyphs/warn-fire.png"
  },
  "structure": {
    "viewBox": [0, 0, 1024, 1024],
    "primitives": [
      {"id": "p1", "type": "triangle", "bbox": [48, 64, 420, 384]},
      {"id": "p2", "type": "line", "role": "exclamationStem", "bbox": [220, 140, 24, 140]},
      {"id": "p3", "type": "dot", "role": "exclamationDot", "bbox": [226, 300, 16, 16]},
      {"id": "p4", "type": "flare", "role": "fire", "bbox": [520, 120, 360, 420]}
    ],
    "relations": [
      {"type": "contains", "source": "p1", "target": "p2"},
      {"type": "contains", "source": "p1", "target": "p3"},
      {"type": "rightOf", "source": "p4", "target": "p1"},
      {"type": "sequence", "order": ["p1", "p4"]}
    ]
  },
  "embeddings": {
    "visualGlobal": {"model": "siglip2-so400m-patch16-384", "dim": 1152},
    "visualStructural": {"model": "svgformer", "dim": 768},
    "semanticText": {"model": "text-tower", "dim": 768},
    "ontologyProjection": {"model": "iota-ontology-proj-v1", "dim": 256}
  },
  "ontology": {
    "concepts": ["hazard", "fire", "warning"],
    "roles": ["modifier", "event"],
    "canonicalType": "WarningOfDanger"
  },
  "metadata": {
    "provenance": ["unicode", "cldr", "manual-curation"],
    "confidence": 0.94,
    "approvedForPublicOutput": true,
    "privateUseRejected": true
  },
  "attention": {
    "rarityScore": 0.71,
    "candidateEntropy": 0.19,
    "attentionAgreement": 0.83,
    "phaseLockScore": 0.64
  }
}

That design is deliberately compatible with Protocol5’s public-symbol rule: the visible output is still fully inspectable and public, but the converter can now reason over geometry, roles, and provenance before it emits the result. It also matches the Protocol5 documentation’s insistence that embeddings should include descriptor text, anchors, code-point evidence, and public provenance rather than bare code points alone.

Composition, scoring, and example transformations

Recursive composition should be handled by a spatial graph grammar, not by a flat nearest-neighbor lookup. The grammar should infer three primary relation families: order, proximity, and containment. Order is derived from centroids, explicit z-order, writing direction, and optionally stroke order if known. Proximity is derived from normalized center distance, boundary distance, overlap, and alignment. Containment is derived from bounding boxes, winding rules, enclosed area, and topological-inside tests. This is the right abstraction because SVG is natively path-based and because both SVGformer and DeepSVG show that path hierarchy and long-range relations are central to representation quality.

A practical parse algorithm is:

1. Canonicalize the glyph into a primitive graph.
2. Generate candidate subgraphs using connectedness, containment, and symmetry heuristics.
3. Compute local primitive embeddings and a whole-glyph fused embedding.
4. Retrieve top-k semantic neighbors in visual, structural, and text spaces.
5. Re-rank under ontology constraints, discarding candidates that violate allowed type-role relations.
6. Build a canonical IOTA expression by choosing the highest-scoring compositional parse.
7. Emit both the expression and the evidence trace.

The scoring layer should be explicit and auditable. A workable family of converter-native scores is:

proximityScore(a,b) = exp(-d_norm(a,b) / sigma)
containmentScore(a,b) = inside(a,b) * area(intersection(a,b)) / area(b)
orderScore(S) = reading_order_consistency(S, direction)
rarityScore(g) = -log((freq_unicode(g) + freq_structure(g) + alpha) / Z)
candidateEntropy(q) = -Σ p(c|q) log p(c|q) / log K
attentionAgreement(g) = 1 - mean_head_entropy(region_attention(g))
phaseLockScore(g) = w1 * reuse_stability
                  + w2 * (1 - embedding_drift)
                  + w3 * neighbor_consensus
                  + w4 * cross-session_persistence

Two notes matter here. First, phase-lock score is not a standardized external metric in the literature you asked me to prioritize, so I am defining it operationally for this converter: it measures whether a glyph’s inferred meaning remains stable across repeated conversions, nearby contexts, and model versions. Second, attention metadata should be treated as diagnostic evidence, not as truth. Attention maps, entropy, and saliency are useful for debugging and trust, but they should not override ontology violations or human validation on their own. The motivation for entropy-sensitive diagnostics is supported by recent work on token entropy and reasoning as well as on tokenization-driven meaning loss.

A minimal canonical expression form should be tree-shaped and typed. For example:

{
  "canonicalExpression": {
    "nodeType": "Relation",
    "relation": "modifies",
    "children": [
      {"nodeType": "Concept", "id": "warning", "confidence": 0.98},
      {"nodeType": "Concept", "id": "fire", "confidence": 0.95}
    ]
  },
  "publicOutput": "⚠火",
  "approximate": true
}

Here are two worked examples.

Example transformation

Example A: existing public symbols

Input glyph(s): ⚠火 Visual decomposition: warning triangle + exclamation stem/dot, followed by fire ideograph. Semantic embedding result: nearest fused neighbors cluster around warning, hazard, fire, burn, alert. Ontology validation: warning is a valid modifier of a hazard/event concept; fire is an event/object concept. Canonical IOTA expression: WarningOf(Fire). The use of ⚠火 is also consistent with the example symbols shown on the Protocol5 language-converter page.

{
  "input": "⚠火",
  "decomposition": {
    "sequence": ["warning-symbol", "fire-symbol"],
    "relations": [{"type": "modifier", "source": "warning-symbol", "target": "fire-symbol"}]
  },
  "semanticEmbedding": {
    "topNeighbors": [
      {"concept": "warning", "score": 0.98},
      {"concept": "hazard", "score": 0.96},
      {"concept": "fire", "score": 0.95}
    ]
  },
  "ontologyValidated": {
    "expression": "WarningOf(Fire)",
    "typeCheck": "pass"
  },
  "publicOutput": "⚠火",
  "approximate": true
}

Example B: containment-driven glyph

Input glyph: ◎ Visual decomposition: outer ring contains center dot. Semantic embedding result: ambiguous between target, focus, core, sun, eye. Ontology validation: if the surrounding sentence concerns systems, hubs, or centers, containment semantics should raise core-within-field over pictorial neighbors like sun; if the context is perception or observation, eye may win. This is the crucial place where ontology and context need to overrule raw nearest-neighbor similarity.

{
  "input": "◎",
  "decomposition": {
    "primitives": ["outer-circle", "inner-dot"],
    "relations": [{"type": "contains", "source": "outer-circle", "target": "inner-dot"}]
  },
  "semanticEmbedding": {
    "topNeighbors": [
      {"concept": "focus", "score": 0.88},
      {"concept": "core", "score": 0.86},
      {"concept": "target", "score": 0.84},
      {"concept": "sun", "score": 0.81}
    ]
  },
  "ontologyValidated": {
    "context": "system architecture",
    "expression": "CoreWithin(Container)",
    "typeCheck": "pass"
  },
  "publicOutput": "◎",
  "approximate": true
}

Evaluation strategy

The evaluation plan should combine Protocol5-style evidence inspection with formal symbol-comprehension testing. ISO 9186-1 is the most relevant external standard: it specifies a method for testing the comprehensibility of graphical symbols and explicitly aims to measure how well a graphical symbol communicates its intended message without explanatory text. For an IOTA converter, that maps naturally to “glyph → concept” and “concept → glyph” testing, both with and without contextual scaffolding.

A strong benchmark suite should contain five evaluation lanes. The first is canonical-expression accuracy: exact match and tree-edit distance between the predicted IOTA canonical expression and a curated gold parse. The second is retrieval quality: top-1, top-5, and MRR for glyph-to-concept, concept-to-glyph, and composite-glyph tasks. The third is human forced-choice validation: show a glyph and ask users to choose among four candidate meanings, then invert the task by asking users to choose the intended glyph for a meaning. The fourth is round-trip fidelity: text → glyph → canonical expression → English gist, and glyph → canonical expression → public output → gist. The fifth is constraint integrity: percentage of outputs that violate declared ontology role/type constraints or Protocol5 public-symbol rules.

Semantic-distance testing should be explicit, because icon and symbol research shows that semantic distance materially affects recognition and cognitive performance. That suggests using a benchmark where the converter must discriminate among near neighbors such as danger, warning, fear, fire, and alarm, rather than only among distant concepts. In practice I would use both embedding-space metrics—cosine distance among gold concepts—and human-rated semantic-distance strata to see whether the converter collapses fine-grained distinctions under visual similarity.

Attention diagnostics should be evaluated, but as diagnostics rather than primary quality targets. For each glyph, store a region-saliency map and a primitive-ablation curve: remove the ring, dot, or enclosing boundary and measure how much the canonical-expression confidence changes. Good models should place saliency on semantically load-bearing regions, not on incidental raster artifacts or tokenization quirks. Entropy should be measured at two points: the entropy of the candidate distribution at retrieval time, and the entropy of attention mass over glyph regions. Sudden spikes are useful unknown-glyph indicators.

The evaluation suite should also include ablations. At minimum, compare: text-only, visual-only, SVG-only, visual+text, and visual+text+ontology. The expected result is that text-only and visual-only each underperform on hard cases, while ontology-constrained fusion reduces plausible-but-wrong neighbors. That expectation is consistent with the general lessons from CLIP-style multimodal encoders, SVG-native representation learning, and Shih’s picture-versus-description evaluation on rare scripts.

Roadmap

For effort sizing, I am making explicit planning assumptions because you said unspecified values should be treated as assumptions: an initial curated set of roughly 5,000 gold glyph records, a broader weakly labeled training pool of 50,000 to 200,000 glyph or SVG examples, and a latency target of ≤500 ms p95 for retrieval-only requests and ≤2 s p95 for hybrid reranking. I am also assuming a delivery team of two to three engineers, one part-time ontology/curation lead, and one part-time QA or UX owner.

gantt
    title Proposed implementation timeline
    dateFormat  YYYY-MM-DD
    section Phase 1
    Schema and provenance layer        :a1, 2026-05-12, 14d
    SVG parsing and primitive graph    :a2, after a1, 14d
    SQL vector integration and traces  :a3, after a1, 21d
    section Phase 2
    SigLIP 2 visual-text fusion        :b1, 2026-06-23, 21d
    SVGformer structural encoder       :b2, 2026-06-30, 21d
    Ontology reranker and parser       :b3, 2026-07-14, 28d
    Evaluation harness                 :b4, 2026-07-21, 21d
    section Phase 3
    Human validation tools             :c1, 2026-08-18, 21d
    Active learning and diagnostics    :c2, 2026-08-25, 28d
    Production optimization            :c3, 2026-09-08, 21d

If the goal is speed with minimum platform change, the best near-term architecture is:

• Protocol5 C# facade remains the contract boundary.
• A Python inference sidecar hosts SigLIP 2, SVGformer, and optional DINOv2.
• SQL Server 2025 stores versioned vectors, provenance, and ontology tags.
• Local-only mutation tooling stays consistent with the current Protocol5 rule that embedding population should not occur from the public web host.
• Public output remains limited to approved public symbols or public sequences.

Risks and open questions

The biggest technical risk is ontology overconstraint. If the ontology is too narrow or too rigid, the converter will reject semantically reasonable candidates and become brittle on creative or emergent glyphs. The mitigation is to use ontology in two stages: first as a soft reranker, then as a hard validator only for production-facing public output. Curator tooling should make it easy to inspect why a candidate was rejected and to add exceptions or new relations without retraining the whole stack. The BEAM and EASY-AI literature is helpful here because both treat machine-readable formalism and human-facing glyphs as linked but distinct layers.

A second risk is data sparsity, especially for rare or bespoke glyphs. SVGformer itself notes that representation quality depends on having enough SVGs during training. The mitigation is to keep a small high-quality gold set, then expand coverage with synthetic perturbations, structured SVG augmentations, DeepSVG/IconShop generation, and active-learning selection of uncertain examples for human review. Omniglot and OpenMoji are also useful as supplementary structure-rich datasets and tooling references, though their licensing and domain fit need to be handled carefully.

A third risk is infrastructure mismatch. SQL Server’s native vector support is attractive because it matches Protocol5’s stack, but it has a 1998-dimension ceiling and several vector-column limitations. That is not fatal—most practical towers can be projected below that threshold—but it argues for explicit dimensionality management and versioning from the beginning. If the system later needs more complex filtered ANN behavior or multi-vector query patterns than SQL handles gracefully, Qdrant is the cleanest scale-out path.

A fourth risk is false confidence from attention diagnostics. Saliency and attention are useful for debugging, but they are not ground truth explanations. The mitigation is to keep attention metadata in the evidence bundle, not in the authority chain: canonical expression quality should still be judged by human tests, round-trip benchmarks, ontology constraints, and retrieval metrics.

A fifth risk is protocol drift against Protocol5’s public-symbol rule. A very natural temptation will be to store custom SVG glyphs and start rendering them directly. That would violate the current boundary if those glyphs become the visible authority. The mitigation is simple and important: internal SVGs may be used for analysis, authoring, and semantic decomposition, but the public rendering path should still resolve to approved, inspectable public symbols or public canonical sequences unless Protocol5 governance changes the rule.

The main open questions are short and practical. The first is corpus size: how many glyph records, composite examples, and ontology labels are actually intended for the first production epoch? The second is latency budget: is the target closer to a UI autocomplete experience or a slower but fully explained forensic mode? The third is governance: will Protocol5 allow internal non-rendered SVG assets as semantic source material, provided public output remains Unicode-only? The fourth is ontology scope: should the initial ontology be a narrow converter ontology for concepts like entity/action/quality/relation/modifier, or should it immediately adopt a richer system such as an EASY-AI/BEAM-aligned graph? Those answers affect sizing, but they do not change the core conclusion: the converter should move from row-centric approximate lexical mapping to evidence-fused glyph semantics with ontology-constrained canonicalization.