Improving the Protocol5 JustAnIota IOTA-1 Converter

Full report

On this page

Executive summary

Protocol5’s public JustAnIota converter already implements a coherent approximate-conversion stack: it segments paragraphs into sentences, tries longer English segments in Category.Categories before falling back to Category.Words, then ranks public Unicode candidates from Category.ISO10646; it exposes ranked candidates, trace evidence, vector evidence, approximation labels, provenance, and a stable IJustAnIotaConverterFacade; and it stores embeddings through an ADO.NET layer over SQL Server vector columns with optional LM Studio assistance. The protocol boundary is also explicit: IOTA-1 is approximate, public-symbol-only, and must not become a private codebook or secret bilingual map.

The main architectural weakness is not that the converter is “wrong” about those boundaries. It is that semantics are still anchored primarily in registry rows and lexical descriptors, while true glyph interpretation often depends on visual structure, primitive relations, and composition. On the public host, that risk is amplified by the current runtime state: the status endpoint reports only 36 public seed concepts, liveAiConfigured: true, a vector width of 1998, and a SQL corpus that is configured but currently unreachable. In practice, that means the live system can degrade from “vector-backed semantic lookup” toward a much thinner seed-registry fallback.

The most effective upgrade is therefore a semantic overlay architecture, not a replacement of Protocol5’s public-symbol rule. The converter should continue to render only public, inspectable Unicode characters or public sequences, but it should add a new internal layer that treats each glyph as a structured semantic object with: a surface representation, SVG- or path-level decomposition, fused visual and semantic embeddings, ontology tags, attention diagnostics, and a converter-specific stability score for repeated meaning convergence. Recent primary sources point in exactly that direction: CLIP-style dual encoders are strong at image-text alignment; SigLIP 2 improves multilingual retrieval, localization, and dense features; SVGformer is purpose-built for continuous SVG structure; DINOv2 is a strong auxiliary visual backbone; and Glyce is a concrete demonstration that glyph cues plus symbol identity outperform identity alone for script representation.

My recommended near-term stack is: SigLIP 2 as the primary vision-language encoder, SVGformer as the primary vector-native structural encoder, DINOv2 as an auxiliary robustness branch, and SQL Server 2025 native vectors as the initial storage and query substrate because that aligns with Protocol5’s current .NET and SQL architecture. The one important caveat is dimensionality and query design: SQL Server vector columns top out at 1998 dimensions, exact search is generally recommended when search predicates reduce the candidate set below about 50,000 vectors, and approximate vector indexing is still a preview feature. That combination strongly argues for multiple vector columns with late fusion, not a single concatenated mega-vector.

The practical implication is simple. Protocol5 does not need to relax its public-symbol governance to become more semantically capable. It needs to stop treating glyph meaning as something that lives mostly in token rows, and instead infer meaning from a fused evidence bundle—geometry, structure, text descriptors, ontology constraints, and retrieval diagnostics—before collapsing that bundle into an IOTA canonical expression and then into a public visible rendering.

Assumptions and system boundary

Several important implementation constraints were unspecified in the request, so the roadmap and design below use explicit working assumptions. The only items I treat as hard protocol constraints rather than assumptions are the public-symbol rule, the no-secret-map rule, and the current split between read-only hosted endpoints and local-only mutation or embedding population.

The reason these assumptions matter is that Protocol5’s existing architecture already creates a natural boundary condition for the redesign: do not change the public trust model; deepen the internal semantics layer. That should guide every technical choice in the report.

Current architecture and failure modes

Protocol5’s public documentation describes the current converter as an English-first approximate semantic engine. The live path prefers stored SQL corpus vectors when available, falls back to public seed concepts when the corpus is unavailable, segments by paragraphs and sentences, tries long stored English segments before single-word fallback, and then ranks public Unicode glyph rows for visible IOTA output. The logic layer handles NFC normalization, scalar and grapheme handling, semantic segmentation, candidate ranking, evidence summaries, provenance atlases, and private-use rejection, while the repository layer uses SQL Server vector columns and vector-search functions where available.

That overall shape is sensible for “English gist to approximate public symbol,” but it produces five material failure modes when the goal becomes deeper glyph semantics.

Failure-mode map

The rendering-coupling issue needs careful interpretation. Protocol5’s insistence on public Unicode output is a feature, not a bug, because it preserves inspectability and rejects private semantic authority. The weakness is not the rule itself; it is the absence of an internal, non-rendered semantics layer that can reason over geometry and composition before the public rendering step.

The lexical bias is similarly understandable but limiting. For English-source conversion, long-segment matching before word fallback is a pragmatic way to preserve phrase meaning. For glyph-source interpretation, however, it means the pipeline is still fundamentally “language-in, symbol-out,” not “glyph-in, meaning-out.” That becomes especially problematic when composition matters more than lexical gloss.

A final issue is the current public-host runtime state. Because SQL is presently unreachable on the public status endpoint, the hosted experience cannot reliably demonstrate the richer vector-corpus path that the architecture intends. That does not invalidate the design, but it does mean that any public-facing evaluation today likely underestimates what a corpus-backed version could do—and it also highlights why a more explicit semantic overlay should not depend on a single fragile storage lane.

Proposed semantic architecture

The recommended redesign is a four-layer glyph architecture:

1. a surface layer for public Unicode sequences, SVG, and rendered previews;
2. a structure layer for primitives, path segments, relations, and composition graphs;
3. an embedding layer for visual, structural, semantic-text, and ontology-projected vectors;
4. a canonical layer for the ontology-validated IOTA expression that becomes the source of public rendering and explanation.

That design is consistent with Protocol5’s evidence-first philosophy and with the way the most relevant model families divide the problem: CLIP and SigLIP-style models align vision and text globally, SVGformer learns directly from continuous SVG structure, DINOv2 supplies robust visual features without labels, and Glyce demonstrates that glyph information and symbol identity should be modeled jointly rather than treated as mutually exclusive alternatives.

Recommended semantic fields per glyph

A key governance point follows from this architecture. Protocol5 does not need to make SVGs or non-Unicode internal assets the public semantic authority. It can use them internally for analysis while still requiring that the final visible rendering be a public character or public standard sequence. In other words: internal geometry may become part of the evidence chain, but the public symbol inventory remains inspectable. That preserves the protocol boundary while improving capability.

The visual and structural embedding layers should be held apart rather than forced into a single undifferentiated vector. SigLIP 2’s contribution is multilingual image-text alignment plus improved localization and dense features, which is directly useful for glyph parts and composite symbol regions. SVGformer’s contribution is continuous SVG representation learning with explicit geometric handling, which is directly useful when glyph meaning depends on stroke geometry, long-range dependencies, and shape composition. DINOv2 should be treated as an auxiliary branch for robustness on rasterized or imperfect glyph images, not as the main semantic authority.

Concrete pipeline and data models

The pipeline below keeps Protocol5’s current public-output rules but adds a true semantic kernel between ingestion and rendering. The structural design is motivated by the current converter surface, the W3C SVG path model, and the strengths of SigLIP 2, SVGformer, and DINOv2.

flowchart LR
    A[Input text or glyph] --> B[Unicode and SVG canonicalization]
    B --> C[Sentence and grapheme segmentation]
    B --> D[SVG path parsing and primitive extraction]
    C --> E[Text embedding tower]
    D --> F[Visual embedding tower]
    D --> G[SVG structural encoder]
    E --> H[Fusion and query vector set]
    F --> H
    G --> H
    H --> I[Filtered retrieval across visual structural and semantic indexes]
    I --> J[Ontology constraint filter]
    J --> K[Composition parser and reranker]
    K --> L[IOTA canonical expression]
    L --> M[Public Unicode rendering]
    K --> N[Trace evidence and diagnostics]

The practical pipeline should work like this in implementation terms. First, normalize text with the current Protocol5 NFC and grapheme machinery; for glyph input, normalize Unicode sequences and, where available, canonicalize SVG path transforms and equivalent shapes. The W3C SVG specifications matter here because SVG basic shapes can be reduced to path-equivalent forms, which makes primitive extraction and structural indexing stable across sources.

Second, generate three parallel embedding families:

• a vision-language family from rendered glyph images using SigLIP 2;
• a vector-native structural family from SVG path commands and primitive relations using SVGformer;
• a semantic text family from public descriptors, CLDR/UCD/Unihan metadata, canonical-expression glosses, and curated ontology labels.

Protocol5 already points to UCD, CLDR, Unihan, and emoji data as public metadata sources for its symbol atlas; those are appropriate first-party sources for descriptors and anchors, and Unicode’s official resources document exactly those repositories.

Third, retrieve in multiple spaces and then fuse. Because SQL Server vector columns max out at 1998 dimensions—and the current Protocol5 status endpoint is already configured at that ceiling—the design should not concatenate all modalities into one vector. Store separate columns such as visual_global, svg_structural, semantic_text, and ontology_projection, then fuse at query time with either reciprocal-rank fusion or a lightweight learned reranker. This is a major architectural point: multi-column late fusion is not just better semantically; it is also directly aligned with the current SQL dimensionality limit.

A workable late-fusion score is:

score(c | q) =
  α * cos(q_visual, c_visual)
+ β * cos(q_svg, c_svg)
+ γ * cos(q_text, c_text)
+ δ * ontology_match(q, c)
+ ε * relation_match(q, c)
+ ζ * RRF_rank_bonus(c)
- λ * public_rule_penalty(c)
- μ * composition_violation_penalty(c)

where ontology_match and relation_match are typed compatibility scores, and RRF_rank_bonus is a reciprocal-rank-fusion term computed across the different retrieval lanes.

Candidate model comparison

The model choice should be based on original papers or official repositories, not on secondary summaries. The table below is a practical synthesis of those primary sources.

Candidate vector-store comparison

Storage choice matters because the retrieval design here is not just “ANN over one vector”; it is multiple vectors plus ontology-tag filters plus provenance plus diagnostics. The table below compares the best options against that requirement set.

Example glyph-record schema

The schema below is intentionally aligned with Protocol5’s public-symbol boundary while adding the missing internal semantics. It is a design proposal, not an existing Protocol5 schema.

{
  "glyphId": "iota:glyph:warning-fire:v1",
  "status": {
    "approvedForPublicOutput": true,
    "approximate": true,
    "privateUseRejected": true
  },
  "surface": {
    "unicodeSequence": ["U+26A0", "U+706B"],
    "display": "⚠火",
    "svgHash": "sha256:7d9c...",
    "renderProfiles": [
      {"name": "default", "rasterUri": "blob://glyphs/warn-fire-384.png"}
    ]
  },
  "structure": {
    "viewBox": [0, 0, 1024, 1024],
    "pathCommands": 184,
    "primitives": [
      {"id": "p1", "type": "triangle", "bbox": [48, 64, 420, 384]},
      {"id": "p2", "type": "line", "role": "exclamationStem", "bbox": [220, 140, 24, 140]},
      {"id": "p3", "type": "dot", "role": "exclamationDot", "bbox": [226, 300, 16, 16]},
      {"id": "p4", "type": "ideograph", "role": "fire", "bbox": [520, 120, 360, 420]}
    ],
    "relations": [
      {"type": "contains", "source": "p1", "target": "p2"},
      {"type": "contains", "source": "p1", "target": "p3"},
      {"type": "rightOf", "source": "p4", "target": "p1"},
      {"type": "sequence", "order": ["p1", "p4"]}
    ]
  },
  "embeddings": {
    "visualGlobal": {"column": "visual_global", "model": "siglip2-so400m", "dim": 1152},
    "visualAux": {"column": "visual_aux", "model": "dinov2-base", "dim": 768},
    "svgStructural": {"column": "svg_structural", "model": "svgformer", "dim": 768},
    "semanticText": {"column": "semantic_text", "model": "siglip2-text", "dim": 768},
    "ontologyProjection": {"column": "ontology_projection", "model": "iota-onto-v1", "dim": 192}
  },
  "ontology": {
    "concepts": ["warning", "hazard", "fire"],
    "canonicalType": "WarningOfDanger",
    "roles": [
      {"token": "warning", "role": "modifier"},
      {"token": "fire", "role": "event"}
    ],
    "constraints": [
      "modifier->event allowed",
      "publicOutputRequired"
    ]
  },
  "provenance": {
    "sources": ["UCD", "CLDR", "Unihan", "Protocol5-curation"],
    "versions": {
      "ucd": "latest",
      "cldr": "stable",
      "ontology": "iota-onto-v1"
    }
  },
  "diagnostics": {
    "rarityScore": 0.71,
    "candidateEntropy": 0.19,
    "attentionAgreement": 0.83,
    "phaseLockScore": 0.64
  }
}

Composition and scoring algorithms

The most important algorithmic shift is that glyph composition should be treated as a typed spatial graph problem, not as flat token lookup. W3C’s SVG path model gives the structural substrate for this, SVGformer provides a representation-learning precedent for continuous SVG structure, and DeepSVG reinforces the importance of path hierarchy and high-level versus low-level shape separation. Shih et al. also show that narrative descriptions of complex glyph tokens frequently lose orientation and relational detail, which is exactly why the converter should reason over explicit relations such as order, proximity, and containment rather than only over gloss text.

flowchart TD
    A[Primitive graph] --> B[Detect order relations]
    A --> C[Detect proximity relations]
    A --> D[Detect containment relations]
    B --> E[Generate candidate subgraphs]
    C --> E
    D --> E
    E --> F[Assign concept and role types]
    F --> G[Apply ontology constraints]
    G --> H[Beam search over valid parses]
    H --> I[Emit canonical IOTA expression]

Composition rules

The converter should infer at least three relation families.

Order means sequence along an intended reading axis or local compositional axis. For a sequence candidate S = [g1, ..., gn], a workable score is:

order_axis(g) = projection of centroid(g) onto dominant reading axis
order_score(S) = 1 - (1 / Z) * Σ_i |rank_axis(gi) - rank_expected(gi)|

This should be computed after normalization of rotation, scale, and layout direction. For bidirectional or ambiguous layouts, retain multiple hypotheses and let the ontology plus retrieval model disambiguate.

Proximity means components are semantically coupled because they are near each other after scale normalization:

d_norm(a,b) = euclidean(centroid(a), centroid(b)) / sqrt(mean(area(a), area(b)))
proximity_score(a,b) = exp(-d_norm(a,b) / τ)

Use τ as a learned or tuned scale parameter. Maintain both centroid distance and boundary distance, because center-based distance can be misleading for large enclosing shapes.

Containment means one primitive or glyph lies inside another topological region:

inside(a,b) = 1 if bbox(b) ⊆ interior(a) and winding-test passes else 0
containment_score(a,b) =
  inside(a,b) * area(intersection(interior(a), b)) / area(b)

Containment should be favored over mere proximity when both apply, because containment often carries stronger semantic weight in glyph-like systems.

Retrieval and parse assembly

A practical retrieval-plus-parse algorithm looks like this:

1. Build q_visual, q_svg, q_text from the input.
2. Retrieve top-k candidates from each index.
3. Merge candidate sets with reciprocal-rank fusion.
4. Filter out candidates that violate:
   - public-output eligibility
   - ontology type-role compatibility
   - hard composition constraints
5. Construct candidate subgraphs from detected relations.
6. Run beam search over valid parses.
7. Emit:
   - canonical IOTA expression
   - public Unicode rendering
   - evidence trace
   - diagnostics payload

This works especially well with SQL Server because ontology filters and public-output predicates often reduce the candidate set enough that exact kNN can still be used, which Microsoft explicitly recommends for candidate pools below roughly 50,000 vectors. If the filtered pool remains too large or the query pattern becomes too metadata-heavy, Qdrant’s filterable HNSW is the most natural next step.

Attention, rarity, entropy, and phase-lock scoring

The formulas below are proposed converter metrics, not standardized external benchmarks. I recommend storing them as diagnostics and calibration aids rather than letting them directly override ontology or human validation.

rarity_score(g) =
  -log((f_runtime(g) + λ1*f_unicode_block(g) + λ2*f_glyph_family(g) + α) / Z)

candidate_entropy(q) =
  - Σ_i p_i(q) log p_i(q) / log(K)

attention_agreement(g) =
  1 - mean_h JS(attn_h || mean_heads(attn))

phase_lock_score(g) =
  σ(
      w1 * reuse_stability(g)
    + w2 * neighbor_consensus(g)
    + w3 * (1 - embedding_drift(g))
    + w4 * human_agreement(g)
    + w5 * context_stability(g)
    - w6 * contradiction_rate(g)
  )

Implementation notes:

• rarity_score should combine runtime frequency, Unicode-block prior, and family-level prior, not just raw character frequency. This prevents rare but structurally generic marks from being over-rewarded.
• candidate_entropy should be computed over the normalized top-K candidate probabilities after fusion. High entropy is usually a sign that the query is ambiguous or that the symbol is out-of-distribution.
• attention_agreement should compare region-level or primitive-level saliency across heads or across model branches. High head disagreement can be a useful warning signal, especially on dense composite glyphs.
• phase_lock_score is the most novel metric here. It should be interpreted as meaning stability over time: does the same glyph keep converging on the same canonical expression across nearby contexts, repeated queries, and model updates? A high score means the glyph’s meaning is becoming operationally stable inside the converter; a low score means the glyph is still semantically unstable.

Worked transformations

The examples below are design demonstrations. The first uses a public symbol combination shown on the Protocol5 language-converter page; the second shows how containment changes interpretation.

Example A

Input glyph(s): ⚠火 Visual decomposition: warning triangle with exclamation components, followed by a fire ideograph. Semantic embedding neighborhood: warning, hazard, fire, burn, danger. Ontology validation: warning is a valid modifier over a hazard or event concept. Canonical expression: WarningOf(Fire).

{
  "input": "⚠火",
  "visualDecomposition": {
    "glyphs": [
      {
        "surface": "⚠",
        "primitives": ["triangle", "vertical-stem", "dot"],
        "relations": [
          {"type": "contains", "source": "triangle", "target": "vertical-stem"},
          {"type": "contains", "source": "triangle", "target": "dot"}
        ]
      },
      {
        "surface": "火",
        "primitives": ["left-flare", "right-flare", "center-stroke", "falling-strokes"],
        "relations": []
      }
    ],
    "crossGlyphRelations": [
      {"type": "sequence", "order": ["⚠", "火"]},
      {"type": "modifier", "source": "⚠", "target": "火"}
    ]
  },
  "semanticEmbedding": {
    "topNeighbors": [
      {"concept": "warning", "score": 0.98},
      {"concept": "hazard", "score": 0.96},
      {"concept": "fire", "score": 0.95}
    ]
  },
  "ontologyValidated": {
    "canonicalExpression": {
      "type": "Relation",
      "relation": "WarningOf",
      "arguments": [
        {"type": "Concept", "id": "Fire", "confidence": 0.95}
      ]
    },
    "typeCheck": "pass"
  },
  "publicOutput": "⚠火",
  "approximate": true
}

Example B

Input glyph: ◎ Visual decomposition: outer ring, inner dot. Semantic embedding neighborhood: focus, core, target, center, optionally sun depending on context. Ontology validation: in a systems or architecture context, containment semantics should favor CoreWithin(Container) over purely pictorial glosses.

{
  "input": "◎",
  "visualDecomposition": {
    "primitives": [
      {"id": "p1", "type": "outer-circle"},
      {"id": "p2", "type": "inner-dot"}
    ],
    "relations": [
      {"type": "contains", "source": "p1", "target": "p2"}
    ]
  },
  "semanticEmbedding": {
    "topNeighbors": [
      {"concept": "focus", "score": 0.88},
      {"concept": "core", "score": 0.86},
      {"concept": "target", "score": 0.84},
      {"concept": "sun", "score": 0.81}
    ]
  },
  "ontologyValidated": {
    "context": "system architecture",
    "canonicalExpression": {
      "type": "Relation",
      "relation": "CoreWithin",
      "arguments": [
        {"type": "Concept", "id": "Container"},
        {"type": "Concept", "id": "Core"}
      ]
    },
    "typeCheck": "pass"
  },
  "publicOutput": "◎",
  "approximate": true
}

Evaluation strategy

The evaluation plan should be both machine-measurable and human-legible, because Protocol5’s stated philosophy is that approximate outputs should be inspected through scores, evidence, and provenance rather than simply trusted. ISO 9186-1 is the most relevant formal standard here: it specifies a method for testing the comprehensibility of graphical symbols and measures how effectively a symbol communicates its intended message without explanatory text. That maps naturally onto glyph-to-meaning and meaning-to-glyph tests for IOTA-1.

Recommended benchmark lanes

The human testing protocol should include at least three varieties of study. First, no-text glyph comprehension in the ISO 9186 sense: show the symbol without explanation and ask participants to identify or paraphrase the intended meaning. Second, forced-choice disambiguation with semantically near alternatives, not only easy distractors. Third, compositional interpretation where participants judge whether order, containment, or proximity changes meaning in the expected way. ISO 9186 gives the right conceptual frame for the first lane; Protocol5’s evidence-first philosophy makes it natural to log candidate sets and confidence traces for the latter two.

The machine-side evaluation should include ablations across all major branches:

• text-only retrieval;
• rendered-image-only retrieval;
• SVG-only retrieval;
• image + text fusion;
• image + SVG + text fusion;
• image + SVG + text + ontology reranking.

This is especially important because the literature suggests that tokenization and textual description alone do not preserve all of the relevant structure for glyph-like systems. Haslett shows that token boundaries can affect meaning representations for character systems with semantically informative substructure, and Shih et al. show that even strong multimodal models still struggle to generate faithful descriptions of complex glyph tokens and to reason over non-Unicode scripts. The evaluation harness therefore needs to verify that the structural and visual branches are doing real work, not just adding complexity.

Attention diagnostics should be treated as debugging and trust signals, not as the final arbiter of correctness. For every evaluated glyph, store: region saliency maps, primitive-ablation deltas, branch agreement, and candidate entropy. High candidate entropy plus low attention agreement is a useful operational signature for “do not over-trust this result.” High phase-lock plus low entropy is a useful signature for “this meaning is becoming stable.” Since Shih et al. found that inaccurate geometric descriptions are especially common on complex tokens, primitive-ablation tests are more reliable than free-form narrations for determining whether the model depended on the right substructure.

Roadmap, risks, and recommended choices

The roadmap below keeps the public IOTA-1 boundary constant while improving the internals in three deliberate stages. The stack choices follow directly from the current Protocol5 facade and repository design, the current SQL Server vector capabilities, and the capabilities of the recommended model family.

Roadmap milestones

gantt
    title Proposed implementation timeline
    dateFormat  YYYY-MM-DD
    section Phase 1
    Schema and provenance layer        :a1, 2026-05-11, 14d
    SVG parsing and primitive graph    :a2, after a1, 14d
    SQL multi-vector integration       :a3, after a1, 21d
    section Phase 2
    SigLIP 2 integration               :b1, 2026-06-22, 21d
    SVGformer structural encoder       :b2, 2026-06-29, 21d
    Ontology reranker and parser       :b3, 2026-07-13, 28d
    Evaluation harness                 :b4, 2026-07-20, 21d
    section Phase 3
    Human validation tools             :c1, 2026-08-17, 21d
    Active learning and diagnostics    :c2, 2026-08-24, 28d
    Production optimization            :c3, 2026-09-07, 21d

Risk and mitigation map

Recommended choices

The highest-confidence recommendation is:

• Primary vision-language encoder: SigLIP 2
• Primary structural encoder: SVGformer
• Auxiliary visual branch: DINOv2
• Primary serving store in Phase 1–2: SQL Server native vectors with multiple vector columns and late fusion
• Scale-out filtered ANN option: Qdrant
• Offline benchmark and tuning lab: Faiss

This combination best matches the problem and the existing Protocol5 stack. SigLIP 2 is the strongest fit for multilingual image-text retrieval with better localization and dense features; SVGformer is the best direct fit for SVG-native semantics; DINOv2 is the safest auxiliary image backbone; and SQL Server vectors fit the current Protocol5 facade, repository, and ADO.NET shape better than a full early storage migration would. Qdrant is the right second step if ontology-heavy filtered search outgrows SQL ergonomics, and Faiss remains the most useful offline evaluation substrate rather than a serving layer.

The main open questions are operational, not conceptual: how large the first curated glyph inventory should be, how aggressive the latency target must be for the public UI, and whether internal non-rendered SVG assets are formally acceptable as semantic evidence so long as public rendering remains Unicode-only. None of those questions change the central conclusion. The current Protocol5 converter is already a strong public-symbol approximate conversion surface. The next step is to give it a real glyph semantics kernel underneath that surface.