Protocol5 Iota USR Implementation Report

Full report

On this page

Executive summary

The current Iota site already contains the right conceptual direction for a USR-first architecture: it explicitly treats meaning as anchored in concept IDs, canonical vectors, provenance, validation policy, and drift reports rather than in glyphs alone; it also frames language conversion as bounded semantic preservation rather than exact translation. The main problem is that the public site exposes a broader capability surface than the currently reachable backend can support. The strongest public evidence is that the SQL corpus is configured but unreachable on the public host, the category corpus readiness is offline with totalRows: 0, the public site exposes multiple overlapping API families, and several UI surfaces depend on runtime states that are not consistently available. The steganography page is also unusually clear that zero-width transport is fragile and that sidecar/wrapper transport is the safest pattern, which means the public encoder/decoder should be simplified immediately.

The most important recommendation is to make USR the sole canonical system of record across the site and backend. In practice, that means introducing a versioned USR packet that stores: source expression, Unicode normalization, language tags, a UMR-inspired semantic graph, selected concept IDs, provenance, validation status, drift analysis, public rendering candidates, and optional embedding references. The public website should then become a projection layer over that packet: search queries retrieve USR-linked concepts and expressions, the language converter resolves into a USR packet before rendering Iota output, steganography wraps or references the packet without becoming semantic authority, and metadata/versioning flow from temporal tables plus provenance records. This aligns the site’s own stated doctrine with external standards: JSON-LD and RDF for linked semantics, SKOS for concept schemes, PROV for provenance, SHACL for graph validation, BCP 47 for language tags, Unicode NFC for text stability, and RFC 8785 canonical JSON for signatures.

From an implementation standpoint, the safest target is ASP.NET Core on a currently supported .NET LTS release, with SQL Server 2025 used for relational storage, full-text search, optional SQL semantic search, and exact vector similarity via VECTOR_DISTANCE; VECTOR_SEARCH and the SQL native json type are useful, but both need explicit feature-flagging because preview status still matters in SQL Server 2025. The public host should remain read-only, while corpus mutation, bulk embedding, and registry curation stay in internal/admin tooling, which is also consistent with the site’s documented boundary. LM Studio should be used as an internal inference service for embeddings, structured-output graph construction, and validation/reranking, but the user-facing site must continue to function in DatabaseOnly mode when live AI is unavailable.

A realistic roadmap is a phased rebuild rather than an incremental patching exercise. In the first phase, remove or demote brittle public features, especially zero-width public encoding and SQL-dependent public controls that currently sit atop an offline corpus. In the second phase, introduce the USR schema, canonical packet, and API facade while keeping compatibility shims for legacy /api/iota/* and /api/justaniota/* routes. In the third phase, redesign the language converter and steganography tools as renderers and wrappers over USR rather than as separate meaning systems. In the final phase, cut over search, authoring, telemetry, and migration logic, then validate round-trip quality, multilingual fidelity, and Unicode safety. For a small team of roughly four to six people, the overall effort is more like sixteen to twenty-four weeks than a quick patch release.

Current site audit

The public Iota overview, language converter, universal semantics, evidence workbench, and steganography pages are conceptually aligned around a consistent doctrine: approximation rather than exact translation, registry-backed concept IDs, Unicode public-symbol safety, visible drift, and evidence-rich responses. That is a strong foundation. The problems are mostly operational and product-structural: the UI surface is larger than the current backend readiness, the API surface is fragmented, and some public controls appear to expose lab-stage functionality that should either be hidden, clearly marked experimental, or replaced by more stable equivalents.

The most concrete functionality issue is backend readiness. The public /api/iota/status response shows sqlConfigured: true but reachable: false, readiness: "offline", totalRows: 0, and embeddedRows: 0. At the same time, the evidence pages and protocol rules describe SQL-backed corpus search, ranking demos, stored vectors, category search, and SQL-first ranking paths. That mismatch is enough to treat SQL-backed public controls as nonworking from a user perspective until the corpus is reachable and populated. It also means the public site is currently leaning on seed-registry and fallback behavior more than its UI suggests.

There is also public API inconsistency. /api/iota/status reports publicSeedConcepts: 56 and modes including semantic-hybrid and semantic-interlingua, while /api/justaniota/status reports publicSeedConcepts: 36 and only database-only, llm-assisted, and hybrid. The OpenAPI route summary lists both /api/iota/* and /api/justaniota/* families. That dual-surface design may reflect product history, but it creates confusion for client developers, complicates documentation, and increases the chance of divergent behavior.

UX is the second major issue. The site’s theory pages are clear for a technically engaged reader, but the workbench pages expose many controls at once: source language, locale, several modes, evidence toggles, vector previews, quantized code, category search, validator, concept bridge, and hidden-transport scan. That is appropriate for a lab console, but not for a public-facing converter page. The evidence workbench also shows a broken rendering of the intended sample (好救 ⚠火), which indicates a real encoding-quality problem in sample data or rendering and directly undercuts the site’s Unicode-safety claims.

The security posture is partly strong in doctrine and partly incomplete in public evidence. Strongly positive signals include explicit rejection of private-use characters, hidden bidi controls, unsafe zero-width patterns, and homoglyph substitution as semantic authority; the site also says public routes are read-only and that embedding population belongs to local/admin tooling rather than public HTTP routes. Those are exactly the right boundaries. But the public crawl does not expose enough HTTP header detail to score CSP, HSTS, cookie flags, or CORS policy, so the site cannot be considered fully audited at the transport-security layer from the public evidence alone.

Performance appears to be less about front-end weight and more about readiness and dependency management. The pages are largely content-heavy, but the interactive surfaces rely on client-side state and runtime health checks. Protocol5’s public status currently reports a vector dimension of 1998, which is notable because SQL Server 2025’s VECTOR type allows a maximum of 1998 dimensions. That does not prove the current design is wrong, but it strongly suggests the implementation may be pinned to the engine ceiling rather than to an intentionally selected embedding profile. In a USR system, model dimension should be an explicit profile choice, not an implicit database maximum.

The table below is the recommended prioritized remove/replace plan, based on the public site’s own doctrine and current operational evidence.

USR framework and standards

Protocol5 is already describing a project-specific USR idea even if the site does not yet formalize it as a complete standard. The site’s own formulation is that “universal” means a reviewable target of convergence, not an exact universal language, with authority living in concept IDs, canonical vectors, provenance, validators, and drift reports. That is a sound position, and it should be kept. The implementation mistake to avoid is turning glyphs, quantized codes, or model outputs back into the system of record.

For a rigorous implementation, USR should be defined as a layered contract rather than a single file format. The semantic core should take direct inspiration from Uniform Meaning Representation. UMR explicitly combines a sentence-level representation adapted from AMR with document-level coreference, temporal, and modal relations, and it is intended as a cross-lingual annotation framework with a shared set of abstract concepts and relations. Recent UMR work also shows that the ecosystem is maturing: there are now papers on UMR-to-text generation, UMR-assisted translation prompts for Indigenous languages, sentence-level UMR parsing, and aspect enrichment in structured meaning representations.

That semantic core should then be coupled to web and metadata standards rather than reinventing ontology infrastructure. RDF 1.2 provides a graph-and-dataset abstract model; JSON-LD 1.1 gives an interoperable JSON serialization for linked data; SKOS provides a concept-centric vocabulary model for preferred labels, alternate labels, broader/narrower/related relations, and concept schemes; PROV supplies a provenance model; and SHACL provides constraint validation over RDF graphs. For language and text stability, BCP 47 language tags and Unicode NFC should be mandatory parts of every packet. For signing and canonical hashing, JSON Canonicalization Scheme is the cleanest option.

The site’s own internal vocabulary already suggests the right USR field families: source expression, normalization, source language, locale, phrase segments, concept candidates, selected concept, canonical vector hash, provenance, public rendering candidates, Unicode safety checks, and drift. The recommendation is therefore not to replace Protocol5’s semantic doctrine, but to normalize it into a formal packet with specific versioning and validation guarantees.

A practical USR packet for Protocol5 should contain the following conceptual layers:

This layered design is also supported by multilingual semantic-representation research beyond UMR. The Parallel Meaning Bank uses a language-neutral, layered annotation process including segmentation, symbolization, semantic tagging, word sense disambiguation, syntax, thematic roles, and coreference; MASSIVE-AMR shows that multilingual graph-style meaning representations can now be built across 50+ languages at much larger scale than older resources. Those studies strengthen the case for a graph-centered USR core rather than a glyph-first or translation-first architecture.

For serialization, the best implementation choice is not a single winner but a primary-plus-secondary pattern. Use canonical JSON as the operational packet and signing unit, optionally stored in SQL Server’s json type when preview adoption is acceptable; expose JSON-LD for linked-data interoperability; and support RDF/Turtle export for ontology and validation tooling. The SQL Server 2025 native json type is attractive because it stores JSON in a native binary format and is optimized for querying and manipulation, but it is still preview on SQL Server 2025, so production rollout should keep a portability fallback to nvarchar(max) plus validation checks.

Backend architecture, SQL schema, and C# patterns

The recommended backend is ASP.NET Core on .NET 10 if this is a meaningful rebuild rather than a narrow patch. Microsoft’s support policy shows .NET 10 as the active LTS release supported until November 14, 2028, whereas .NET 8 remains supported only until November 10, 2026. If hosting constraints force short-horizon compatibility work, .NET 8 is still defensible, but for a USR-centered redesign with schema, API, and pipeline changes, .NET 10 is the better target.

On the database side, SQL Server 2025 now has the core primitives needed for this design. The VECTOR type stores vectors in optimized binary form while exposing them as JSON arrays for convenience; the maximum supported dimension count is 1998. VECTOR_DISTANCE provides exact similarity search and does not use vector indexes; VECTOR_SEARCH is the approximate-search path, but it is preview in SQL Server 2025 and should not become the correctness baseline. Full-Text Search remains the right lexical engine for phrase, prefix, inflectional, and thesaurus-style matching, and SQL’s older Statistical Semantic Search is usable for document-level keyphrase and related-document discovery, but it is not a substitute for a USR graph. Temporal tables and rowversion complete the storage picture for history and optimistic concurrency.

The resulting architecture should look like this:

flowchart LR
    A[Public Web UI] --> B[ASP.NET Core API Facade]
    B --> C[USR Resolver Service]
    B --> D[Renderer Service]
    B --> E[Stego Scan Service]
    C --> F[SQL Server 2025]
    C --> G[LM Studio Internal Service]
    D --> F
    E --> F
    H[Admin Ingestion Worker] --> G
    H --> F
    F --> I[Temporal History]
    F --> J[Full-Text Index]
    F --> K[Vector Columns]

The public facade should remain read-only. That is consistent both with the site’s current OpenAPI statement that no public endpoint mutates SQL or persisted embeddings, and with the protocol rule that embedding generation and corpus mutation belong in local/admin tooling.

A good relational model is a hybrid model: relationalize the fields you must query, keep the complete packet as JSON, and store embeddings separately by profile. The schema below is representative rather than mandatory, but it is specific enough to implement.

CREATE SCHEMA usr;
GO

CREATE TABLE usr.EmbeddingProfile
(
    EmbeddingProfileId       INT IDENTITY(1,1) PRIMARY KEY,
    ProfileCode              NVARCHAR(100) NOT NULL UNIQUE,
    ModelId                  NVARCHAR(200) NOT NULL,
    Dimensions               INT NOT NULL,
    DistanceMetric           NVARCHAR(20) NOT NULL,   -- cosine, euclidean, dot
    PromptProfile            NVARCHAR(100) NULL,
    IsActive                 BIT NOT NULL DEFAULT(1),
    CreatedUtc               DATETIME2(7) NOT NULL DEFAULT SYSUTCDATETIME()
);
GO

CREATE TABLE usr.Concept
(
    ConceptUid               UNIQUEIDENTIFIER NOT NULL DEFAULT NEWSEQUENTIALID(),
    ConceptCode              NVARCHAR(64) NOT NULL,   -- e.g. C0001842
    SchemeIri                NVARCHAR(400) NULL,
    PreferredLabel           NVARCHAR(400) NOT NULL,
    DefinitionText           NVARCHAR(MAX) NULL,
    ConceptJson              JSON NULL,
    CanonicalVector          VECTOR(1536) NULL,
    CanonicalVectorHash      BINARY(32) NULL,
    EmbeddingProfileId       INT NULL,
    IsPublic                 BIT NOT NULL DEFAULT(1),
    IsActive                 BIT NOT NULL DEFAULT(1),
    ValidFrom                DATETIME2 GENERATED ALWAYS AS ROW START NOT NULL,
    ValidTo                  DATETIME2 GENERATED ALWAYS AS ROW END NOT NULL,
    Rv                       ROWVERSION NOT NULL,
    PERIOD FOR SYSTEM_TIME (ValidFrom, ValidTo),
    CONSTRAINT PK_Concept PRIMARY KEY CLUSTERED (ConceptUid),
    CONSTRAINT UQ_Concept_ConceptCode UNIQUE (ConceptCode),
    CONSTRAINT FK_Concept_EmbeddingProfile
        FOREIGN KEY (EmbeddingProfileId) REFERENCES usr.EmbeddingProfile(EmbeddingProfileId)
)
WITH (SYSTEM_VERSIONING = ON (HISTORY_TABLE = usr.ConceptHistory));
GO

CREATE TABLE usr.ConceptEdge
(
    ParentConceptUid         UNIQUEIDENTIFIER NOT NULL,
    ChildConceptUid          UNIQUEIDENTIFIER NOT NULL,
    EdgeType                 NVARCHAR(40) NOT NULL,   -- broader, narrower, related, exactMatch
    ProvenanceJson           JSON NULL,
    CreatedUtc               DATETIME2(7) NOT NULL DEFAULT SYSUTCDATETIME(),
    CONSTRAINT PK_ConceptEdge PRIMARY KEY (ParentConceptUid, ChildConceptUid, EdgeType),
    CONSTRAINT FK_ConceptEdge_Parent FOREIGN KEY (ParentConceptUid) REFERENCES usr.Concept(ConceptUid),
    CONSTRAINT FK_ConceptEdge_Child  FOREIGN KEY (ChildConceptUid)  REFERENCES usr.Concept(ConceptUid)
);
GO

CREATE TABLE usr.Expression
(
    ExpressionUid            UNIQUEIDENTIFIER NOT NULL DEFAULT NEWSEQUENTIALID(),
    ConceptUid               UNIQUEIDENTIFIER NOT NULL,
    SurfaceText              NVARCHAR(2000) NOT NULL,
    LanguageTag              NVARCHAR(35) NOT NULL,   -- BCP 47
    LocaleTag                NVARCHAR(35) NULL,
    ScriptTag                NVARCHAR(10) NULL,
    NormalizationForm        NVARCHAR(8) NOT NULL DEFAULT N'NFC',
    DirectionTag             NVARCHAR(8) NULL,
    IsPreferred              BIT NOT NULL DEFAULT(0),
    IsNegativeExample        BIT NOT NULL DEFAULT(0),
    DisplayRank              SMALLINT NULL,
    SearchText               NVARCHAR(MAX) NULL,
    ExpressionJson           JSON NULL,
    SearchVector             VECTOR(1536) NULL,
    SearchVectorHash         BINARY(32) NULL,
    EmbeddingProfileId       INT NULL,
    Rv                       ROWVERSION NOT NULL,
    CONSTRAINT PK_Expression PRIMARY KEY CLUSTERED (ExpressionUid),
    CONSTRAINT FK_Expression_Concept FOREIGN KEY (ConceptUid) REFERENCES usr.Concept(ConceptUid),
    CONSTRAINT FK_Expression_EmbeddingProfile
        FOREIGN KEY (EmbeddingProfileId) REFERENCES usr.EmbeddingProfile(EmbeddingProfileId)
);
GO

CREATE TABLE usr.Packet
(
    PacketUid                UNIQUEIDENTIFIER NOT NULL DEFAULT NEWSEQUENTIALID(),
    ProfileCode              NVARCHAR(100) NOT NULL,
    SourceText               NVARCHAR(MAX) NOT NULL,
    SourceLanguageTag        NVARCHAR(35) NOT NULL,
    SourceLocaleTag          NVARCHAR(35) NULL,
    SourceNormalization      NVARCHAR(8) NOT NULL DEFAULT N'NFC',
    SourceTextSha256         BINARY(32) NOT NULL,
    SelectedConceptUid       UNIQUEIDENTIFIER NULL,
    ConfidenceScore          DECIMAL(9,6) NULL,
    UnknownRate              DECIMAL(9,6) NULL,
    DriftJson                JSON NULL,
    PacketJson               JSON NOT NULL,
    PacketJcsSha256          BINARY(32) NOT NULL,
    CreatedUtc               DATETIME2(7) NOT NULL DEFAULT SYSUTCDATETIME(),
    ValidFrom                DATETIME2 GENERATED ALWAYS AS ROW START NOT NULL,
    ValidTo                  DATETIME2 GENERATED ALWAYS AS ROW END NOT NULL,
    Rv                       ROWVERSION NOT NULL,
    PERIOD FOR SYSTEM_TIME (ValidFrom, ValidTo),
    CONSTRAINT PK_Packet PRIMARY KEY CLUSTERED (PacketUid),
    CONSTRAINT FK_Packet_SelectedConcept FOREIGN KEY (SelectedConceptUid) REFERENCES usr.Concept(ConceptUid)
)
WITH (SYSTEM_VERSIONING = ON (HISTORY_TABLE = usr.PacketHistory));
GO

CREATE TABLE usr.RenderingCandidate
(
    RenderingCandidateUid    UNIQUEIDENTIFIER NOT NULL DEFAULT NEWSEQUENTIALID(),
    PacketUid                UNIQUEIDENTIFIER NOT NULL,
    ConceptUid               UNIQUEIDENTIFIER NULL,
    RenderingKind            NVARCHAR(30) NOT NULL,   -- iota-visible, english-gist, multilingual
    RenderingText            NVARCHAR(2000) NOT NULL,
    UnicodeSafetyJson        JSON NULL,
    RankScore                DECIMAL(9,6) NULL,
    EvidenceJson             JSON NULL,
    CONSTRAINT PK_RenderingCandidate PRIMARY KEY CLUSTERED (RenderingCandidateUid),
    CONSTRAINT FK_RenderingCandidate_Packet FOREIGN KEY (PacketUid) REFERENCES usr.Packet(PacketUid)
);
GO

CREATE NONCLUSTERED INDEX IX_Expression_Concept_Language
ON usr.Expression (ConceptUid, LanguageTag, IsPreferred DESC, DisplayRank ASC);

CREATE NONCLUSTERED INDEX IX_Packet_SelectedConcept_CreatedUtc
ON usr.Packet (SelectedConceptUid, CreatedUtc DESC);

CREATE FULLTEXT CATALOG usr_ft AS DEFAULT;
CREATE FULLTEXT INDEX ON usr.Expression
(
    SurfaceText LANGUAGE 1033,
    SearchText LANGUAGE 1033
)
KEY INDEX PK_Expression;
GO

This schema deliberately keeps the packet JSON and graph export alongside query-friendly relational columns. That is the right balance because SQL Server’s native json type has no dedicated JSON index type, cannot be used as an index key, and is still preview on SQL Server 2025, while relational fields and full-text indexes are mature query paths.

The query strategy should be hybrid in a very literal sense. Use full-text for expression lookup and exact phrases, use VECTOR_DISTANCE for exact semantic ranking, and optionally layer in VECTOR_SEARCH behind a feature flag only after recall has been benchmarked against exact search. SQL’s document-level semantic search can be useful for “related page” features or automatic tags on long-form content, but not as the meaning authority for converter output.

A representative exact hybrid retrieval query looks like this:

DECLARE @query_vector VECTOR(1536) = @InputVector;

WITH Lexical AS
(
    SELECT TOP (200)
           e.ExpressionUid,
           e.ConceptUid,
           KEY_TBL.RANK AS FtRank
    FROM usr.Expression AS e
    INNER JOIN CONTAINSTABLE
    (
        usr.Expression,
        (SurfaceText, SearchText),
        @FullTextQuery
    ) AS KEY_TBL
        ON KEY_TBL.[KEY] = e.ExpressionUid
),
Semantic AS
(
    SELECT TOP (200)
           e.ExpressionUid,
           e.ConceptUid,
           VECTOR_DISTANCE('cosine', e.SearchVector, @query_vector) AS DistanceScore
    FROM usr.Expression AS e
    WHERE e.SearchVector IS NOT NULL
    ORDER BY VECTOR_DISTANCE('cosine', e.SearchVector, @query_vector) ASC
)
SELECT TOP (50)
       COALESCE(l.ExpressionUid, s.ExpressionUid) AS ExpressionUid,
       COALESCE(l.ConceptUid, s.ConceptUid) AS ConceptUid,
       l.FtRank,
       s.DistanceScore
FROM Lexical AS l
FULL OUTER JOIN Semantic AS s
    ON l.ExpressionUid = s.ExpressionUid
ORDER BY
    CASE WHEN l.FtRank IS NULL THEN 0 ELSE 1 END DESC,
    ISNULL(l.FtRank, 0) DESC,
    ISNULL(s.DistanceScore, 999.0) ASC;

On the C# side, the most important pattern is to treat USR as an immutable domain object and persist it through an explicit repository boundary, using native SqlVector when available and JSON fallback otherwise. SQL Server’s vector docs now explicitly state that Microsoft.Data.SqlClient 6.1.0 introduces SqlVector, while older clients can still read and write vectors as JSON arrays.

using System.Data;
using System.Security.Cryptography;
using System.Text;
using System.Text.Json;
using Microsoft.Data.SqlClient;
using Microsoft.Data.SqlTypes;

public sealed record UsrPacket(
    Guid PacketUid,
    string ProfileCode,
    string SourceText,
    string SourceLanguageTag,
    string? SourceLocaleTag,
    string SourceNormalization,
    string PacketJson,
    byte[] PacketJcsSha256,
    Guid? SelectedConceptUid,
    decimal? ConfidenceScore,
    decimal? UnknownRate);

public sealed class UsrPacketRepository
{
    private readonly string _connectionString;

    public UsrPacketRepository(string connectionString)
    {
        _connectionString = connectionString;
    }

    public async Task<Guid> SavePacketAsync(
        UsrPacket packet,
        float[]? sourceEmbedding,
        CancellationToken cancellationToken)
    {
        using SqlConnection connection = new SqlConnection(_connectionString);
        await connection.OpenAsync(cancellationToken);

        using SqlCommand command = connection.CreateCommand();
        command.CommandText = @"
INSERT INTO usr.Packet
(
    PacketUid,
    ProfileCode,
    SourceText,
    SourceLanguageTag,
    SourceLocaleTag,
    SourceNormalization,
    SourceTextSha256,
    SelectedConceptUid,
    ConfidenceScore,
    UnknownRate,
    PacketJson,
    PacketJcsSha256
)
VALUES
(
    @PacketUid,
    @ProfileCode,
    @SourceText,
    @SourceLanguageTag,
    @SourceLocaleTag,
    @SourceNormalization,
    @SourceTextSha256,
    @SelectedConceptUid,
    @ConfidenceScore,
    @UnknownRate,
    @PacketJson,
    @PacketJcsSha256
);";

        command.Parameters.Add(new SqlParameter("@PacketUid", SqlDbType.UniqueIdentifier) { Value = packet.PacketUid });
        command.Parameters.Add(new SqlParameter("@ProfileCode", SqlDbType.NVarChar, 100) { Value = packet.ProfileCode });
        command.Parameters.Add(new SqlParameter("@SourceText", SqlDbType.NVarChar, -1) { Value = packet.SourceText });
        command.Parameters.Add(new SqlParameter("@SourceLanguageTag", SqlDbType.NVarChar, 35) { Value = packet.SourceLanguageTag });
        command.Parameters.Add(new SqlParameter("@SourceLocaleTag", SqlDbType.NVarChar, 35) { Value = (object?)packet.SourceLocaleTag ?? DBNull.Value });
        command.Parameters.Add(new SqlParameter("@SourceNormalization", SqlDbType.NVarChar, 8) { Value = packet.SourceNormalization });
        command.Parameters.Add(new SqlParameter("@SourceTextSha256", SqlDbType.Binary, 32) { Value = SHA256.HashData(Encoding.UTF8.GetBytes(packet.SourceText)) });
        command.Parameters.Add(new SqlParameter("@SelectedConceptUid", SqlDbType.UniqueIdentifier) { Value = (object?)packet.SelectedConceptUid ?? DBNull.Value });
        command.Parameters.Add(new SqlParameter("@ConfidenceScore", SqlDbType.Decimal) { Precision = 9, Scale = 6, Value = (object?)packet.ConfidenceScore ?? DBNull.Value });
        command.Parameters.Add(new SqlParameter("@UnknownRate", SqlDbType.Decimal) { Precision = 9, Scale = 6, Value = (object?)packet.UnknownRate ?? DBNull.Value });
        command.Parameters.Add(new SqlParameter("@PacketJson", SqlDbType.NVarChar, -1) { Value = packet.PacketJson });
        command.Parameters.Add(new SqlParameter("@PacketJcsSha256", SqlDbType.Binary, 32) { Value = packet.PacketJcsSha256 });

        await command.ExecuteNonQueryAsync(cancellationToken);
        return packet.PacketUid;
    }

    public async Task<float[]?> ReadVectorFallbackAsync(Guid expressionUid, CancellationToken cancellationToken)
    {
        using SqlConnection connection = new SqlConnection(_connectionString);
        await connection.OpenAsync(cancellationToken);

        using SqlCommand command = connection.CreateCommand();
        command.CommandText = "SELECT CAST(SearchVector AS NVARCHAR(MAX)) FROM usr.Expression WHERE ExpressionUid = @ExpressionUid";
        command.Parameters.Add(new SqlParameter("@ExpressionUid", SqlDbType.UniqueIdentifier) { Value = expressionUid });

        object? raw = await command.ExecuteScalarAsync(cancellationToken);
        if (raw is null || raw == DBNull.Value)
        {
            return null;
        }

        string json = (string)raw;
        return JsonSerializer.Deserialize<float[]>(json);
    }
}

If the deployment is fully on current drivers, add a vector-optimized repository path using SqlVector<float> and GetSqlVector<float>(); if not, keep the JSON fallback path available because SQL Server explicitly preserves backward compatibility by exposing vectors as JSON arrays to older clients.

LM Studio pipelines, steganography, and language-converter redesign

LM Studio is well suited to this architecture, but it should be used as an internal model host, not as the public system of record. Official docs show three particularly important capabilities: openAI-compatible /v1/* endpoints for chat, responses, and embeddings; JSON-schema structured output on /v1/chat/completions; and headless/server-native operation via llmster, with just-in-time model loading, idle TTL, and auto-eviction. The docs also explicitly recommend the native v1 REST API for new projects, while the older v0 API remains useful because it exposes token/sec and TTFT stats that are valuable for benchmarking and telemetry.

The model-hosting rule should therefore be: public ASP.NET app talks only to an internal LM Studio service; DatabaseOnly mode remains fully functional without live AI; and admin ingestion workers are the only place where bulk embedding, re-embedding, or corpus mutation occur. That preserves the site’s own doctrine and keeps public outages from collapsing the converter.

A recommended inference/dataflow pattern is:

flowchart TD
    A[Input text + locale] --> B[Normalize NFC + tag language]
    B --> C[Segment into expressions]
    C --> D[LM Studio embeddings]
    D --> E[Exact SQL VECTOR_DISTANCE retrieval]
    C --> F[Full-Text retrieval]
    E --> G[Concept candidate merge]
    F --> G
    G --> H[LM Studio structured USR packet]
    H --> I[USR validator]
    I --> J[Iota renderer]
    I --> K[Natural language renderer]
    H --> L[Optional signed sidecar]
    J --> M[Public response with drift + evidence]
    K --> M
    L --> M

For model selection, the most robust approach is role-based rather than “one model does everything.” LM Studio’s own docs note that not all models are capable of structured output, especially models below 7B. The examples in the developer docs use a Qwen2.5 7B instruct model for tool use and text-embedding-nomic-embed-text-v1.5 for embeddings, which makes those reasonable baselines. The right production choice is still empirical benchmarking on your corpus, but the pipeline roles are clear.

A representative LM Studio configuration for internal deployment looks like this:

lmstudio:
  mode: headless
  startup:
    command: "lms daemon up"
  api:
    public_exposure: false
    internal_base_url: "http://lmstudio-internal:1234/v1"
  models:
    usr_structured:
      identifier: "7b-or-larger-instruct-model"
      ttl_seconds: 900
    embeddings:
      identifier: "text-embedding-nomic-embed-text-v1.5"
      ttl_seconds: 3600
  jit_loading: true
  auto_evict: true
  auth:
    required: true
  telemetry:
    sample_v0_calls_for_ttft: true

LM Studio’s documented defaults make this practical: JIT loading is enabled, idle TTL defaults to 60 minutes, and auto-evict is enabled by default for JIT-loaded models. Headless llmster is the recommended deployment mode, especially for Linux servers or background services.

Structured output should be mandatory for USR generation, because it reduces parser fragility and makes validation deterministic. LM Studio supports JSON-schema constrained output and, for GGUF models, uses grammar-based sampling through llama.cpp; for MLX it uses Outlines. That is exactly what you want for a semantic packet generator.

A useful prompt pattern is:

{
  "model": "usr-structured-model",
  "messages": [
    {
      "role": "system",
      "content": "Convert the input into a Protocol5 USR packet. Do not translate unless asked. Preserve uncertainty. Use concept IDs only when supported by evidence. Mark retained, lost, added, and ambiguous meaning separately."
    },
    {
      "role": "user",
      "content": "text=Hola\nlanguage=es\nlocale=es-ES\nmode=semantic-interlingua\npublic_unicode_only=true"
    }
  ],
  "response_format": {
    "type": "json_schema",
    "json_schema": {
      "name": "protocol5_usr_packet",
      "strict": true,
      "schema": {
        "type": "object",
        "properties": {
          "profile": { "type": "string" },
          "source": {
            "type": "object",
            "properties": {
              "text": { "type": "string" },
              "language": { "type": "string" },
              "locale": { "type": "string" },
              "normalization": { "type": "string" }
            },
            "required": ["text", "language", "normalization"]
          },
          "selectedConcept": {
            "type": "object",
            "properties": {
              "id": { "type": "string" },
              "confidence": { "type": "number" }
            },
            "required": ["id"]
          },
          "drift": {
            "type": "object",
            "properties": {
              "retained": { "type": "array", "items": { "type": "string" } },
              "lost": { "type": "array", "items": { "type": "string" } },
              "added": { "type": "array", "items": { "type": "string" } },
              "ambiguous": { "type": "array", "items": { "type": "string" } }
            },
            "required": ["retained", "lost", "added", "ambiguous"]
          }
        },
        "required": ["profile", "source", "selectedConcept", "drift"]
      }
    }
  }
}

The steganography redesign should follow the site’s own strongest statement: steganography comes after semantic conversion, and hidden transport data is never semantic authority. That single rule should drive the whole rebuild. The current public page already ranks transport methods sensibly: markup sidecar is high-fit, zero-width postfix is only medium-fit and fragile, whitespace is low-fit, homoglyph substitution is very low-fit and should be rejected, and semantic paraphrase is experimental rather than reversible.

That means the public-feature redesign is straightforward. The public site should expose only: visible conversion, signed sidecar export/import, scanner/stripper for hidden carriers, and carrier reports. Public “hide message in text” affordances should be removed. If you keep stego encoding at all, it belongs in authenticated lab tooling with stringent constraints and clear warnings that the authoritative record is the external USR packet.

The appropriate stego options for USR are:

An implementation note worth making explicit: if you must support reversible file-based embedding, prefer a profile like usr-sidecar+png-rdh-v1, where the visible/cover asset remains a transport wrapper and the same packet hash exists both in the relational DB and in the embedded payload. That makes extraction verifiable and lets you treat stego as a redundant transport, not as a hidden alternate semantics layer.

The language converter redesign should also become much simpler once anchored in USR. Today’s public page already says the converter output is an approximate semantic neighbor, not exact translation, and that a stronger result preserves the useful relation between expression, concept, candidate glyph, validator behavior, and provenance under a declared policy. The redesign should therefore be: resolve text to USR; render Iota from USR; optionally regenerate natural language from USR; and compute round-trip drift on the USR graph, not just on surface strings.

A USR-centered converter should report at least five user-facing artifacts on every call. The first is the normalized source expression and locale metadata. The second is the selected concept set or graph. The third is the top-K candidate evidence with abstention when necessary. The fourth is the public rendering candidate list for Iota and/or target language. The fifth is an explicit round-trip-drift report split into retained, lost, added, and ambiguous meaning. That last requirement is particularly well aligned with both Protocol5’s own “drift stays visible” doctrine and contemporary UMR work, where document-level relations, aspect, modality, and coreference matter to meaning beyond token substitution.

Migration, testing, deployment, and security

The migration strategy should be parallel, versioned, and reversible. Do not replace the current Iota tables and endpoints in place. Stand up the USR schema alongside the legacy category tables, backfill concepts and expressions from the existing seed registry and SQL corpus, begin dual-writing new packets, then move the public facade over only when retrieval parity and drift metrics are acceptable. This approach also respects the site’s documented split between public read-only routes and local/admin population tooling.

A practical database migration sequence looks like this:

-- Example: create a legacy-to-USR mapping table first
CREATE TABLE usr.LegacyMap
(
    LegacySource          NVARCHAR(100) NOT NULL,
    LegacyKey             NVARCHAR(200) NOT NULL,
    ConceptUid            UNIQUEIDENTIFIER NOT NULL,
    CreatedUtc            DATETIME2(7) NOT NULL DEFAULT SYSUTCDATETIME(),
    CONSTRAINT PK_LegacyMap PRIMARY KEY (LegacySource, LegacyKey)
);
GO

-- Example: backfill concept rows from a legacy category table
INSERT INTO usr.Concept
(
    ConceptUid,
    ConceptCode,
    PreferredLabel,
    DefinitionText,
    IsPublic,
    IsActive
)
SELECT
    NEWSEQUENTIALID(),
    CONCAT(N'LEGACY-', CAST(c.CategoryId AS NVARCHAR(50))),
    c.CategoryName,
    c.Description,
    1,
    1
FROM Category.Categories AS c
WHERE NOT EXISTS
(
    SELECT 1
    FROM usr.LegacyMap AS m
    WHERE m.LegacySource = N'Category.Categories'
      AND m.LegacyKey = CAST(c.CategoryId AS NVARCHAR(200))
);
GO

-- Example: backfill expressions from legacy words table
INSERT INTO usr.Expression
(
    ExpressionUid,
    ConceptUid,
    SurfaceText,
    LanguageTag,
    NormalizationForm,
    IsPreferred,
    SearchText
)
SELECT
    NEWSEQUENTIALID(),
    m.ConceptUid,
    w.WordText,
    N'en',
    N'NFC',
    0,
    w.WordText
FROM Category.Words AS w
INNER JOIN usr.LegacyMap AS m
    ON m.LegacySource = N'Category.Categories'
   AND m.LegacyKey = CAST(w.CategoryId AS NVARCHAR(200));
GO

At the API layer, introduce a new canonical route family, ideally something like /api/usr/v1/*, and make the old routes thin adapters. The adapter rules should be simple: /api/iota/convert resolves to USR then renders Iota-visible output; /api/iota/round-trip resolves to USR, regenerates English, and returns drift details; /api/iota/steganography/scan remains a scanner and never becomes an encode API again. Legacy /api/justaniota/* routes should either forward to the same facade or be documented as compatibility mode with an explicit sunset header. The public OpenAPI surface should then document one authoritative packet shape.

Testing should be centered on semantics, not only on code coverage. The unit-test suite should cover Unicode normalization, grapheme segmentation, BCP 47 tag validation, concept resolution policies, JSON canonicalization, and carrier scanning. The integration suite should cover SQL vector insert/read behavior, full-text query behavior, temporal-table history, LM Studio structured-output validation, and failure-mode fallbacks when SQL or live AI are unavailable. The semantic QA suite should measure top-K concept hit rate, unknown rate, retained/lost/added/ambiguous drift, multilingual round-trip quality, and reviewer accept/revise/abstain/reject outcomes, which closely matches the metrics the site already says it wants to track.

Deployment should separate concerns aggressively. Put the ASP.NET application and public API behind a standard reverse proxy and WAF. Place SQL Server 2025 on its own data tier. Run LM Studio internally in headless mode via llmster, not on the public edge. Use health probes for SQL reachability, index freshness, seed-registry availability, LM Studio model readiness, and structured-output validator status. Cache those health states so the public UI can show truthful capability status without hanging on live checks. LM Studio’s JIT, TTL, and auto-evict controls make this feasible even on constrained hardware.

The monitoring plan should include both standard service telemetry and domain metrics. Standard service telemetry is latency, error rate, SQL wait time, queue depth, and 5xx rate. Domain telemetry should include unknown-rate percentage, top-K concept agreement, drift category counts, Unicode-safety warnings, carrier-scan detections, and renderer abstentions. For LM Studio, sample v0 benchmarking calls during staging or low-rate shadow traffic to capture TTFT and tokens/sec, then correlate those with packet complexity and prompt length.

Security and privacy should be handled in layers. At rest, SQL Server TDE should be enabled for the entire database and backup chain. For especially sensitive user metadata or PII, Always Encrypted can be appropriate, but not for vector columns or semantic fields that need ranking and flexible query behavior; Microsoft’s docs explicitly warn about operational restrictions and show that deterministic encryption leaks patterns while randomized encryption limits query capabilities unless enclaves are used. In the web tier, ASP.NET Core Data Protection should protect cookies, sidecar tokens, and any short-lived trusted round-trip state.

Model privacy is a distinct risk. Recent academic work on local AI artifacts found that runners including LM Studio can leave recoverable plaintext prompt histories, configuration traces, and model usage artifacts on disk. Because this architecture will process multilingual input that may include sensitive content, LM Studio data directories should live on encrypted storage, retention should be minimized, and prompts/logs should be redacted or hashed where operationally possible. Never expose LM Studio directly to the public internet; terminate access at the ASP.NET facade and require internal API tokens between services.

Effort, phased roadmap, risks, and open questions

The roadmap below assumes a team of roughly one product engineer, two backend engineers, one front-end engineer, one ML/NLP engineer, and part-time linguistic/reviewer support. If the same people are also maintaining the live site and corpus curation process, use the longer end of the ranges.

The largest risks are not ordinary CRUD risks; they are semantic and operational. The first risk is false certainty: the system may present clean-looking output when the concept match is under-evidenced. The mitigation is compulsory drift and abstention reporting, plus reviewer feedback loops. The second risk is schema lock-in around preview SQL features. The mitigation is to treat VECTOR_SEARCH and the SQL json type as optional enhancements, while the relational core, exact VECTOR_DISTANCE, and JSON-string fallback remain fully supported. The third risk is Unicode spoofing and hidden-carrier abuse. The mitigation is mandatory TR39-style scanning, strict NFC normalization, public scan/strip tools, and rejection of homoglyph-based channels. The fourth risk is LM Studio artifact leakage. The mitigation is internal-only deployment, storage encryption, token-gated access, and log minimization. The fifth risk is migration drift between legacy category tables and USR concepts. The mitigation is a dedicated legacy mapping table, dual-read validation, and packet-hash reconciliation before cutover.

The final architectural recommendation is therefore very concrete. Keep the site’s theory. Formalize it. Put a UMR-inspired USR packet at the center. Make SQL Server 2025 the durable store for concepts, expressions, packets, provenance, and exact vectors. Use LM Studio internally for embeddings and structured-output graph generation, but never let it become the authority layer. Replace public stego encoding with sidecar-first transport and scan-only diagnostics. Collapse the public API surface to one canonical versioned facade. Then measure success not by whether the converter outputs a pleasing glyph string, but by whether it preserves the intended concept graph, exposes its uncertainty, and remains reviewable after every transformation.

Open questions and limitations are narrow but important. I could not execute private or POST-only workflows against the site, so runtime conclusions are based on the public pages, the public status/openapi responses, and crawlable documentation rather than on private backend code or an authenticated test environment. Also, several SQL Server 2025 AI-related features are still documented as preview, so the safest production design is the one outlined above: treat preview features as optional accelerators, not as core dependencies.