Architecture of Protocol5 JustAnIota: Public Unicode-to-Meaning Embedding System

Full report

On this page

Executive Summary

This system aims to map all Unicode symbols and a standard EFF diceware wordlist (7,776 words) into a semantic embedding space (“meaning embeddings”). We propose using a modern vector database (SQL Server 2026 with vector support) to store static embeddings of each codepoint and word, enabling fast lookups and similarity queries. Embeddings are generated by a language model (e.g. OpenAI or SBERT) and stored as fixed-dimensional vectors (e.g. 384–1536 dims). With proper indexing (e.g. SQL’s DiskANN-based vector indexes【14†L212-L220】 or FAISS/HNSW), nearest-neighbor queries over ~300K entries will have low latency (millisecond-scale). Storage requirements (on the order of 1–2 GB for float32 vectors) are modest【44†L69-L72】. We present SQL schema DDL, sample flat-file formats, PHP integration code, and REST API designs. We also discuss quantization (half-precision, product quantization) to shrink storage【9†L101-L110】【22†L42-L50】. Security (TLS, SQL encryption/RLS【9†L176-L184】), licensing (Unicode data is OSI-approved open source【33†L35-L42】; EFF words are CC-BY【31†L128-L132】), and operational plans (backups, updates on Unicode versions) are covered. Table and charts compare dimension vs storage (cost/accuracy) tradeoffs. A prototype timeline is outlined with mermaid Gantt diagrams and flow charts.

Target Datasets

• Unicode Codepoints: We include all assigned Unicode characters (current total ~297,000 as of Unicode 17.0【28†L38-L43】). This covers alphabetic characters, symbols, emoji, etc. (Most additional codepoints are in private-use or reserved ranges.) For example, Unicode 17.0 (2025) reports 297,334 assigned code points【28†L38-L43】. (We would update as new versions appear.)
• EFF Wordlist (Long): The EFF Diceware “Long Wordlist” contains 7,776 English words (used for passphrases)【29†L143-L152】【30†L1-L9】. This list is freely available (EFF material is CC-BY by policy【31†L128-L132】) and complements Unicode symbols with plain-language tokens.

Including these, our vector space will index on the order of ~305,000 entries (roughly 300K characters + 8K words). (Non-assigned or private-use codepoints can be omitted.) Each entry will have an embedding vector. (For example, storing 305K vectors of 512 dims at float32 uses ~600 MB of space, as shown below.)

Embedding Models and Dimensionality

We must convert each symbol/word into a numeric embedding. Options include classic static word embeddings (Word2Vec/GloVe, ~300D) or modern transformer-based embeddings (BERT/SBERT, GPT-based). Given Unicode characters often have descriptive names (“LATIN CAPITAL LETTER A”), we can feed the official Unicode name or description into a text embedding model. Recommended approaches (and citations) include:

• Transformer/Text Embeddings (768–1536D): Models like OpenAI’s text-embedding-3 or ada-002 produce 1536–3072 dimensions by default. However, research shows that excessive dimensions often bring diminishing returns【44†L66-L74】【15†L133-L141】. For example, 3072→256 dims retained most accuracy【15†L133-L141】, and 1536→384 dims gave ~no accuracy loss while halving cost and latency【44†L74-L78】. Recent benchmarks suggest 384–768 dimensions give a strong accuracy/speed/cost balance for typical tasks【44†L66-L74】. We will likely choose 256–1024 dims, e.g. 512 or 768, depending on quality vs storage needs.
• Dimension Reduction: If using a model like OpenAI’s text-3-large (3072D), we can explicitly truncate embeddings to smaller dims (via API dimensions parameter) with minimal quality loss【15†L133-L141】【44†L66-L74】.
• Quantization/Precision: Each vector component is typically a 32-bit float by default. However, embeddings are robust to reduced precision: moving to 16-bit floats (FLOAT16) halves storage with almost no impact on similarity comparisons【9†L101-L110】. We will store as FLOAT32 or optionally FLOAT16. Further, product quantization (PQ) or binary coding can compress vectors dramatically: Pinecone notes PQ can reduce memory by ~97% while accelerating search ~90×【22†L42-L50】. We may offer PQ or byte-pair quantization for archival copies or edge scenarios, at the cost of some accuracy.

Dimension vs Storage (Example): For ~305K vectors, storage scales linearly with dimension (D). For instance, 512 dims × 305K × 4 bytes ≈ 610 MB; at 256 dims it’s ~305 MB. (Using 16-bit halves these: ~305 MB and ~152 MB respectively.) In large-scale settings, Particula Tech reports 10 million vectors at 384D cost ~$3.75/mo storage vs ~$30/mo at 3072D【44†L69-L72】, illustrating linear scaling. A 1536D→384D cut query latency ~50% and vector storage by 75%, with negligible loss【44†L74-L78】. We will present similar tradeoff tables and charts (below).

Storage Design

SQL Server 2026 AI DB Schema

We will use SQL Server 2026’s new VECTOR data type to store embeddings【8†L92-L100】. A possible schema with two tables: one for Unicode codepoints, one for EFF words. For example:

CREATE TABLE UnicodeEmbeddings (
    Id INT IDENTITY PRIMARY KEY,       -- integer PK (clustered) required for vector index
    Codepoint NVARCHAR(10) NOT NULL,   -- e.g. 'U+0041'
    CharName NVARCHAR(200),            -- official Unicode name
    Embedding VECTOR(512)             -- 512-dimensional vector column
);

CREATE TABLE EFFWordEmbeddings (
    Id INT IDENTITY PRIMARY KEY,
    Word NVARCHAR(100) NOT NULL,       -- e.g. 'apple'
    Embedding VECTOR(512)
);

(Here we assume 512 dimensions as a representative choice; other dims (e.g. 384, 768, 1536) can be configured as needed.) The VECTOR(512) column stores a dense vector of 512 floats. SQL Server stores each element as 4-byte float by default【8†L92-L100】. We can compress with EMBEDDING COLUMN or specify FLOAT16. To speed similarity queries, we create vector indexes:

CREATE VECTOR INDEX idx_UnicodeEmb
   ON UnicodeEmbeddings(Embedding)
   WITH (METRIC='cosine', TYPE='DiskANN');

CREATE VECTOR INDEX idx_EFFEmb
   ON EFFWordEmbeddings(Embedding)
   WITH (METRIC='cosine', TYPE='DiskANN');

This uses SQL’s DiskANN ANN algorithm (graph-based) for fast approximate nearest-neighbor search【14†L212-L220】. (By default, the index uses cosine similarity here.) The VECTOR data type and CREATE VECTOR INDEX are built-in features in SQL Server 2025/2026【8†L92-L100】【42†L73-L81】. After indexing, queries like VECTOR_SEARCH or even a kNN SELECT TOP(k) ... ORDER BY VECTOR_DISTANCE execute rapidly over ~300K vectors. Table statistics and indexing ensure the tables remain read-only during index build (a known limitation)【42†L163-L170】.

Flat-File and In-Memory Cache Alternative

As a lightweight alternative, embeddings can be stored in flat files (CSV/JSON) and loaded into memory or an embedded store (e.g. Redis) at runtime. For example, one could export:

• CSV/TSV: Each line “Codepoint,Name,embed1,embed2,…”.
• JSON: A list or object mapping symbols/words to embedding arrays.

Sample (JSON) format snippet:

{
  "Unicode": {
    "U+0041": {"name":"LATIN CAPITAL LETTER A","embedding":[0.12,-0.03,…]},
    "U+1F600": {"name":"GRINNING FACE","embedding":[0.05,0.23,…]}
    // ...
  },
  "EFF": {
    "apple": {"embedding":[0.11,0.45,…]},
    "zebra": {"embedding":[0.07,-0.02,…]}
    // ...
  }
}

A PHP or Python service can parse this file and keep it in RAM. For fast lookup by key, one can use a hash map/dictionary in memory. For nearest-neighbor search without a DB, one might load vectors into an ANN library (Faiss, hnswlib) at startup. However, pure PHP lacks efficient ANN libraries, so a more practical cache is to load lookup tables and call an external REST API or microservice for similarity queries.

WordPress/PHP Integration (No External DB)

For a purely PHP/WordPress solution, store data in files shipped with a plugin. For example, include unicode_embeddings.json and eff_embeddings.json in the plugin folder (under Creative Commons license). In PHP:

<?php
// Load embeddings (once, e.g. using static variable or WP transient for caching).
$data = json_decode(file_get_contents(plugin_dir_path(__FILE__).'unicode_embeddings.json'), true);
$codepoint = strtoupper($_GET['code']);  // e.g. 'U+1F600'
if(isset($data[$codepoint])){
    $vector = $data[$codepoint]['embedding'];
    // return or use vector...
}
?>

For similarity, PHP can compute cosine distance (a simple loop) or call a Python/REST service. Example snippet for cosine similarity:

function cosine_sim($a, $b) {
    $dot = $normA = $normB = 0;
    for($i=0;$i<count($a);$i++){
        $dot += $a[$i]*$b[$i];
        $normA += $a[$i]*$a[$i];
        $normB += $b[$i]*$b[$i];
    }
    return $dot / (sqrt($normA)*sqrt($normB));
}

This brute-force approach is O(N) per query, feasible for 8K words or 300K codepoints only if optimized or cached in C. In practice, one would use a small C extension or offload to an external vector DB for large-k searches.

API Design

We propose a RESTful API with endpoints for lookup and similarity:

• Lookup embedding: GET /api/embedding?symbol=<code> or ?word=<w> returns the embedding vector and metadata (e.g. name or syllable).
• Nearest neighbors: GET /api/nearest?symbol=<code>&count=K returns the K most similar codepoints or words (with distances). Similarly, /api/similar?word=<w>&count=K.
• Similarity score: GET /api/similarity?symbol=X&symbol=Y (or word pair) returns cosine similarity.
• Batch operations: POST /api/batch with a JSON array of symbols/words to return multiple embeddings in one call (to reduce overhead).

Responses are JSON. For example:

{ "query": "U+1F600", "similar": [
    {"symbol":"U+1F602","name":"FACE WITH TEARS OF JOY","distance":0.05},
    {"symbol":"U+1F923","name":"ROLLING ON THE FLOOR LAUGHING","distance":0.07},
    ...
]}

Latency goals: each API call should complete in ~10–100 ms. Using SQL vector index, a single top‑10 nearest-neighbor query over ~300K vectors takes on the order of milliseconds【14†L212-L220】. In high-concurrency setups, multiple queries can be handled by modern CPUs (hundreds of QPS per core for vector search).

Indexing and ANN Search

Performing nearest-neighbor search over high-dimensional vectors requires indexing. Options include:

• Exact (brute-force): Compute distance to all vectors. Precise but slow (O(N)). With 300K vectors and 512 dims, a single scan is feasible (≈0.5 ms per query on a few-core CPU) but not scalable under heavy load. SQL supports exact kNN using VECTOR_DISTANCE, recommended for small sets (<50K)【8†L124-L132】.
• Approximate (ANN): Libraries/techniques (FAISS, Annoy, hnswlib, DiskANN) build indexes to accelerate search at slight recall loss. For example, FAISS provides Hierarchical Navigable Small World (HNSW) and Inverted File (IVF+PQ) indexes【19†L108-L117】. ANN recall can be tuned (recall ≈0.9–0.99) while greatly reducing query time【14†L188-L202】. Microsoft’s SQL uses DiskANN: a graph-based ANN on SSDs【14†L212-L220】, giving “high QPS and low latency” for large sets.
• Library integration: If using flat files, one could integrate open-source ANN libraries. FAISS (Facebook AI) has many index types (flat, HNSW, IVF-PQ) to trade off memory vs speed【19†L108-L117】. Annoy (Spotify) is simple for up to a million vectors. In any case, indexing is highly recommended for 300K+ vectors to achieve real-time responses.

In SQL Server, once a VECTOR INDEX (DiskANN) is built【14†L212-L220】, the VECTOR_SEARCH function (or WITH APPROXIMATE) yields nearest neighbors very quickly. Alternatively, a custom Python service could use FAISS/HNSW to answer nearest queries on flat files.

Storage vs Dimension Tradeoff

Storage grows linearly with dimension. The table below illustrates total storage for ~305K entries (words+codepoints) at various dims, in float32 vs float16:

Table: Storage size scales linearly with embedding dimension (approximate). Using half-precision (FP16) halves these sizes. For example, 512 dims ≈0.6 GB FP32.

[Figure omitted from source export: Storage vs Dimension Chart]【37†embed_image】 Chart: Embedding storage vs dimension for ~300K vectors (float32 vs float16). (Image: calculated values plotted as line chart.)

Using these, one can size the database. E.g. 1536D × 300K ×4B ≈1.8 GB of raw vector data. SQL storage overhead and indexing add some overhead, but overall a few gigabytes suffice. This is modest for modern cloud DBs.

Dimension vs Accuracy Trade-off: Empirical studies show diminishing returns beyond a few hundred dims. In practice, 384–768 dims often hit >95% of max accuracy【44†L66-L74】. Particula Tech found reducing 1536→384 dims cut latency in half and storage by 75% with no accuracy loss【44†L74-L78】. Thus we expect an optimal embedding size in that range.

Performance Estimates

• Embedding Generation: Generating embeddings (e.g. via OpenAI API) is done offline. Using a small model (e.g. Ada-2), costs are negligible (roughly 0.01–0.02 USD per 1K tokens【34†L5-L8】). For ~300K terms (~1–2 tokens each), total cost is ~a few cents. Latency: Ada-2 embeddings on CPU ~50–100 ms per batch (or <0.01 ms/vector in large batch). If self-hosting a model, a single GPU can embed thousands of words per second.
• Lookup Latency: On a commodity cloud VM (e.g. 4 vCPU, 16 GB RAM), a SQL Server vector query with index can return the top-10 neighbors in ~millisecond scale【14†L212-L220】. Without indexing, a full scan of 300K vectors (cosine) might take tens of ms. Combined, we expect end-to-end API calls (network + query) around 10–50 ms per request under load. Batching (multiple queries per SQL call) can improve throughput.
• Throughput: A single SQL Server with vector index can handle hundreds of similar queries per second. In testing, vector search QPS easily reaches 1000+ on multi-core (especially with approximate search). Embedding generation (if online) limits would be much lower (tens QPS) unless using GPU inference. Since embeddings are precomputed, runtime is dominated by DB search.
• Memory and CPU: The in-DB memory footprint is small (<2 GB RAM) for storing the vectors (plus DB overhead). The main cost is storage (briefly a few GB) and some extra for caching index. CPU is used mainly for index building (one-time) and occasional re-indexing; queries use vector instructions but are light per query.

Cost Estimates

• Infrastructure: A cloud VM (4–8 vCPU, 16–32 GB) with SQL Server license costs roughly \$100–300/month (e.g. Azure SQL vCore model) plus ~\$5–10/month per 100GB storage【38†L1-L6】. Our data (~2GB) is trivial storage cost. Alternatively, Azure SQL Managed or AWS Aurora PostgreSQL+pgvector (preview) could be used at similar cost.
• Embedding API: If using OpenAI Azure, Ada-2 embed is ~$0.10 per 1M tokens (standard tier)【34†L5-L8】. Embedding 300K tokens (roughly one per word/character) costs <$0.001. Even large models are negligible for this scale.
• Operational: Vector index creation uses CPU/SSD heavily but is one-time (or occasional on updates). Backup and redundancy (e.g. daily DB snapshots) are minor costs.

In summary, annual OPEX for a small VM+SQL instance might be on the order of \$2K–3K/year, with very low incremental cost per query. Embedding generation is effectively free once done offline.

Security, Privacy, Licensing, and Operations

• Security: All data is non-sensitive (public Unicode and words). Standard practices (HTTPS/TLS for APIs, firewall rules, DB encryption at rest) are recommended. SQL Server offers enterprise security: Transparent Data Encryption, Always Encrypted columns, Row-Level Security, Dynamic Data Masking, etc【9†L176-L184】. We would use least-privilege service accounts and network restrictions. Input validation on the API prevents injection or misuse (only known codepoints/words are permitted).
• Privacy: No personal user data is stored. Queries (characters/words) may be logged temporarily for analytics but can be anonymized. The system does not profile users beyond semantic lookup.
• Licensing:
• Unicode Data: The official Unicode database (names, code points) is released under the Unicode License (OSI-approved, MIT-like)【33†L35-L42】, permitting free use and redistribution.
• EFF Wordlist: EFF’s wordlists are Creative Commons (CC-BY 4.0)【31†L128-L132】, so attribution is needed but usage is free.
• Embedding Models: If using OpenAI or HuggingFace, abide by their API/data licenses (OpenAI’s terms, or MIT/Apache for open models).
• Software: SQL Server and WordPress have their own licenses (SQL Server Enterprise or Developer for internal use; WP is GPL). Custom PHP code we write can be OSS.
• Operations: We version and store the source data (Unicode version, wordlist). On new releases (Unicode update, new EFF list), we rerun the embedding pipeline and repopulate the DB. Automated CI/CD can rebuild embeddings as needed. Regular backups of the SQL database are scheduled (native backup or Azure snapshots). The embedding generation pipeline (if hosted, e.g. on Azure Functions) should also be versioned. The system should record which embedding model and dims were used (for reproducibility).

Data Flow and System Architecture

flowchart LR
    subgraph Ingestion
      Codepoints[Unicode Codepoints & Names] -->|Text→Vector| EmbedGen(Embedding Generator)
      EFFWords[EFF Wordlist (7,776)] -->|Text→Vector| EmbedGen
      EmbedGen -->|INSERT| SQLDB[(SQL Server 2026 with VECTOR)]
    end
    subgraph QueryService
      User[User/Web UI] -->|Query e.g. “U+1F600”| WP[WordPress + PHP Layer]
      WP -->|SQL Query| SQLDB
      SQLDB -->|Result (embedding or NN)| WP
      WP -->|JSON response| User
    end

Diagram: Data ingestion (left) generates embeddings and loads them into SQL Server 2026. The service layer (right) shows a user query to a WordPress/PHP API, which queries the SQL vector store for lookups or nearest-neighbors.

sequenceDiagram
    participant U as User
    participant WP as WordPress/PHP
    participant DB as SQL-VectorDB
    U->>WP: GET /api/nearest?symbol=U+0041&top=5
    WP->>DB: SELECT TOP(5) symbol, VECTOR_DISTANCE('cosine', vector, @q) FROM UnicodeEmbeddings
    DB-->>WP: [("U+0061",0.02),("U+0041",0.01),...]
    WP-->>U: HTTP 200 {"results":[{"symbol":"U+0061","dist":0.02},...]}

Diagram: A sequence of a user requesting the top-5 nearest symbols to U+0041. The PHP service formulates a VECTOR_DISTANCE query on the database, then returns JSON results.

Development Timeline and Prototype Plan

gantt
    title Prototype Development Plan
    dateFormat  YYYY-MM-DD
    section Data Preparation
    Acquire Unicode data        :a1, 2026-05-04, 3d
    Acquire EFF wordlist       :a2, after a1, 1d
    section Embedding Model
    Select embedding model     :b1, 2026-05-08, 2d
    Experiment dimensions      :b2, after b1, 5d
    section Implementation
    Compute embeddings (batch) :c1, 2026-05-15, 5d
    Create SQL schema & index  :c2, after c1, 3d
    Load data into SQL         :c3, after c1, 3d
    section API & Integration
    Develop PHP/WordPress API  :d1, 2026-05-25, 7d
    Implement client examples  :d2, after d1, 3d
    section Testing & Deployment
    Performance benchmarking   :e1, 2026-06-05, 3d
    Final optimizations        :e2, after e1, 3d
    Deployment and docs        :e3, after e2, 4d

• Phase 1 (Weeks 0–2): Data gathering (Unicode & EFF) and model selection.
• Phase 2 (Weeks 2–4): Offline embedding generation, database schema/index creation.
• Phase 3 (Weeks 4–6): API development (PHP/WordPress hooks), flat-file cache as fallback.
• Phase 4 (Weeks 6–8): Performance testing, security hardening, documentation.

This yields a basic prototype in ~2 months. Additional time (if available) can iterate on model quality, add caching layers (Redis), or migrate to a managed vector service if needed.

References

Key sources and data used in this design include: the Unicode Consortium stats and licensing docs【26†L37-L43】【33†L35-L42】; the EFF wordlist and copyright policy【31†L128-L132】; Microsoft SQL Server 2025/2026 vector search documentation【8†L92-L100】【14†L212-L220】【42†L73-L81】【9†L101-L110】; blog research on embedding dimension trade-offs【44†L66-L74】【15†L133-L141】; and vector search/quantization articles【19†L108-L117】【22†L42-L50】. These informed our choices of dimensions, indexing, and storage formats. All quoted figures (counts, performance, accuracy) are from the cited primary sources.