Achieving Semantic Isomorphism Across Mutable Languages and ISO-10646

Full report

On this page

Executive summary

This report uses semantic isomorphism as a working engineering term, not as a term defined by ISO/IEC 10646 or the Unicode Standard. In this report, it means a reversible, structure-preserving correspondence between a programming-language text model and the ISO-10646/Unicode text model such that the program-observable behavior of text remains stable under a declared policy: a fixed Unicode/UCD version, a normalization policy, an explicit notion of element boundaries, and explicit locale/collation rules. That framing is grounded in denotational semantics and semantics-preserving transformation literature, and in Unicode’s own requirements that text behavior depends not only on repertoire but also on normative properties and algorithms.

The most important conclusion is that semantic isomorphism is only achievable as a policy-relative property. It does not exist by default at the raw code-unit level, because Unicode semantics span multiple layers: bytes and code units, code points and scalar values, canonical and compatibility equivalence classes, grapheme clusters, bidirectional display, case folding, and locale-sensitive collation. Unicode and ISO/IEC 10646 are synchronized for repertoire, code points, and encoding forms, but Unicode adds the character property database, algorithms, and conformance rules needed for interoperable semantics.

For implementers, the practical rule is simple: never let a single “string” abstraction silently stand for all Unicode semantics. Equality, display, cursor movement, identifiers, sorting, storage, and interchange require different semantic models. Raw byte equality, code-point equality, canonical equivalence, compatibility equivalence, grapheme-cluster equality, and locale-sensitive collation are different relations and should be surfaced as different APIs or at least as different explicitly named modes.

Mutable language features make failures more likely. In-place string mutation can destroy normalization invariants at edit boundaries; UTF-16 code-unit indexing can split surrogate pairs or user-perceived characters; runtime code generation and hot loading can diverge from editor or reviewer display; dynamic runtimes that blur bytes and text can collapse domains that the standards keep distinct; and locale-sensitive operations can change behavior across machines or versions unless locale and collation data are pinned.

The strongest implementation pattern is a layered pipeline: validate encoding strictly, decode to Unicode text, preserve the original string where fidelity matters, compute normalized and case-folded derived keys where equivalence matters, segment UI operations by extended grapheme cluster, perform sorting with UCA or locale tailoring rather than code-point order, and keep Unicode/UCD and collation versions explicit in artifacts that persist across upgrades. Use ICU or an equivalent standards-conformant library; test with Unicode conformance data, property-based invariants, corpus-based edge cases, and fuzzing.

Definitions and scope

Because neither ISO/IEC 10646 nor Unicode defines the phrase “semantic isomorphism,” this report adopts the following working definition: a language-level text model is semantically isomorphic to ISO-10646/Unicode text when there is a reversible mapping between program values and UCS/Unicode text plus required metadata such that the denotation of text operations is preserved. The required metadata usually includes at least Unicode version, normalization policy, locale/collation policy, and source-code/display policy. This is consistent with denotational semantics’ requirement that meanings be compositionally assigned to program phrases, and with semantics-preserving transformation work that treats isomorphic representations as interchangeable only when their semantics are provably equivalent.

That definition is intentionally stricter than mere “Unicode support.” ISO/IEC 10646 specifies the UCS architecture, assigned planes, character names, control and private-use characters, and the UTF-8/UTF-16/UTF-32 encoding forms and schemes. Unicode stays synchronized with ISO/IEC 10646 for character codes and encoding forms, but adds the functional semantics that implementations actually need: character properties, normalization, bidirectional behavior, case mapping, segmentation, identifier guidance, collation, and conformance requirements. ISO/IEC 10646 itself explicitly says that suitability of characters for programming-language identifiers is outside the document and must be obtained from an external reference.

Assumptionally, this report treats “mutable languages” broadly: languages or platforms in which text values or the semantics of program text can change at runtime through mutable strings or buffers, dynamic typing or implicit coercions, runtime code generation or modification, locale-sensitive operations, or version-sensitive library/runtime behavior. No single programming language is assumed. Language-specific examples below are illustrative because different mainstream platforms choose different native text models: Python str is specified as a sequence of Unicode code points, while Java String and Apple NSString are UTF-16-based and index by 16-bit units; ECMAScript strings are sequences of 16-bit unsigned integers and operations that interpret them as UTF-16 must account for ill-formed subsequences.

A useful way to scope semantic isomorphism is to ask which level of text meaning must be preserved:

The key analytical point is that an implementation can be isomorphic at one layer and non-isomorphic at another. Many systems are repertoire-isomorphic but not grapheme-isomorphic; many are canonical-equivalence-aware for storage but not for identifiers; many are correct for logical text processing but not for display or source-code review. That is why a single “Unicode string” type is rarely enough as a semantic contract.

ISO-10646 and Unicode text semantics

At the base layer, ISO/IEC 10646 defines the UCS and the three standard encoding forms UTF-8, UTF-16, and UTF-32. A graphic character is assigned only one code point in the standard, either in the BMP or in a supplementary plane. Unicode uses the same repertoire and encoding forms, while clarifying that conformant implementations operate over Unicode scalar values rather than arbitrary code units: surrogate code points are excluded from scalar values, isolated UTF-16 surrogates are ill-formed, surrogate values are ill-formed in UTF-8 and UTF-32, and processors must not interpret ill-formed subsequences as characters.

Unicode semantics extend far beyond assigning code points. The Unicode Character Database is an integral part of the standard and supplies the normative and informative property data used by algorithms such as normalization, bidirectional ordering, and case folding. Unicode explicitly states that it associates a rich set of semantics with characters and sometimes with code points; these semantics are required for conformance and are expressed through properties such as General_Category, bidirectional class, decomposition mappings, canonical combining class, case mappings, and text-boundary properties. Those properties are versioned and stable once published, though some property values can be revised in later versions within the stability policies.

A central distinction is among code units, code points, scalar values, and user-perceived characters. UTF-8, UTF-16, and UTF-32 are interoperable encoding forms, but none of them makes code units equal to user-perceived characters in general. UTF-16 uses surrogate pairs for supplementary characters; UTF-8 uses between one and four bytes; and even UTF-32’s one-code-unit-per-scalar-value model still does not make a code point equivalent to a user-perceived character because combining sequences and emoji sequences may span multiple code points. Unicode Text Segmentation therefore defines grapheme clusters as the machine-approximable unit for user-perceived characters, and recommends extended grapheme clusters for general processing.

Normalization is the core mechanism for equivalence management. Unicode defines canonical equivalence for sequences representing the same abstract character and expected to have the same appearance and behavior when correctly displayed, and compatibility equivalence for cases where text distinctions such as font, ligature, numeral style, or layout compatibility are folded away for particular uses. The four standard normalization forms are NFD, NFC, NFKD, and NFKC; NFC and NFD manage canonical equivalence, while NFKC and NFKD additionally apply compatibility decomposition. Unicode emphasizes that higher-level processes that transform or compare strings should either respect canonical equivalence directly or normalize under a declared policy.

Combining characters and the canonical combining class are where naive representations often stop being semantically faithful. ISO/IEC 10646 defines a base character as a graphic character that is not a combining character. Unicode assigns each combining character a canonical combining class, and the normalization process uses canonical ordering to give combining marks a unique order. Most characters are “starters” with combining class zero; combining sequences reorder only where the normalization algorithm requires it. This matters because string equality and offsets can diverge even when the rendered result is the same.

The rules for user-perceived boundaries deliberately operate above the code-point layer. UAX #29 defines user-perceived characters, words, and sentences, and states that grapheme cluster boundaries are important for regular expressions, collation, UI interactions, counting “characters,” and editing operations such as arrow movement and backspacing. Crucially, the default grapheme cluster rules are defined so that boundaries remain unchanged across canonically equivalent forms, meaning the same grapheme boundaries hold for NFC and NFD representations of the same text.

Emoji make this even more explicit. Unicode Emoji defines text-presentation and emoji-presentation selectors U+FE0E and U+FE0F, emoji modifier sequences, regional-indicator flag sequences, and ZWJ sequences. It also states that emoji sequences are single grapheme clusters and that there is never a grapheme cluster boundary within an emoji sequence. A valid emoji flag sequence is exactly two Regional Indicator characters; a singleton RI is not a well-formed flag sequence. These are not exotic exceptions; they are mainstream Unicode semantics that any “Unicode-aware” system must surface correctly if it wants semantic isomorphism for modern text.

Equality and ordering are different semantic contracts. Unicode’s collation standard notes that culturally correct string comparison should be done through the Unicode Collation Algorithm, not through raw code-point order or byte order. UCA produces sort keys and supports tailoring by locale or application. Unicode also warns that collation order is not fixed over time and must therefore be versioned. By contrast, UTF-8 byte order happens to preserve code-point order lexicographically, but RFC 3629 explicitly notes that code-point order is “almost never culturally valid.”

Finally, several reserved or special categories delimit where shared semantics stop. Noncharacters are permanently reserved for internal use and are not recommended for open interchange, though they survive transformations among Unicode encoding forms. Private-use characters supply code points without standardized semantics; interoperability is possible only by out-of-band agreement. That means any claim of semantic isomorphism over private-use text is necessarily local to a closed system or profile, not global to Unicode itself.

Mutable language features and where isomorphism breaks

The most common failure mode is treating code-unit mutability as if it preserved text semantics. Unicode normalization forms are not closed under concatenation; even if two strings are normalized, concatenating them can yield a non-normalized result, and the same general problem occurs for many in-place edits near boundaries. Unicode therefore recommends local repair around the join within stable code points and provides quick-check machinery for fast detection. In other words, “normalized string” is not a durable invariant unless the type system or API surface enforces it after every mutation.

A second failure mode is indexing by the wrong unit. In Java, index positions in String refer to UTF-16 code units, so a supplementary character occupies two positions; Apple NSString also reports lengths and indexes in terms of 16-bit UTF-16 code units; ECMAScript strings are sequences of 16-bit units and may contain ill-formed subsequences. That means mutation, slicing, or cursoring by index can split surrogate pairs, variation-selector sequences, or ZWJ-linked emoji if the API consumer thinks the index denotes characters in the everyday sense. Python avoids one class of this problem by specifying str as a sequence of Unicode code points, but even that still does not make code points equal to grapheme clusters.

A third failure mode appears in dynamic typing and implicit coercion. ISO/Unicode and the IETF treat octet streams, Unicode strings, and encoding forms as distinct domains. RFC 5198 defines a network-text profile over UTF-8 and NFC; RFC 3629 defines UTF-8 byte syntax and requires invalid sequences to be rejected; ISO/Unicode distinguish encoding forms, code units, and characters. From those standards, an important engineering inference follows: a runtime that implicitly blurs bytes and text makes semantic isomorphism harder because it collapses layers the standards keep separate. In dynamically typed systems, explicit types or tagged wrappers for “raw bytes,” “validated UTF-8,” and “Unicode text” are therefore not ceremony; they are semantic boundary objects.

Runtime code generation and modification introduce a fourth failure mode: compiler semantics can diverge from human-visible semantics. Unicode Source Code Handling explains that source code poses special risks because reviewers must be able to perceive lexical extent, token order, and identifier equivalence from the rendered text. It recommends that language design, editors, review tools, and diagnostics all participate in maintaining that alignment. The Trojan Source paper demonstrates that Unicode bidi controls can cause text to appear in a different order to humans than the compiler interprets, across many mainstream languages. Once a runtime allows eval, macros, hot reload, template compilation, or generated code, semantic isomorphism requires the same normalization, identifier, display, and diagnostics policy everywhere the code can be created or consumed.

Locale-sensitive operations create the fifth major breakage mode. Unicode collation is explicitly tailored and versioned. ECMA-402 states that its internationalization functionality is drawn from well-established APIs such as ICU, .NET, and Java, and recommends that string comparison follow UTS #10. ICU itself implements CLDR/UCA/ISO 14651-style collation. The implication is that string ordering, and sometimes case and matching behavior, are not properties of “the string alone”; they are properties of the string under locale and version parameters. A language that hides locale behind process-global settings or platform defaults makes semantic isomorphism non-portable by design.

Identifiers show why declarative policy matters. UAX #31 states that for identifiers, NFC is generally appropriate for case-sensitive languages, while NFKC is more appropriate for case-insensitive ones, and it defines requirements for equivalent normalized identifiers and equivalent case-insensitive identifiers. It also distinguishes default identifiers from immutable identifiers, warning that immutable identifiers which allow unassigned characters cannot reliably support NFC, NFKC, or case-insensitive equality and are unsuitable for some security restrictions. That means source-level semantic isomorphism requires a language to declare its identifier policy, not leave it implicit.

Mapping strategies and architectural patterns

The most robust architecture is to model text as a pipeline of semantic projections, not as a single scalar abstraction. A practical implementation preserves the original payload where fidelity is required, but derives explicit alternate forms for equivalence, searching, sorting, and UI behavior. Unicode normalization, grapheme segmentation, collation, and identifier processing are all logical specifications that allow optimized implementations so long as they produce the same results; the architecture can therefore be both rigorous and fast.

flowchart LR
    A[Bytes or legacy encoding] --> B[Validate and decode]
    B --> C[Unicode text model]
    C --> D{Policy layer}
    D --> E[Original text for display/fidelity]
    D --> F[NFC or NFD key for canonical equivalence]
    D --> G[NFKC or NFKC_Casefold key for identifiers/search]
    D --> H[Extended grapheme clusters for UI]
    D --> I[UCA or locale-tailored sort keys]
    D --> J[Security diagnostics: bidi, confusables, ignorable chars]
    J --> K[Storage, network, database, source tools]

The first design decision is what notion of equality you need. For opaque storage or exact protocol fields, byte equality may be correct. For canonical text equality, use NFC or NFD. For identifier equivalence or compatibility-insensitive matching, use NFKC or toNFKC_Casefold under a declared identifier profile. For ordering, use collation, not equality plus code-point order. For UI, use grapheme clusters for navigation and edit operations. The mistake is not choosing “the wrong” policy; the mistake is silently using one policy for all of these jobs.

A concise comparison is useful:

Normalization should normally happen at boundaries, not indiscriminately in the middle of every operation. UAX #15 explicitly notes that normalization forms are not closed under concatenation, but that an optimized concatenation function only needs to normalize around the join boundary within stable code points. This leads to a useful pattern: enforce or derive normalization at system boundaries, then maintain a validated/normalized invariant inside the subsystem, repairing only on edits that cross affected boundaries. ICU’s Normalizer2 and quick-check facilities are built around this style of implementation.

Character indexing and user indexing should be separated. Code-unit offsets are still needed for low-level encoders, protocol parsers, and API interop; code-point iteration is useful for many transforms; extended grapheme clusters are the correct default for user-visible cursoring and deletion. UAX #29 also notes that grapheme-cluster boundaries can be tailored for particular languages or uses. A semantically isomorphic system therefore does not try to “pick one index”; it exposes multiple index domains and converts between them explicitly.

For legacy encodings, the correct pattern is a reversible Unicode bridge, not ad hoc partial transcoding. UAX #15 defines normalization for legacy encodings in terms of an invertible transcoding to Unicode, normalization in Unicode, and transcoding back. It also warns that some legacy encodings are prenormalized, some normalizable, and some effectively unnormalizable for a chosen normalization form. That means semantic isomorphism with legacy encodings is only possible where the transcoding is reversible and the encoding’s structure can actually support the target notion of equivalence.

For network interchange, the standards strongly favor UTF-8. RFC 3629 defines UTF-8 over the Unicode/ISO-10646 range, prohibits surrogate code points, requires invalid-sequence handling, and highlights helpful properties such as character-boundary recovery and self-synchronization. RFC 5198’s Net-Unicode profile chooses UTF-8, recommends NFC before transmission, prohibits the BOM at the head of the text stream, and forbids unassigned code points for conforming senders. In practice, that makes “UTF-8 + validation + declared normalization policy” the default wire-level choice for semantic isomorphism.

Pitfalls, tooling, trade-offs, and testing

Several Unicode features are common sources of hidden semantic drift. Surrogates are not scalar values and must never be treated as characters in UTF-8 or UTF-32, or as independent characters in UTF-16. Regional indicators only form well-formed flags in pairs. Variation selectors request text or emoji presentation and can change rendering without being letters. ZWJ sequences form atomic emoji clusters for editing. Bidirectional controls change display order but are otherwise supposed to be ignored by comparison and parsing logic, which is exactly why they can confuse reviewers if editors do not reveal them. Private-use characters have no global semantics. Noncharacters should not be used in open interchange.

Long combining-mark runs are another underestimated edge case. Unicode normalization permits extremely long non-starter sequences, but UAX #15 defines the Stream-Safe Text Format to bound buffering requirements by inserting CGJ in exceptional cases. Unicode cautions that stream-safe processing preserves the normalization forms but is not canonically equivalent to the original if it actually inserts CGJ. That is a classic semantic trade-off: bounded buffering and protocol safety versus exact canonical preservation. It is appropriate only in protocols or subsystems that explicitly accept that trade.

Collation introduces a different trade-off. UCA and ICU collation produce stable and culturally appropriate orderings, but they are heavier than binary compare and are not fixed forever. Unicode says collation order will vary over time and must be versioned; ICU emphasizes that the implementation is designed to be fast and memory-efficient, but it still requires locale data and sort-key generation for reproducible behavior. Any persistent index or ordered database that relies on collation should therefore store or pin the collation version and locale profile used to create it.

A brief compatibility view of common tooling is helpful:

Security and interoperability should be treated as first-class semantics, not add-ons. Unicode security guidance exists because mixed scripts, confusables, bidi controls, default-ignorable characters, and display-vs-logic mismatches can all create exploitable ambiguity. UTS #39 provides confusable-detection and mixed-script mechanisms; UTS #55 adds source-code-specific mitigations and display guidance; Trojan Source shows that the risks are not theoretical. If the application domain is source code, identifiers, authentication, or security-sensitive UI, approximate Unicode handling is not enough.

Testing should mirror the semantic layers. Use the Unicode conformance data for normalization and segmentation, and use property-based invariants such as normalization idempotence, round-trip transcoding where applicable, and preservation of declared equivalence classes. Unicode’s own NormalizationTest.txt and GraphemeBreakTest.txt exist for this purpose, and ICU documents how it develops and reproduces fuzz targets for Unicode APIs. Corpus-based tests should include composed/decomposed pairs, stacked combining marks, lone surrogates and invalid UTF sequences, emoji ZWJ families, RI flags, bidi controls, private-use characters, and legacy-encoding round trips.

Recommended practices and checklist

The most useful recommendations vary by use case:

A prioritized implementation checklist follows:

1. Declare the semantic contract for each text path: encoding, Unicode/UCD version, normalization policy, locale/collation policy, and whether operations are defined over code units, code points, or grapheme clusters. Without this, semantic isomorphism is undefined, not merely incomplete.
2. Validate encoding strictly at ingress and never interpret ill-formed subsequences as characters. Reject or quarantine malformed UTF, lone surrogates, and invalid UTF-8.
3. Separate raw/original text from derived semantic keys. Preserve fidelity in one representation; derive NFC, NFKC_Casefold, grapheme maps, or sort keys only for the purposes that require them.
4. Normalize at subsystem boundaries and repair near edit boundaries after mutation, rather than assuming concatenation preserves normalization.
5. Use extended grapheme clusters for user-facing editing and counting, and reserve code-unit indices for low-level APIs only.
6. Use collation for ordering, never code-point or UTF-8 byte order, and persist the collation version/profile alongside sortable artifacts.
7. Apply UAX #31 plus UTS #39/UTS #55 for source code and identifiers. Warn on confusables, bidi controls, mixed-script identifiers, and normalization-policy migrations.
8. Avoid private-use characters in open interchange unless you also define and version the private agreement that gives them meaning.
9. Prefer ICU or an equivalently conformant library for normalization, collation, segmentation, and bidi instead of assembling partial behavior from unrelated APIs.
10. Test with Unicode conformance files, property-based invariants, real corpora, and fuzzers, including emoji, combining marks, bidi, invalid input, and legacy transcoding cases.

The main limitation in this report is intentional: because no target programming language was specified, the recommendations are expressed as language-agnostic architecture and policy requirements, with specific language/runtime notes used only as examples. If a particular target language were fixed, some of the engineering advice would become sharper, especially around native string mutability, error handling for ill-formed UTF, and identifier syntax.

Sources

Primary standards and standards-adjacent sources used in this report include ISO/IEC 10646:2020 and its ISO/Unicode relationship notes; the Unicode core specification; UAX #15 on normalization; UAX #29 on segmentation; UAX #31 on identifiers; UAX #44 on the Unicode Character Database; UAX #9 on bidirectional text; UTS #10 on collation; UTS #39 on Unicode security; UTS #51 on emoji; UTS #55 on Unicode source code handling; RFC 3629 on UTF-8; and RFC 5198 on Net-Unicode.

Implementation and platform references used for compatibility notes include ICU documentation, Python documentation, Oracle Java documentation, Microsoft .NET documentation, TC39 ECMAScript and ECMA-402 references, and Apple Foundation documentation.

Academic and research references used for the analytical framing and security discussion include denotational-semantics course notes, semantics-preserving type-changing transformation research, the Trojan Source paper, and research on Unicode visual spoofing.