TinyRustLM Distributed Memory Sync and Recovery Architecture

Full report

On this page

Current boundary and recommendation

The current MemoryEndpoints model already separates active local startup memory from hosted durable memory. The .uai suite is the active startup continuity layer, file handoff enters agent-file-handoff/Content or agent-file-handoff/Improvement, and hosted MemoryEndpoints is the intended mid-to-long-term boundary for workspace memory, current messages, receipts, and redacted audit trails. The public repository layout also shows the core surfaces that matter to this design: .uai/, agent-file-handoff/, docs/, memoryendpoints/, static/, and tests/. The agent-file-handoff/Improvement folder is currently empty except for .gitkeep, so this report is effectively a net-new architecture note rather than an edit of an existing sync design.

MemoryEndpoints already exposes the right hosted primitives to build a minimal distributed-memory system without inventing a second distributed database. The public contracts advertise protected routes for workspace load, agent registration, memory submission and retrieval, search, meeting rooms, current messages, acknowledgements, receipts, and audit-log readback. The live capability matrix and readiness result also show that the production backend is currently MySQL with storeBackendVerified=true, while the canonical schema already includes memory records, memory revisions, search documents, receipts, a review queue, idempotency records, outbox events, and an audit log.

The smallest architecture that correctly satisfies the requirements is a local-first, server-authoritative revision system with three persistence planes and one optional exchange plane. The three required planes are local files, browser-local durable queue/cache storage, and hosted MemoryEndpoints revisions. The optional plane is metadata-only P2P announcement exchange. In this model, every durable memory object has a stable memory_id, every mutation is an append-only event with a stable idempotency key, every authoritative hosted acceptance returns a receipt, and every conflicting hosted update produces an explicit conflict object instead of being silently overwritten. This uses append-only logs and optimistic concurrency where they add real value, but avoids the operational and product complexity of turning all agent memory into a full CRDT system. That recommendation is consistent with classic event-sourcing guidance, with vector-clock literature about causal branches, with CRDT literature on when unsynchronized multi-master editing is worthwhile, and with the current repository’s existing revision/idempotency/outbox tables.

A full-text or rich-object CRDT design is unnecessary here for three reasons. First, the hosted contract today is centered on public-safe summaries, review, receipts, and workspace authority rather than on peer-equal collaborative character editing. Second, deletion and “forget this” are materially easier in a revisioned server-authoritative model than in an immutable always-replicating CRDT lattice. Third, the optional P2P lane is explicitly constrained to shareable metadata only, not private bodies, so it does not need an automatic merge algebra for the underlying memory content. Last-write-wins is similarly too weak for durable memory because it discards concurrent intent; production systems that use LWW document that the most recent timestamp wins, which means concurrent writes can overwrite each other. In this design, LWW is allowed only for derived browser-session cache state such as “last expanded panel” or “last selected filter,” never for .uai, Markdown memory, hosted status, or hosted handoff content.

The resulting decision is straightforward: use append-only mutation records plus server-authoritative heads for durable memory, plus local optimistic queuing and explicit conflicts. Do not adopt full global event sourcing for every byte of state, because the system still needs erasure and compactness. Do not adopt CRDTs for durable memory bodies yet, because the product does not currently advertise multi-master body replication and because the existing schema already models revisions and authoritative receipts. Do keep a compact logical clock on the client, because offline tabs and devices still need deterministic local ordering and idempotent replay.

Data model and scopes

The canonical identity model should separate opaque identity from semantic grouping. The canonical memory identity is memory_id, an opaque, non-semantic, globally unique identifier generated at first creation and never reused. It should not be derived from mutable fields such as title, summary, or tags, because semantic hashing would create accidental merges across devices and would make explicit conflict handling harder. Semantic grouping belongs in a separate subject_key, derived from a normalized subject or reference slug and used only for search, duplicate suggestion, and “possible merge” hints. This fits the existing schema, which already distinguishes stable memory_id records from separate revision rows and tag rows.

The model needs four core identifiers. event_id identifies one mutation attempt and is generated client-side before any local or hosted write. revision_id identifies a proposed new state of a memory stream; it is stable across retries of the same queued mutation. idempotency_key identifies the transport-level replay contract for protected mutation routes; the current connector contract already states that protected mutation routes support Idempotency-Key except one-time setup, and the canonical schema already has a matm_idempotency table keyed by a stable record key with stored body hash and response JSON. receipt_id identifies the authoritative hosted acknowledgement that a mutation was accepted, persisted, and assigned a canonical outcome.

The parent/supersedes model should be explicit instead of implicit. Every non-create mutation carries parent_revision_id, which is the head revision the editor believes it is modifying. A clean update also carries supersedes_revision_ids, which is normally a one-element array containing that same parent. A merge resolution carries more than one superseded revision. A tombstone carries the current live head in supersedes_revision_ids. This is the minimal structure that supports optimistic concurrency, branch detection, explicit merge, and deletion propagation without introducing a general-purpose DAG engine. The existing schema already stores revision_number, body_hash, and change_summary in revision rows, which is enough to host this model with additive fields rather than a replacement schema.

The logical-clock requirement is intentionally modest. Each device keeps a durable per-workspace device_seq counter and a Lamport clock lamport_ts. Each queue operation increments both. Every mutation also includes device_id, device_seq, and lamport_ts. The server assigns its own monotonic server_seq inside each workspace on authoritative acceptance. A full version vector is not required in the steady state because conflict detection is primarily based on parent_revision_id versus the current authoritative head. A compact vector-like structure is still useful in two places: first, in the synchronization checkpoint, where each scope stores the highest server_seq seen; second, in the conflict object, where the system records the submitting device tuple and the current head’s device/server tuple so the user can see causality. This design borrows the useful part of logical clocks and causal metadata without paying the full cost of version vectors on every record.

The scope boundary should be stricter than the currently advertised hosted scope enum. The public connector contract currently lists hosted scopes as company, workspace, project, goal, and task. The user requirement adds user, device, agent, and public_p2p. The safest minimal implementation is therefore a two-layer scope model. TinyRustLM should maintain an internal authority_scope with values user, device, agent, project, goal, task, and public_p2p. When synchronizing to MemoryEndpoints, project, goal, and task map directly to hosted scopes. user, device, and agent remain local-only until the hosted contract is extended to advertise them explicitly. public_p2p never maps to MemoryEndpoints durable memory unless the user explicitly promotes a public-safe summary into a normal hosted scope. This avoids cross-scope leakage while respecting the current hosted contract as it exists today.

The tier classification should be strict enough that each memory lane has a single source of truth. Local short-term .uai memory is the source of truth for startup continuity and immediate working context. Local durable Markdown memory and references are the source of truth for durable local documents and citations. Browser-session hosted memory state is the source of truth only for session-local UI and auth state. MemoryEndpoints status records are hosted, public-safe, short-to-mid-term operational summaries. MemoryEndpoints handoff records are hosted durable summaries intended for later retrieval and review. Optional LocalEndpoints connectivity is only a local approval and receipt plane, not a hosted authority plane. Optional P2P is only a metadata-announcement plane. The current documentation already distinguishes active .uai, reviewed long-term local docs, hosted workspace memory, and current-message coordination as separate lanes; the design below keeps that same separation and makes the sync rules explicit.

The following schema definitions are the minimum contract I recommend:

Private versus shareable fields should also be formalized. Private-only fields include raw memory bodies before redaction, local file paths, workspace keys, .uai internals not explicitly shared, local device fingerprints, draft hidden prompts, proprietary references, and anything that can identify a private project boundary. Shareable fields include memory_id, revision_id, memory_type, authority_scope only when public-safe, public-safe title, tags, redacted summary hash, citation/reference hash, created/updated times, and signed announcement metadata. This is aligned with the current MemoryEndpoints firewall and connector contract, which explicitly require public-safe summaries only, prohibit raw credentials and private payloads, and expose receipts and audit data only in redacted form.

Synchronization and recovery flows

The browser reality matters as much as the distributed-systems reality. sessionStorage is tab-scoped and lasts only for the page session, while localStorage persists across browser sessions. IndexedDB is asynchronous and intended for larger offline datasets. The Broadcast Channel API allows same-origin cross-tab communication, and the Web Locks API allows same-origin tabs and workers to coordinate exclusive work such as one-tab-at-a-time sync. navigator.onLine is only a hint, not a trustworthy test of real reachability, and browser background sync may retry only a limited number of times. The right consequence is simple: keep credentials in memory or sessionStorage only, keep offline queue state in IndexedDB, coordinate tabs with BroadcastChannel and Web Locks, and treat sync scheduling as an opportunistic foreground process that can also use service-worker help when available but never depends on it.

The initial sync algorithm should be deterministic. First, the user authenticates and the workspace key enters session-only state; it is never copied into IndexedDB, localStorage, reports, or markdown memory. Second, the browser acquires a workspace-scoped Web Lock such as tinyrustlm-sync::<workspace_id> so only one tab performs authoritative sync. Third, the sync leader loads local checkpoints from IndexedDB and then calls hosted read routes in this order: workspace metadata, latest receipts after the last known receipt, memory events or memory heads after the last server_seq, current-message state, and review/conflict state. Fourth, it materializes those results into local IndexedDB views and updates local markdown/reference projections only after the hosted pull has committed. Fifth, it broadcasts a sync-applied message over BroadcastChannel so follower tabs can refresh cheaply without competing for the lock. If no credentials are present, the same flow still runs in local-only mode and stops after loading local checkpoints and queues.

The incremental-pull algorithm should be cursor based, not full-rescan based. Each authority scope keeps a SyncCheckpoint with the highest accepted server_seq and the last receipt applied. Pull requests use after_server_seq, limit, and scope filters, plus a replay-safe cursor token when the API adds one. Pages are applied in ascending server_seq order. Because the canonical schema separates matm_memory_records from matm_search_documents, search must be treated as an eventually consistent projection rather than the source of truth. Read-after-write and convergence checks therefore read head revisions and receipts first; search catches up second. This is the cleanest way to satisfy eventual consistency, stale-index handling, and large-history pagination simultaneously.

The offline-save algorithm must persist useful work without promising hosted durability. For .uai short-term memory, write locally first and return a local receipt immediately. For local durable Markdown memory, write the markdown/reference update locally and then append a QueuedOperation in the same logical transaction boundary using a preparing -> prepared -> committed state machine so crash recovery is deterministic. For browser session state, persist only non-secret state in IndexedDB and keep the workspace key in session-only memory. For hosted status and handoff mutations, create a queue entry containing the route, public-safe payload, body hash, event_id, revision_id, parent_revision_id, and idempotency_key; mark it awaiting_auth if no live credentials exist and ready_to_send otherwise. The UI must show this state as “saved locally, pending hosted synchronization,” never as synchronized. That distinction is required both by the user requirement and by the existing contract that protected writes are meaningful only when a workspace key is presented and a receipt is returned.

The reconnect-push algorithm reuses queue state without reusing credentials. Once the user authenticates again, the sync leader scans the queue in stable order: first by next_attempt_at_utc, then by device_seq, then by op_id. Before each send, it verifies that the queued payload is still public-safe and that the queue row is not marked blocked_by_revocation or superseded_locally. It submits the exact same idempotency_key and revision_id every retry. If the server returns success with persisted=true and a canonical memory or receipt identifier, the queue row becomes acked, the local checkpoint advances, and a redacted ReconciliationReceipt is stored. If the response is timeout or network failure after bytes were sent, the client does not mutate the key or body; it retries the same operation later. This is exactly the situation the current Idempotency-Key contract and matm_idempotency table are meant to handle.

Service degradation should degrade to local-only mode, not to false confidence. If MemoryEndpoints is unreachable, if the browser only has a doubtful navigator.onLine=true signal, or if the hosted environment stops proving durable backend verification, TinyRustLM should keep accepting local .uai and Markdown writes, keep queueing hosted mutations, and mark workspace sync state as local_only_pending. The current platform already distinguishes protected writes, redacted receipts, safe-no-op boundaries, and backend verification, so the correct UI contract is “local save succeeded, hosted sync pending or degraded,” never “synced” without a receipt.

Restoration after browser storage loss should also be explicit. If sessionStorage is cleared, the user simply re-authenticates; if IndexedDB is cleared, hosted state can be rebuilt by pulling head revisions, receipts, and current messages after login, while local .uai and Markdown memory remain available because local active memory is intentionally separate from hosted MATM. The only irrecoverable case is an unsent hosted mutation that existed only in browser storage and was never mirrored into local durable memory. The architecture should therefore require every durable user-visible action to land first in .uai or local Markdown and only then enqueue hosted sync, so browser-storage loss degrades hosted convergence but does not erase the user’s useful memory.

Conflict detection should happen at the hosted head, not in search. A clean update is accepted only if parent_revision_id equals the current authoritative head revision for that memory_id. If the same idempotency_key reappears with the same body hash, the server returns the previous outcome. If the same key reappears with a different body hash, the server returns a deterministic client error because the transport contract has been violated. If parent_revision_id is stale and the new body hash differs from the head or from the already accepted child, the server creates a ConflictObject instead of choosing a winner. If the current head is a tombstone, the stale update is blocked and converted into either a resurrection proposal or a new-memory proposal, depending on user intent. This is the minimum conflict contract that avoids silent loss while keeping the implementation close to standard optimistic-concurrency patterns.

Conflict resolution should be lane aware. For browser-session state only, explicit LWW is acceptable because the data is derived and disposable. For .uai short-term memory, auto-merge across devices is unsafe; the system should instead generate a derived durable conflict note summarizing the divergence and let the agent or user decide what becomes the new short-term seed. For local durable Markdown memory and hosted handoff memory, use a three-way merge if both branches share a common ancestor and the merge is non-overlapping; otherwise create an unresolved conflict object that requires an explicit merge revision with supersedes_revision_ids pointing to both branches. For hosted status memory, auto-supersession is allowed only when the system can prove same-memory, same-agent, same-parent, and non-overlapping field updates; otherwise use the same conflict path as handoff. Because the current contract already separates current messages from durable memory and recommends update-by-new-event rather than overwrite-in-place, an append-and-resolve strategy fits the existing system better than in-place mutation.

Deletion propagation needs both tombstones and retention. A delete request creates a local tombstone immediately so the item disappears from local views and local search. The queued tombstone is then pushed like any other mutation. Hosted acceptance creates an authoritative tombstone revision and removes the memory from default search and recall paths. Tombstones must carry retention_until_utc and erase_mode. erase_mode=soft_delete retains the redacted revision chain for audit and conflict prevention until retention expires. erase_mode=hard_forget_requested starts a purge workflow that removes the body, summary, references, and search projections while retaining only minimal audit data such as memory_id, receipt_id, timestamps, and body hashes necessary to prove deletion occurred. Because MemoryEndpoints currently advertises receipts, audit log, review queue, and public-safe summaries, but not a dedicated delete route, this is a repo-level extension that should be added explicitly rather than smuggled into status updates.

Recovery after an interrupted write should use a local intent log. Every queued mutation passes through preparing, prepared, sending, acknowledged, conflict, blocked, or corrupt states. On startup, the client scans for any operation left in preparing or sending. If the local durable target exists and the queue row is incomplete, it reconstructs the missing metadata from local body hash and marks the row prepared. If the queue row exists but the local body is missing, it marks the row corrupt and raises operator-visible repair. If the row was in sending, it performs a receipt probe or safe idempotent retry using the same key. This design is conventional write-ahead durability dressed for the browser: commit intent first, commit local durable state second, replay on restart, and never guess whether a timed-out hosted write succeeded when a receipt query or idempotent replay can tell you.

Conflicts, deletion, security, and P2P

The credential boundary should remain brutally simple. The workspace key is returned once and the server stores only a hash. The connector contract says browser connectors must keep the workspace key out of localStorage, should use secure local settings, and must not print the key or store raw credentials in memory or meeting summaries. sessionStorage is the right ceiling for browser persistence because it is partitioned by origin and tab and lasts only for the page session. IndexedDB is the right place for queued writes, checkpoints, receipts, redacted references, and public-safe cached head revisions, because it persists data across sessions without requiring the credential itself to persist alongside it.

That means queued writes survive credential loss by design. A queued write needs only workspace_id, memory_id, revision_id, the public-safe payload, hashes, causal metadata, retry state, and the intended route. It does not need the workspace key until send time. After a browser restart, the app loads the queue from IndexedDB, marks all unsent hosted operations awaiting_auth, and continues local-only operation. Once the user authenticates again, those exact queued operations resume with the original idempotency_keys. This satisfies the user requirement not to persist workspace credentials merely to make offline sync easier, while still preserving offline work across refresh or browser restart.

Device revocation and workspace-key rotation should be separate but interoperable. The existing canonical schema already has matm_api_keys with token_hash, last_used_at, and revoked_at, which is enough to support key rotation and revocation history. I recommend adding a first-class device_id association to each issued workspace key and storing a public device label plus revocation metadata. Rotation then becomes “issue new key, keep queue, require re-auth, resume with same queued ops.” Revocation becomes “mark device blocked, reject further hosted writes from that credential, keep the queue locally but mark it blocked_by_revocation until a human rebinds or discards it.” Because the queue is credential-free, rotation and revocation do not destroy offline work; they only gate future hosted acceptance.

LocalEndpoints should be treated as an optional local authority helper, not as a hidden browser backdoor. The LocalEndpoint public materials are explicit that the website is metadata-only, does not probe localhost, does not collect credentials, and keeps runtime authority on the device in the desktop companion. That aligns nicely with the TinyRustLM requirement. If LocalEndpoints is present, use it for desktop approvals, local receipts, and local-device review of queued hosted sync. If it is absent, TinyRustLM still works because local persistence and later re-authenticated sync are already self-sufficient. The architecture should therefore define a narrow ILocalAuthorityBridge style boundary: approveSync, exportLocalReceipt, showConflict, and confirmP2PShare. No browser page should ever assume hidden desktop control.

Optional P2P exchange should be announcement-only, explicitly shareable, and cryptographically attributable. WebRTC data channels exist specifically for arbitrary browser-to-browser or browser-to-device data exchange and are protected by DTLS, while the Web Crypto API provides the primitives needed for signatures, body hashes, and ephemeral key agreement. The safe design is therefore not peer replication of private memory bodies, but signed metadata announcements containing stable public identifiers, titles, tags, body hashes, expirations, and an optional fetch hint. The receiving peer treats every announcement as untrusted until schema validation, signature verification, expiry checks, and local user or policy approval succeed. Malicious peers can therefore waste attention, but they cannot exfiltrate private memory bodies because those bodies are never in the announcement.

A practical public-safe P2P announcement shape looks like this:

{
  "announcementId": "ann_01J...",
  "publicShareId": "share_01J...",
  "publisherEphemeralPublicKey": "base64url(...)",
  "memoryKind": "handoff-metadata",
  "title": "Build cache invalidation notes",
  "tags": ["rust", "build", "public-safe"],
  "bodyHash": "sha256:...",
  "referenceHint": "optional-public-endpoint-or-file-hint",
  "createdAtUtc": "2026-07-10T15:55:00Z",
  "expiresAtUtc": "2026-07-17T15:55:00Z",
  "signature": "base64url(...)"
}

No workspace key, no private summary, no raw body, no local file path, no hidden prompt, and no internal device identifier belongs in that object. This is stricter than the current connector contract and LocalEndpoint public boundary, but it is the right strictness for public P2P memory metadata.

Auditable proof of synchronization should be receipt based, not screenshot based and not “we think it worked” based. A sync is complete only when TinyRustLM holds a ReconciliationReceipt containing receipt_id, op_id, memory_id, accepted revision_id, server_seq, persisted=true, and a proof context that includes the deployed sourceSha and storeBackendVerified state from /api/version. The report or local evidence bundle should store only hashes and public ids, never the workspace key. This requirement fits the current system well: MemoryEndpoints already exposes redacted receipts, a redacted audit log, and a version endpoint that declares the current source SHA and backend verification state.

TDD scenarios, SLOs, and observability

The tests have to prove distributed behavior, not just route reachability. The public repository already treats local tests, live route verification, .uai audits, secret scanning, and deployment verification as separate evidence gates. This sync architecture should follow the same philosophy and add deterministic distributed-memory scenarios with fixed ids, fixed clocks, fixed retry schedules, and deterministic fake server responses. The browser pieces should be exercised with IndexedDB-backed tests, BroadcastChannel simulation, and lock-ownership simulation; the hosted pieces should be exercised against an in-process fake or test backend that reproduces idempotency, stale-parent conflicts, search lag, and revocation.

The production SLOs should distinguish local persistence from hosted durability. Local .uai save latency should target p95 under 25 ms and p99 under 75 ms on a supported desktop browser. Local durable Markdown save plus queue append should target p95 under 75 ms and p99 under 200 ms. Hosted save latency, measured from queued send start to authoritative receipt, should target p95 under 2.5 s and p99 under 5 s on a normal internet path. Local search over cached heads and references should target p95 under 100 ms; hosted search over current workspace memory should target p95 under 800 ms when the service is healthy. Incremental synchronization lag after connectivity and authentication return should target p95 under 10 s and p99 under 30 s for active workspaces under 10,000 retained records. Monthly availability should be defined separately as local-first save availability >= 99.99% and hosted authoritative sync availability >= 99.9%. Durability should be stated honestly: local durability is achieved only after local receipt, and hosted durability is achieved only after authoritative hosted receipt.

The observability contract should be redacted by default. Log workspace_id_hash, device_id_hash, agent_id, op_id, memory_id, revision_id, authority_scope, memory_type, queue_state, attempt_count, retry_delay_ms, http_status, server_seq, receipt_id, body_hash_prefix, conflict_kind, persisted, visible_in_search, store_backend, and source_sha. Do not log raw summaries, markdown bodies, file paths, workspace keys, local references, or peer payloads. Alerts should fire on oldest pending op age, queue depth growth, repeated idempotency mismatch, repeated stale-parent conflicts on the same memory, revocation rejections, search projection lag, receipt/audit mismatch, and storeBackendVerified=false when the environment claims hosted durability. Because the platform already exposes redacted receipts, audit-log readback, readiness, and backend verification, these alerts can be anchored to existing proof surfaces rather than invented out of band.

Implementation roadmap

The roadmap should be delivered as safe vertical slices that each improve correctness without forcing a flag day. The public repository documentation already emphasizes bounded claims, rerunnable verification, and post-deploy proof. This roadmap follows the same principle: each slice has a narrow code boundary, a reversible migration, a test bundle, explicit rollout controls, explicit rollback behavior, and evidence that must exist before promotion.

The compaction rule should stay conservative. Keep all live heads, all unresolved conflicts, all tombstones still inside retention, and all authoritative receipts. Superseded revisions can be compacted into a summarized historical bundle after the newest live head and its common ancestor chain are retained. Search documents can always be rebuilt from retained live heads and retained references, so search projections are compactable much earlier than authoritative revisions or receipts. The current schema already distinguishes memory records, revisions, search documents, idempotency, outbox events, and audit state, which makes this style of compaction structurally aligned with the repository instead of being a foreign pattern.

The practical promotion gate for each slice is the same. Before promotion, the slice must show local tests, deterministic distributed tests, no-secret logging proof, explicit rollback proof, and at least one evidence artifact that demonstrates the system refuses to call a write synchronized without an authoritative receipt. For hosted slices, the proof bundle should include the receipt shape, the version endpoint state showing source SHA and backend verification, and an audit-log entry showing a redacted protected operation. That maintains the project’s existing transparency discipline while giving TinyRustLM the local-first, multi-device, failure-recoverable memory system the user asked for.