Production Reliability Architecture for the InternationalIntelligence.org Daily Brief

Full report

On this page

Executive finding and research method

Executive finding

I assess the most plausible failure as production split-brain across execution, origin, cache, release, and browser layers, not failure of the internal Daily Brief workflow itself.

A workflow test can pass while the public site still looks broken when any of the following are true:

• The scheduler does not execute, executes late, overlaps, or invokes an obsolete release.
• PHP CLI advances state using a different configuration, user, filesystem, secrets set, or database target than PHP-FPM.
• An alternate hostname, virtual host, CDN origin, or document root serves a different release or site instance.
• A reverse proxy, CDN, browser cache, or service worker returns stale status JSON.
• A browser tab displays provisional HTML or local state after polling has been throttled or suspended.
• An old response arrives after a newer response and overwrites it.
• A process exists, but its last meaningful progress is stale.
• A deployment changes the web release while cron, OPcache, workers, or service-worker caches remain attached to the previous release.

The assignment reports that provider connectivity works and that process IDs, monotonic revisions, polling, cross-tab synchronization, trigger leasing, canonical-host handling, and versioned assets have already been attempted. Those controls are necessary but cannot prove that scheduled production execution and public delivery use the same release, site instance, authoritative datastore, filesystem, origin, and cache policy. I therefore treat the uploaded project description as the case study and as the source of reported—but not independently verified—system facts.

My central architectural recommendation is:

Use one server-side publication authority, backed by transactional idempotency and a lease with an expiry; let one primary scheduler and one independently monitored fallback submit trigger requests; expose immutable release and site-instance identities through every execution and status record; make status endpoints uncacheable at every layer; and treat every browser representation as provisional until reconciled with the authoritative API.

An external scheduler may request work, but it must never decide that a publication cycle exists, choose a new process ID, increment a revision, bypass eligibility rules, or declare success. Those decisions belong to the server authority.

Scope and limitations

I have no source-code, shell, hosting-panel, CDN, DNS, credentials, deployment logs, scheduler logs, database records, or private runtime-state access. Consequently, I cannot identify the actual fault with certainty. I can provide a production architecture, evidence requirements, safe diagnostics, failure hypotheses, and acceptance tests that distinguish likely causes without relying on privileged access.

Details not supplied remain unspecified, including:

Methodology

I modeled the system as a chain of independently falsifiable contracts:

1. A scheduler emits an authenticated trigger.
2. The canonical origin validates identity and mode.
3. The server authority determines eligibility.
4. A transaction reserves or resumes a publication process.
5. A worker advances state while renewing its lease and heartbeat.
6. The status API serializes authoritative state.
7. Reverse proxies and CDNs forward rather than reuse that response.
8. Browser code reconciles snapshots monotonically.
9. Cross-tab messaging accelerates—but never creates—convergence.
10. Deployment preserves release and state compatibility through rollout and rollback.

For normative behavior, I prioritized current standards and official implementation documentation as of August 1, 2026. HTTP cache semantics come from RFC 9111; request-target, host, and redirect semantics from RFC 9110; forwarded-header risks from RFC 7239; PHP behavior from the PHP manual; scheduler behavior from cron, systemd, cPanel, Google Cloud, and AWS documentation; and browser behavior from WHATWG, W3C, Chromium, Mozilla, and WebKit documentation. RFC 9111 expressly distinguishes shared and private caches and specifies that no-cache allows storage but requires successful validation before reuse, whereas no-store prohibits storage by conforming caches.

Evidence-source comparison

Recent research reinforces the need for active cache testing rather than trusting response labels alone. A 2024 study proposed timing-based discovery of hidden web caches and reported that some caches did not advertise their involvement through standard status headers; this supports using paired cache-busted and ordinary synthetic requests as diagnostic evidence, not as the normal freshness mechanism.

End-to-end execution model

Production path

The production contract should be understood as this chain:

A successful internal workflow test ordinarily validates only a subset of these rows. It does not prove that cron exists, that it invokes the same release as the website, that a public hostname reaches that release, or that the status response was fetched from the authority.

Applied case-study choice

I chose the InternationalIntelligence.org Daily Brief production path because the uploaded assignment supplies a concrete, plausible system with scheduler, PHP, CDN, browser, bilingual, and deployment concerns. This is preferable to inventing an unrelated example, while still requiring me to mark all unobserved implementation details as unspecified.

The likely causal families are:

Reliability timeline

timeline
    title Daily Brief production reliability lifecycle
    Scheduler configuration : Define UTC cadence
                            : Bind target and credentials
    Trigger receipt          : Validate trigger contract
                            : Record immutable trigger ID
    Reservation              : Evaluate next eligible cycle
                            : Atomically create or resume process
                            : Acquire expiring lease
    Advancement              : Poll provider
                            : Persist stage and revision
                            : Renew heartbeat and lease
    Publication              : Commit bilingual evidence
                            : Mark meaningful progress
    Status delivery          : Generate no-store snapshot
                            : Expose release and site-instance IDs
    Browser convergence      : Fetch authoritative API
                            : Reject stale responses
                            : Notify other tabs
    Deployment               : Switch immutable release
                            : reset FPM OPcache
                            : purge only intended CDN objects
                            : verify service-worker compatibility

Entity-relationship model

erDiagram
    SITE_INSTANCE ||--o{ RELEASE : serves
    SITE_INSTANCE ||--o{ PUBLICATION_CYCLE : owns
    RELEASE ||--o{ TRIGGER : receives
    TRIGGER }o--|| SITE_INSTANCE : targets
    PUBLICATION_CYCLE ||--o{ PROCESS : attempts
    PROCESS ||--o{ RESERVATION : protected_by
    PROCESS ||--o{ HEARTBEAT : emits
    PROCESS ||--o{ SNAPSHOT : produces
    PROCESS ||--o{ EVIDENCE_ITEM : accumulates
    RELEASE ||--o{ SNAPSHOT : serializes
    BROWSER_CLIENT }o--o{ SNAPSHOT : observes

    SITE_INSTANCE {
        string site_instance_id PK
        string canonical_origin
        string state_store_identity
    }
    RELEASE {
        string release_id PK
        string contract_version
        string asset_manifest_hash
    }
    PUBLICATION_CYCLE {
        string cycle_key PK
        date target_date
        timestamp next_eligible_at
    }
    TRIGGER {
        string trigger_id PK
        string trigger_type
        timestamp requested_at_utc
        string mode
    }
    PROCESS {
        string process_id PK
        int revision
        string stage_code
        timestamp last_meaningful_progress
    }
    RESERVATION {
        string lease_token PK
        timestamp acquired_at
        timestamp expires_at
        string owner
    }
    HEARTBEAT {
        timestamp observed_at
        string health_code
        int revision
    }
    SNAPSHOT {
        string snapshot_id PK
        timestamp generated_at
        int revision
        string release_id
    }
    EVIDENCE_ITEM {
        string evidence_id PK
        string type_code
        string locale_scope
    }
    BROWSER_CLIENT {
        string client_instance_id PK
        string origin
        string locale
    }

The database, not the operating-system PID, should own process identity. An OS PID is host-local, reusable, and potentially absent for HTTP or managed-function execution. It can be recorded as supporting evidence, but process_id should be an application-generated immutable identifier.

Scheduler architecture and authority contract

Scheduler-model comparison

Cron behavior is implementation-specific around daylight-saving transitions. Traditional Vixie-style documentation warns that nonexistent local times may never match and repeated times may match twice, while Cronie documents compensation for clock changes of less than three hours. That difference alone is sufficient reason to schedule the authority in UTC and compute local publication eligibility explicitly in application code.

Systemd timers can persist missed calendar activations and start a service after downtime when Persistent=true, while service units provide explicit users, working directories, environments, runtime limits, exit states, logs, and credential mechanisms.

Managed schedulers should be treated as at-least-once submitters. Google states that duplicate requests are possible in rare circumstances and instructs handlers to be idempotent; it also recommends UTC when an exact cadence must avoid DST anomalies. AWS EventBridge Scheduler documents at-least-once delivery, configurable retries, event retention, and dead-letter queues.

cPanel’s current documentation warns that jobs can overlap if scheduled too frequently, tells operators to use absolute paths, and notes that discarding stdout and stderr with /dev/null 2>&1 disables useful notification evidence.

Detailed operational comparison

Recommended scheduler topology

For a small-to-medium PHP publication site, I recommend:

• Primary: systemd timer where server control exists; otherwise an authenticated managed HTTP scheduler.
• Fallback: an independently hosted authenticated scheduler delayed by a grace interval.
• Authority: one transactional server endpoint or worker command shared by all trigger types.
• Execution: short, resumable advancement steps rather than one long monolithic request.
• Monitoring: independent freshness alerting based on last_meaningful_progress, not merely scheduler success.
• Timezone: scheduler cadence in UTC; publication date derived from a named IANA timezone inside the authority.
• Idempotency: unique cycle key such as (site_instance_id, target_date, edition_type) with a database uniqueness constraint.

The fallback should not run exactly with the primary. It should ask, for example, five or ten minutes later whether the eligible cycle has already been reserved or progressed. The authority either returns already_current, resumes a recoverable process, or reserves once.

Scheduled-advancement sequence

sequenceDiagram
    participant S as Primary Scheduler
    participant G as Canonical-Origin Gate
    participant A as Publication Authority
    participant DB as Authoritative Store
    participant W as Worker
    participant P as Provider

    S->>G: POST /internal/trigger<br/>signed contract + trigger_id
    G->>G: Validate trusted route, host, signature, time window
    G->>A: Submit normalized trigger
    A->>DB: INSERT trigger receipt
    A->>DB: Transaction: evaluate cycle and reserve
    DB-->>A: process_id + lease_token + revision
    A-->>S: 202 accepted / already-current
    A->>W: Dispatch resumable advance step
    W->>DB: Verify lease, release, site instance
    W->>P: Poll/fetch provider
    P-->>W: Provider result + request ID
    W->>DB: Commit state + revision + heartbeat
    W->>DB: Renew/release lease

Duplicate-trigger sequence

sequenceDiagram
    participant S1 as Primary Scheduler
    participant S2 as Fallback Scheduler
    participant A as Authority
    participant DB as Store

    par Nearly simultaneous triggers
        S1->>A: trigger T1 for cycle C
        S2->>A: trigger T2 for cycle C
    end
    A->>DB: Transaction reserve C
    DB-->>A: T1 wins; process P created
    A->>DB: Transaction reserve C
    DB-->>A: Unique cycle exists; return process P
    A-->>S1: 202 reserved P
    A-->>S2: 200 already_reserved P
    Note over A,DB: No second process or revision lineage is created

Scheduler contract

Every invocation should carry or derive the following normalized contract:

The trigger must not provide an authoritative process ID, target revision, stage, publication-success flag, or next-eligible time.

Exit-code taxonomy

For CLI and worker modes, use stable machine-readable exit codes:

HTTP trigger acceptance should use a separate response contract. A 202 means the trigger was authenticated and recorded, not that publication succeeded.

Heartbeat record

The heartbeat should be a persisted record, not merely a log line:

{
  "site_instance_id": "prod-us-central-01",
  "release_id": "2026-08-01.3+git.ab12cd3",
  "process_id": "proc_01J...",
  "cycle_key": "daily-brief:2026-08-01",
  "trigger_id": "trg_01J...",
  "trigger_type": "systemd",
  "worker_instance_id": "host-a:pid-18420",
  "lease_token_hash": "sha256:…",
  "lease_expires_at": "2026-08-01T12:06:00Z",
  "heartbeat_at": "2026-08-01T12:05:30Z",
  "last_successful_reservation_at": "2026-08-01T12:00:02Z",
  "last_meaningful_progress_at": "2026-08-01T12:04:54Z",
  "revision": 17,
  "stage_code": "PROVIDER_WAIT",
  "next_eligible_cycle_at": "2026-08-02T12:00:00Z",
  "redacted_failure_class": null
}

A lease must expire according to the authoritative datastore’s clock, not a browser clock and preferably not a worker host clock. Renewal must compare the current lease token and process ID. Once a worker loses the lease, it must stop mutating state even if it continues running.

PHP runtime drift and deployment split-brain

Why CLI and web execution diverge

PHP reads configuration according to the invoked SAPI and startup path. The PHP manual notes that server modules generally read configuration when the server starts, whereas CLI and CGI invocations read configuration for each execution; PHP also supports additional scanned INI directories. PHP-FPM pools can set distinct users, groups, working directories, chroots, environment variables, and PHP options, and FPM’s clear_env defaults to clearing worker environments unless explicitly populated.

The CLI binary is a distinct SAPI whose location depends on installation, and multiple PHP binaries can coexist. Therefore /usr/bin/php, /usr/local/bin/php, and a hosting-provider PHP selector may represent different versions or builds.

Drift diagnostic matrix

Safe diagnostics should expose hashes and metadata, not secret values.

PHP’s OPcache stores precompiled bytecode in shared memory. When timestamp validation is disabled, code changes do not take effect until OPcache is reset, scripts are invalidated, or the web server is restarted. The manual also documents that CLI OPcache is disabled by default, which means a CLI-side reset is not evidence that the FPM shared cache was reset. The latter conclusion is an architectural inference from the process-specific shared-memory model and should be verified against the actual hosting setup.

Immutable-release deployment model

Use this layout conceptually:

/var/www/international-intelligence/
├── releases/
│   ├── 2026-07-28.2+git.9183abc/
│   ├── 2026-08-01.3+git.ab12cd3/
│   └── ...
├── current -> releases/2026-08-01.3+git.ab12cd3/
├── shared/
│   ├── config/
│   ├── runtime/
│   ├── logs/
│   └── uploads/
└── bin/
    └── daily-brief-launcher

The stable launcher should resolve current once, verify that the release contains a signed or hashed manifest, load that release’s bootstrap, and record the resolved realpath and release ID. It should never write process state into the release directory.

Whether cron should target current or a pinned release depends on migration compatibility:

• For routine operation, target the stable launcher and let it resolve the active release.
• During deployment, pause advancement briefly or use a release-compatibility gate.
• A process already reserved under release A may continue only if release B declares its persisted-state contract compatible.
• Otherwise, release A finishes or checkpoints before the active pointer changes.
• The authority—not cron—decides whether a trigger from another release can resume the process.

Preventing two releases from serving one process

Each process should persist:

• created_by_release_id
• minimum_reader_contract
• maximum_reader_contract, where necessary
• writer_contract
• site_instance_id
• state_schema_version

A web release may display a process only if it understands its snapshot contract. A worker may mutate it only if its writer contract is permitted. This allows display compatibility without permitting two writer versions.

Release-rollout sequence

sequenceDiagram
    participant D as Deployment Controller
    participant R as New Immutable Release
    participant DB as Authority Store
    participant F as PHP-FPM
    participant C as CDN
    participant B as Browser Synthetic

    D->>R: Install dependencies and build hashed assets
    D->>R: Run offline runtime and schema compatibility checks
    D->>DB: Verify no incompatible active writer
    D->>R: Run doctor against production read-only dependencies
    D->>D: Atomically switch current symlink
    D->>F: Graceful reload / verified OPcache reset
    D->>C: Purge HTML, service-worker script, and mistaken API entries
    D->>B: Fetch canonical host with cache bypass
    B-->>D: New release + same site_instance_id
    D->>B: Run status, locale, alternate-host, and trigger synthetics
    B-->>D: Acceptance gate passes

Rollback sequence

sequenceDiagram
    participant O as Operator
    participant DB as Authority Store
    participant D as Deployment Controller
    participant F as PHP-FPM
    participant C as CDN
    participant S as Synthetic Monitor

    O->>DB: Check writer-contract compatibility
    DB-->>O: Previous release may read/write current state
    O->>D: Select previous immutable release
    D->>D: Atomically repoint current
    D->>F: Graceful reload / reset OPcache
    D->>C: Purge HTML and service-worker entry
    D->>S: Run rollback test suite
    S-->>D: Release old, site instance unchanged, revisions monotonic
    alt Prior release cannot write current schema
        O->>DB: Put advancement in safe read-only mode
        O->>D: Deploy forward-fix instead of destructive rollback
    end

Rollback must not reverse database revisions, reuse a prior process ID, or replace a new authoritative snapshot with an older snapshot. If the earlier release cannot safely understand current state, the correct rollback is read-only service plus a forward fix.

Canonical origin, cache control, and edge delivery

Canonical-origin gate

Host validation must occur before:

• session startup;
• cookie creation;
• CSRF-token creation;
• state-store selection;
• process initialization;
• lock or lease acquisition;
• publication eligibility checks;
• operator authentication side effects;
• publication logic.

HTTP routing depends critically on the target URI’s authority or Host value. RFC 9110 warns that incorrect handling can misdirect requests and create cache-poisoning risks; servers can reject deceptive or misdirected routing with an error rather than processing it. RFC 7239 likewise warns that forwarded information can be modified by clients or intermediaries and is trustworthy only within a defined proxy trust boundary.

The gate should follow this algorithm:

1. Obtain the connection peer address.
2. Trust Forwarded or X-Forwarded-* only when that peer is an explicitly configured proxy.
3. Parse exactly one effective scheme, host, and port according to the deployment’s proxy topology.
4. Reject multiple, malformed, empty, control-character-bearing, or syntactically invalid host values.
5. Lowercase the DNS hostname, remove a trailing dot according to policy, convert internationalized names through a consistent IDNA policy, and normalize default ports.
6. Compare against an allowlist, never against a suffix or substring.
7. Construct redirects from a static configured canonical origin plus the validated path and query.
8. Do not derive the redirect destination from an untrusted host or forwarded-host value.
9. For safe GET and HEAD, return a permanent canonical redirect.
10. For unsafe or state-changing methods, reject the alternate-host request rather than redirecting it.
11. Set no application cookie and initialize no state before completing this decision.

Redirect semantics

A 308 Permanent Redirect preserves the method and request content. That makes it useful for permanent canonicalization when preserving the method is intentional, but preserving an unsafe method toward another origin is precisely why the application should normally reject alternate-host POST, PUT, PATCH, and DELETE requests instead of redirecting them.

Recommended behavior:

Query strings may be preserved for safe redirects, but secrets should not be placed in query strings. The application must encode and append the existing validated request path/query; it must never accept a user-supplied full destination URL.

Alternate-host rejection sequence

sequenceDiagram
    participant U as Client
    participant E as Edge or Reverse Proxy
    participant G as Pre-Bootstrap Host Gate
    participant A as Application

    U->>E: POST https://alternate.example/operator/advance
    E->>G: Forward with normalized trusted metadata
    G->>G: Host is allowed alternate, but method is unsafe
    G-->>U: 421/400/403 + Cache-Control: no-store
    Note over G,A: No session, cookie, lock, process, or operator action

Two hostnames mapped to different roots

If www.example reaches release B while example or an old CDN origin reaches release A, application code in B cannot force A to execute B’s redirect logic. Every request to the stale origin runs whatever code and web-server configuration exist there.

Controls are therefore required outside the application:

Cache-layer map

Service-worker Cache Storage is independent of the HTTP cache. The service-worker specification states that Cache entries do not update or expire automatically and do not disappear just because the service-worker script changes; authors must version and delete them deliberately.

Exact endpoint policies

Live public status

Recommended origin response:

HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
Cache-Control: no-store, max-age=0
Pragma: no-cache
Expires: 0
Surrogate-Control: no-store
CDN-Cache-Control: no-store
X-Site-Instance-ID: prod-us-central-01
X-Release-ID: 2026-08-01.3+git.ab12cd3
X-Snapshot-Generated-At: 2026-08-01T12:05:30Z
X-Request-ID: req_01J...

Additionally:

• Configure an explicit edge rule to bypass cache for the status pathname.
• Do not rely on a random query parameter as the permanent control.
• Do not emit stale-while-revalidate or stale-if-error.
• Normally omit ETag and Last-Modified; a 304 optimization is not valuable for a small live status document where freshness proof is more important.
• If validators are retained for bandwidth reasons, use no-cache, private, max-age=0, must-revalidate rather than claiming that such a response is unstored. For the process-authority status, I still prefer no-store.
• Include a response request ID and snapshot-generation timestamp so a monitor can prove origin freshness.
• Suppress cookies from the public status endpoint.

RFC 9111 defines no-cache as a requirement to validate before reuse, not a prohibition on storage. no-store is therefore the clearer choice where a live status snapshot must not persist in shared or private HTTP caches. RFC 9111 also permits extensions and local cache configuration to affect behavior, so the CDN route must be tested and configured explicitly.

Cloudflare’s current default-cache documentation says origin cache directives are normally respected, but cache rules and edge-TTL behavior can override normal handling. Its cache-status documentation distinguishes bypassed, stale, revalidated, and cache-hit outcomes. Apache and Nginx likewise expose directives capable of changing or ignoring origin cache behavior.

Authorized operator reads

Cache-Control: no-store, private, max-age=0
Pragma: no-cache
Expires: 0
Vary: Authorization, Cookie

The endpoint must also enforce authentication, CSRF protection where cookies are used, canonical origin, and explicit method handling. private alone is insufficient because it still permits a private browser cache to store the response.

Operator and scheduler mutations

Use:

Cache-Control: no-store, private

Return an immutable trigger receipt rather than a potentially cached full status body. Require idempotency or trigger IDs.

Initial public HTML

A reasonable policy is:

Cache-Control: no-cache, max-age=0, must-revalidate

This allows storage but forces revalidation. The page must visibly indicate that embedded status is provisional until the first live API response.

Hashed JavaScript and CSS

For content-addressed filenames:

Cache-Control: public, max-age=31536000, immutable

A filename such as daily-brief.ab12cd3.js is safe to cache for a long duration because changed bytes produce another URL. This solves stale asset reuse only when every HTML document and service worker points to the new URL. It does not solve cached /api/status, because that URL and representation evolve in place.

Cache-bypassed retrieval sequence

sequenceDiagram
    participant B as Browser
    participant SW as Service Worker
    participant C as CDN
    participant R as Reverse Proxy
    participant A as Status API

    B->>SW: fetch /api/daily-brief/status<br/>cache: no-store
    SW->>C: Network-only; no Cache Storage match
    C->>C: Route-level cache bypass
    C->>R: Forward request with request ID
    R->>R: Proxy cache bypass
    R->>A: Authoritative GET
    A-->>R: no-store snapshot + generated_at
    R-->>C: no-store
    C-->>SW: BYPASS/DYNAMIC response
    SW-->>B: Unstored authoritative snapshot
    B->>B: Verify site instance, process, revision, request sequence

CDN and service-worker update strategy

For the status and operator routes:

• CDN cache rule: bypass.
• Worker/service-worker fetch handler: direct network fetch.
• Cache API: no match, put, runtime fallback, or offline substitution.
• Offline behavior: display the last accepted in-memory snapshot only with a prominent offline label and its age.
• Error behavior: do not return an old successful status body as though it were live.
• CDN “serve stale on error”: disabled for authority status.
• stale-while-revalidate: disabled because it intentionally serves old data while fetching new data. Cloudflare documents that behavior as asynchronous stale delivery followed by background revalidation.

For the service-worker script itself:

• Serve with short revalidation or no-cache, not a year-long immutable policy.
• Use a versioned cache name tied to the release contract.
• During activation, delete caches owned by incompatible releases.
• Do not call skipWaiting() blindly if the new worker cannot safely control pages running old JavaScript.
• Either coordinate activation through a “new version available” reload flow or guarantee cross-version message/API compatibility.
• Expose the controlling service-worker version in the browser diagnostic panel.
• Include a “network-only status route” automated test in every release.

Client convergence, status freshness, and browser behavior

Authoritative snapshot contract

Every status response should include:

{
  "schema_version": 3,
  "site_instance_id": "prod-us-central-01",
  "release_contract": {
    "serving_release_id": "2026-08-01.3+git.ab12cd3",
    "writer_release_id": "2026-08-01.3+git.ab12cd3",
    "status_contract_version": 3
  },
  "process_id": "proc_01J...",
  "target_date": "2026-08-01",
  "revision": 17,
  "stage_code": "PROVIDER_WAIT",
  "provider_state_code": "PENDING",
  "health_code": "HEALTHY_WAIT",
  "last_meaningful_progress_at": "2026-08-01T12:04:54Z",
  "last_provider_poll_at": "2026-08-01T12:05:10Z",
  "server_checked_at": "2026-08-01T12:05:30Z",
  "snapshot_generated_at": "2026-08-01T12:05:30Z",
  "next_eligible_cycle_at": "2026-08-02T12:00:00Z",
  "evidence_counts": {
    "expected": 12,
    "received": 8,
    "accepted": 7
  },
  "deficit_codes": ["PROVIDER_ITEMS_MISSING"],
  "next_action_code": "POLL_PROVIDER",
  "cache_observation": {
    "age_seconds": 0,
    "edge_status": "BYPASS"
  }
}

Localized prose should not appear in the authoritative state model. English and Spanish clients map the same codes through independent locale resources.

Reducer and ordering rules

I recommend a single reducer that applies these rules:

A “new process” should not be accepted merely because its process ID differs. The snapshot should carry a server-authoritative process succession relation or target-cycle ordering so a delayed response from an old origin cannot invent a replacement.

A practical ordering key is:

(site_instance_id, process_generation, process_id, revision)

process_generation should be monotonically allocated by the authority. Timestamps are supporting evidence, not the primary order, because clocks can differ and responses can be delayed.

Stale-response rejection sequence

sequenceDiagram
    participant B as Browser Reducer
    participant API as Status API

    B->>API: Request sequence 41
    B->>API: Request sequence 42
    API-->>B: seq 42, process P, revision 18
    B->>B: Accept P/18; latest_sequence=42
    API-->>B: delayed seq 41, process P, revision 17
    B->>B: Reject because sequence < 42 and revision < 18
    Note over B: DOM never regresses

Two-tab convergence

BroadcastChannel permits same-storage-key contexts to exchange structured messages, but eligible recipients must be active contexts. The HTML standard reports support across current Chromium, Firefox, and Safari generations. It remains a notification mechanism rather than persistent authority.

sequenceDiagram
    participant T1 as Active Tab
    participant API as Authority API
    participant BC as BroadcastChannel
    participant T2 as Background Tab

    T1->>API: GET authoritative status
    API-->>T1: P/revision 18
    T1->>T1: Validate and accept
    T1->>BC: {site, process, revision, stage_code}
    BC-->>T2: Notification
    T2->>T2: Validate identifiers
    alt Payload is newer than local state
        T2->>API: Read-only confirmation fetch
        API-->>T2: P/revision 18
        T2->>T2: Accept authoritative snapshot
    else Equal or older
        T2->>T2: Ignore/no-op
    end

The fallback is a localStorage notification key whose value contains only a random event ID plus process/revision identifiers. The storage standard dispatches events to equivalent storage contexts, but it also explicitly warns that local storage has no general locking mechanism and can fail or be disabled by policy. It must not generate unique process IDs or coordinate publication locks.

English and Spanish convergence

sequenceDiagram
    participant EN as English Client
    participant ES as Spanish Client
    participant API as Status API
    participant BC as Cross-Tab Channel

    EN->>API: GET /api/status
    ES->>API: GET /api/status
    API-->>EN: P/18, stage_code=PROVIDER_WAIT
    API-->>ES: P/18, stage_code=PROVIDER_WAIT
    EN->>EN: Render "Waiting for provider"
    ES->>ES: Render localized Spanish label
    EN->>BC: P/18 + codes only
    BC-->>ES: P/18 + codes only
    ES->>ES: Equal revision; no authoritative state change
    Note over EN,ES: Identical process and revision; locale prose is independent

Status-freshness model

A PID or process ID must never by itself produce a green “live” state.

Suggested server health states:

Stage-specific thresholds matter. Ten minutes without a revision may be normal while waiting for a provider but abnormal during a local render stage expected to last seconds.

Browser compatibility and throttling

Background execution cannot be treated as a scheduler. Chromium documents intensive timer throttling for qualifying long-hidden pages, with timers checked on a much coarser cadence; request-animation callbacks also stop in background pages. WebKit documents background timer throttling and notes that iOS can suspend background tabs completely. Firefox likewise manages or unloads background tabs under resource pressure.

WHATWG defines pageshow.persisted so a page can detect restoration from retained session history rather than a fresh load. On every pageshow, and especially when persisted is true, the client should immediately mark the DOM provisional and fetch authoritative status.

Fallback protocol

The client should:

1. Fetch immediately after page initialization.
2. Refetch on visibilitychange when a page becomes visible.
3. Refetch on pageshow, including back-forward-cache restoration.
4. Refetch after a successful network reconnection.
5. Keep a low-frequency, jittered, read-only poll while hidden when the browser permits.
6. Expect missed hidden polls and reconcile on resume.
7. Use BroadcastChannel as the preferred wake-up hint.
8. Use the storage event as a fallback wake-up hint.
9. Use neither mechanism to create process state or invoke advancement.
10. Abort superseded requests where possible and still enforce sequence/revision checks.
11. Display “offline” or “status unavailable” when no current authoritative response exists.
12. Never solve throttling merely by reducing the polling interval; browsers may coalesce or suspend it regardless.

Production testing, deployment, and rollback controls

Synthetic test matrix

Every synthetic response should record DNS answer, TLS certificate/SNI, resolved origin, HTTP status, redirect chain, release ID, site-instance ID, request ID, cache status, Age, snapshot time, process ID, and revision.

Scheduled-trigger acceptance checks

A scheduler test is not complete when the scheduler reports an HTTP 2xx. The monitor must correlate:

scheduler execution
→ trigger receipt
→ reservation result
→ heartbeat
→ meaningful revision or healthy no-op
→ status API visibility
→ browser-facing synthetic visibility

Alert separately on:

• no scheduler execution;
• execution without receipt;
• receipt without reservation decision;
• reservation without heartbeat;
• heartbeat without meaningful progress;
• authoritative progress not visible at the canonical status endpoint;
• canonical endpoint current but browser synthetic stale.

This decomposition identifies the failing layer rather than reporting only “Daily Brief broken.”

Deployment checklist

Rollback checklist

A rollback should:

• stop new incompatible writers;
• retain the authoritative database and monotonic revision history;
• atomically restore the previous compatible release;
• reload FPM;
• purge affected HTML and service-worker entry points;
• leave hashed assets immutable;
• verify scheduler launcher resolution;
• run canonical-host, status, locale, and duplicate-trigger tests;
• resume advancement only after writer-contract compatibility is established.

Do not delete or “reset” the current process merely to make an older release appear healthy. Preserve the process record and use an explicit supersession or recovery transition.

Remediation roadmap, uncertainties, acceptance criteria, and reading list

Prioritized remediation roadmap

Key uncertainties and evidence gaps

The most important unresolved questions are:

Acceptance criteria

Production should not be declared reliable until all of the following are true:

Scheduler and authority

• At least one production scheduler is demonstrably enabled.
• Every run supplies a unique trigger ID and UTC timestamp.
• Every trigger receipt records trigger type, authenticated identity, expected origin, release, and site instance.
• Duplicate triggers for the same cycle create exactly one process lineage.
• Locks are expiring leases with token-bound renewal.
• A stopped worker becomes recoverable after a bounded expiry.
• The external scheduler cannot set process ID, revision, stage, success, or next cycle.
• Scheduler 2xx acceptance is not confused with publication completion.
• UTC cadence and local publication-date calculation pass both DST-transition simulations.

Runtime and deployment

• CLI and FPM report approved PHP versions, INI contracts, extensions, paths, secrets references, site instance, and state-store identity.
• Scheduler commands use absolute paths.
• No production state depends on the current working directory or ambient temporary directory.
• Releases are immutable and identified in every trigger and response.
• The active release switch is atomic.
• FPM OPcache reset or reload is verified in the FPM runtime.
• Scheduler and web execution resolve a compatible release.
• Rollback preserves monotonic revisions and process identity.
• Two releases cannot concurrently mutate one process unless explicitly writer-compatible.

Origin security

• Canonical-host validation runs before cookies, sessions, locks, state, and authorization side effects.
• Safe alternate-host requests receive a deterministic canonical redirect.
• Unsafe alternate-host requests are rejected.
• Redirect destinations are constructed from static configuration.
• Forwarded hosts are trusted only from explicit proxy addresses.
• Malformed and unknown hosts never reach the normal application.
• All hostnames and origin paths resolve to one site instance or to an edge-level rejection/redirect.

Cache and CDN

• Public status and operator endpoints return Cache-Control: no-store.
• Reverse proxies and CDNs have explicit route-level bypass rules.
• Status responses do not use stale-while-revalidate or stale-if-error.
• CDN failure does not turn an old status into apparently live status.
• Browser fetches use cache: "no-store".
• The service worker uses network-only handling for status and mutation routes.
• Cache Storage contains no status or operator responses.
• Query-string cache busting is diagnostic only, not the primary policy.
• Hashed assets are immutable and HTML/service-worker entry points are revalidated.

Browser convergence

• Initial HTML is labeled provisional until an API response arrives.
• Same-process lower revisions are rejected.
• Equal revisions are no-ops except for browser-only observation times.
• A new process replaces the prior snapshot atomically.
• Different site-instance IDs force a visible warning and origin refresh.
• Older request sequences cannot overwrite newer responses.
• Background tabs reconcile on visibility and page restoration.
• Offline state is unmistakably labeled with offline age.
• BroadcastChannel and local storage cannot authorize or advance work.
• English and Spanish render the same codes and identifiers with independent localized strings.
• A process ID without recent meaningful progress does not produce a live indicator.

Observability

• The status UI separately displays meaningful progress, provider poll, server check, snapshot generation, browser check, cache age, and offline age.
• Every layer can be correlated through trigger, trace, request, process, release, and site-instance identifiers.
• Scheduler output and errors are retained outside transient cron mail.
• Synthetic monitoring covers every public hostname, both languages, clean and established browser profiles, cache bypass, background restoration, and duplicate triggers.
• Post-deploy checks block rollout completion when identity, freshness, or convergence contracts fail.

Prioritized official reading list

The architectural endpoint is not “poll more often.” It is a system in which every layer can prove that it is observing the same site instance, compatible release, process lineage, revision, and authoritative state—and in which a scheduler, cache, alternate host, old service worker, suspended tab, or delayed response can fail without manufacturing a false picture of publication health.