Black-Box Incident Reconstruction and Reliability Plan for the InternationalIntelligence.org Daily Brief

Full report

On this page

Executive finding, evidence boundaries, and likely failure families

OBSERVED — On the public snapshot inspected on August 1, 2026, the Daily Brief was not current: the server date was August 1, the latest published edition was July 30, five dates were reported missing, and the current process targeted August 1. The public status representation identified one active process, dbp_20260801_c9fb48f6d62f8513, at revision 34, in workflow step 2/8, with provider status queued, four automatic attempts out of four, and no August 1 edition URL.

OBSERVED — The same public status said that a provider operation was durably recorded: background_response.present, stored, and response_id_recorded were all true. It also said that the same response should be polled rather than duplicated. This materially reduces the likelihood that the immediate incident is simply an invalid provider credential, an unsupported model, or inability to create background work.

OBSERVED — The exposed state contained several liveness and projection anomalies:

INFERRED — The highest-probability immediate failure family is a reconciliation-liveness failure after provider work has been reserved. Plausible variants are: the scheduler or server-owned supervisor is not running; it runs in a different runtime or filesystem view; it runs but refuses to acquire or renew a lease; it retrieves the provider state without committing the resulting transition; or the public status projection is stale and conceals real progress.

INFERRED — The second-highest-probability failure family is an evidence-contract mismatch. The system has demonstrated that it can receive substantial provider evidence, yet its binder and validators retained none of the candidate set. That pattern is more consistent with mismatched identifiers, source-normalization rules, freshness boundaries, corroboration rules, schema/version drift, or evidence loss between passes than with a complete absence of provider output.

INFERRED — A third important family is multiple process views: cron and web PHP may load different configurations, releases, document roots, state paths, users, timezones, or mounted filesystems. PHP-FPM pools can run with different users, environments, chroots, and configuration files, while per-directory .user.ini settings apply to CGI/FastCGI but not CLI. Therefore, a web validator can pass while a CLI scheduler sees different credentials, paths, extensions, or state.

INFERRED — Pure browser or CDN staleness is a credible visibility amplifier but is not the best single explanation for the sampled backend state. The English Daily Brief page and the JSON status projection agreed on process, revision, stage, attempts, latest publication, and deficits. A stale service worker, browser cache, reverse proxy, or CDN could still explain why some users see older data, because service workers can intercept requests and substitute cached responses independently of ordinary HTTP caching.

INFERRED — Publication-write and link-generation failures remain necessary branches in the complete fault tree, but they rank below scheduler, reconciliation, and evidence-validation faults for this particular snapshot because the process had not reached composition or finalization.

REPORTED BY PROJECT OWNER — DNS resolution, TLS verification, PHP cURL, OpenSSL, the CA bundle, provider credentials, configured models, the Responses API, required web search, strict structured output, and background-response creation and retrieval have reportedly succeeded in prior diagnostics.

REPORTED BY PROJECT OWNER — Earlier releases reportedly introduced a canonical host, a Daily Brief process identifier, monotonic revisions, cross-tab synchronization, browser polling, a browser trigger lease, and a server-owned supervisor.

UNKNOWN — No source code, deployment manifest, scheduler configuration, server log, private response identifier, state database, state-file path, filesystem mount, PHP configuration, or operator record was inspected for this report.

UNKNOWN — The web observations establish what a public representation exposed; they do not establish whether that representation was freshly generated at origin, reconstructed from a state artifact, or delivered through an intermediary cache. Actual raw response headers from independent networks and browsers must be retained during the incident run.

RECOMMENDED — Treat the incident as a Daily Brief control-plane outage, not initially as a provider outage. The first goal is to determine whether the authoritative state is advancing at all; the second is to locate the first durable transition that fails; the third is to prove that successful publication becomes visible consistently across languages, hosts, sessions, and caches.

Evidence classification and current fact ledger

Why passing deterministic tests can coexist with a broken production feature

INFERRED — Deterministic validators usually establish that a function behaves correctly for known inputs under a controlled runtime. They do not prove that the production scheduler invokes that function, that it loads the same release and configuration, that it can read and write the same state, or that a later worker commits the result.

INFERRED — A provider create/retrieve test proves an API capability, but not the application’s durable lifecycle. The production workflow additionally must record the operation identifier before losing control, retrieve the same identifier after restarts, recognize terminal states, preserve output evidence, advance the correct process revision, and prevent a stale writer from overwriting newer state. OpenAI’s background-response model requires clients to retrieve and poll the same response while it is queued or in_progress, then handle terminal statuses such as completed, failed, cancelled, or incomplete.

INFERRED — A test run through web PHP does not prove CLI behavior. PHP CLI and CGI/FastCGI can differ in loaded configuration, working directory, environment, per-directory settings, process user, chroot, extension availability, and filesystem access. PHP-FPM can also define separate pools with distinct users and configuration.

INFERRED — A test that validates state-transition logic in isolation does not prove mutual exclusion. In production, an operator request, public-page trigger, browser poll, and cron invocation may overlap. A lock that merely expires by time is insufficient unless every writer also carries a monotonically increasing fencing token or expected revision; otherwise a paused old holder can resume after lease expiry and overwrite work committed by a new holder. Lease-based locks release ownership when the lease expires, but safe state mutation still requires the application to reject stale holders.

INFERRED — A successful write test does not prove atomic publication. The real flow must write all bilingual artifacts, metadata, index entries, and supervisor state in an order that cannot expose a partial edition. POSIX rename provides the standard primitive for atomically replacing a pathname, but the implementation must verify filesystem constraints and must not update “published” state until the final artifact can be read back and hashed.

INFERRED — A passing UI test does not prove freshness. A browser page may render an old embedded snapshot while JavaScript polls a fresher endpoint; a service worker may replace a network response with its own cache; a CDN may cache one language or hostname differently; or client code may infer progress from elapsed time rather than display server-owned state. Service workers act as request proxies and can serve cached responses independently of normal browser navigation caches.

RECOMMENDED — Define “production success” as an end-to-end state transition observed from outside: one target date is selected, one process is created, one provider operation is reconciled, valid evidence is retained, one bilingual immutable edition is atomically committed, the supervisor becomes terminal, and all public views converge to the same publication within a bounded time.

Symptom taxonomy and snapshot-discrimination model

RECOMMENDED — Use the following taxonomy to assign each observation to the earliest failing lifecycle boundary. A symptom should not be labeled “stuck” without specifying which timestamp, revision, lease, retry deadline, or artifact failed to move.

RECOMMENDED — Determine whether two pages represent the same snapshot by comparing semantic identity, not prose. The minimum tuple should be:

canonical_origin
site_release
process_id
process_revision
snapshot_token
target_date
stage_key
provider_operation_fingerprint
provider_status
last_meaningful_progress_at
next_check_at
publication_date
artifact_manifest_hash
status_generated_at

RECOMMENDED — Apply these decision rules:

INFERRED — A changed revision with an unchanged process ID normally means progression within one process. An unchanged revision with changing client-side elapsed time does not demonstrate server progress.

RECOMMENDED — No UI should be allowed to synthesize a later stage than the status endpoint. Client code may display a local age counter, but it must visibly identify that counter as local and must not alter the server-owned stage, provider status, revision, last-progress time, or next-action label.

Black-box observation, chronology reconstruction, and evidence record

RECOMMENDED — The observation program must remain read-only. Because the public page says that an eligible GET may initiate publication, automated probes must not repeatedly hit that page without owner approval. HTTP defines GET as a safe method whose requested semantics are read-only; coupling a public page view to production mutation makes monitoring ambiguous and permits bots, caches, and users to act as accidental schedulers. The mutating trigger should ultimately be separated into a controlled scheduler or authenticated idempotent operation.

Safe observation matrix

RECOMMENDED — For ordinary HTTP cache testing, retain Date, Age, Cache-Control, Expires, ETag, Last-Modified, Vary, Via, standardized Cache-Status if present, vendor cache headers, request IDs, and redirect headers. RFC 9111 defines cache reuse and validation behavior, while RFC 9211 defines Cache-Status specifically to make intermediary cache handling diagnosable.

RECOMMENDED — Do not equate Cache-Control: no-cache with “not stored.” It means a stored response must be successfully revalidated before reuse. no-store prohibits storage and reuse by conforming caches, while must-revalidate prohibits stale reuse after freshness expires.

RECOMMENDED — Treat ETags as representation validators. If the status body changes semantically but the strong ETag does not, or if identical bodies receive unstable ETags on every request, cache validation becomes unreliable. Conditional requests should be tested with the server-provided validator rather than fabricated values.

Evidence-log schema

RECOMMENDED — Store every observation as an append-only record plus immutable raw artifacts. The following fields satisfy the minimum incident record:

Chronology template

RECOMMENDED — Normalize all events to UTC but retain the original timestamp and source timezone. Do not infer ordering from file modification times alone; use a stable tie-breaker such as process revision, event sequence, provider poll sequence, or append-log index.

UTC chronology reconstruction method

RECOMMENDED — Reconstruct one chronology in six passes:

1. RECOMMENDED — Preserve every raw timestamp exactly as supplied, including offset and timezone name. Add a normalized UTC column rather than replacing the original.
2. RECOMMENDED — Group records first by target date, then process ID, then provider-operation fingerprint. A target date without a process ID is a reservation event, not proof of a process.
3. RECOMMENDED — Order committed server transitions by monotonic process revision. Wall-clock timestamps can be wrong; a lower revision must never be treated as later authoritative state merely because its clock is ahead.
4. RECOMMENDED — Place browser and cache observations around the server timeline using response Date, Age, request IDs, status generated_at, and local request times. Estimated origin-generation time for a cached response is approximately response receipt time minus Age, but only as an inference because intermediaries can transform or omit metadata. RFC 9111 defines Age as the mechanism by which caches communicate estimated response age.
5. RECOMMENDED — Mark impossible relationships rather than silently correcting them: negative job age, a next-check before creation, a revision regression, “published” before artifact creation, or a poll event without an incremented poll sequence.
6. RECOMMENDED — Identify the last indisputably durable event and the first expected event that did not occur. That boundary is the incident’s most useful localization.

RECOMMENDED — Use these precedence rules when timestamps conflict:

durably committed revision
    > append-only event sequence
    > provider terminal timestamp tied to stored operation fingerprint
    > artifact manifest creation and read-back
    > origin Date/generated_at
    > cache-adjusted client observation
    > browser-rendered relative time

INFERRED — For the sampled August 1 state, the last clearly exposed durable application event was the process snapshot at revision 34 with a recorded provider response. The first expected but unproven event was a successful bounded reconciliation after next_check_at, followed by a revision increase or explicit terminal/failure transition.

Request-sequence reference model

RECOMMENDED — The black-box observations should be interpreted against this safe target sequence. The observer performs only read operations; the scheduler owns all mutations.

sequenceDiagram
    participant O as External observer
    participant C as CDN / reverse proxy
    participant W as Web PHP status projection
    participant S as Authoritative supervisor state
    participant J as Server scheduler
    participant P as Async provider
    participant A as Edition artifact store

    O->>C: GET documented read-only status
    C->>W: Forward or revalidate
    W->>S: Read one committed snapshot
    S-->>W: process_id, revision, target, stage, deadlines
    W-->>C: JSON + no-store + request/snapshot IDs
    C-->>O: Status representation

    J->>S: Acquire transition lease with fencing token
    S-->>J: Lease + expected revision
    J->>P: Retrieve the stored provider operation
    P-->>J: queued / in_progress / terminal result
    J->>S: Compare-and-swap transition and append event

    alt evidence qualifies
        J->>S: Reserve composition with next revision
        J->>S: Commit validated bilingual payload hash
        J->>A: Write temporary immutable artifacts
        J->>A: Atomic promote and read-back
        A-->>J: Artifact hashes and URLs
        J->>S: Mark published only after verified read-back
    else evidence fails bounded rules
        J->>S: Preserve batch, rejection report, retry/quarantine decision
    end

    O->>C: GET status and linked edition
    C-->>O: Same published process revision and artifact identity

RECOMMENDED — A status request must never be required to perform the scheduler’s reconciliation. Black-box monitoring should observe the system, not become the system’s liveness mechanism.

HTTP, canonical-origin, scheduler, and asynchronous-job test matrices

HTTP, cache, release, and canonical-origin matrix

OBSERVED — Public API documentation reportedly characterizes the status endpoint as read-only and noncacheable, but documentation is not evidence that every production response carries the intended headers. Actual response headers must be captured from independent clients.

RECOMMENDED — Add Cache-Status at each controlled cache where practical, a stable origin request ID, and an X-Daily-Brief-Snapshot value containing release, process-short-ID, and revision. The values must be nonsecret and must be generated from the same committed snapshot as the body. RFC 9211 exists specifically to standardize cache-debug information.

RECOMMENDED — Version service-worker cache names and delete obsolete caches during activation. Record the worker release in a response/debug panel. A newly installed worker can wait before controlling existing pages, and an update check installs a new worker only when its script differs; therefore multiple open tabs can legitimately remain under an older worker until lifecycle conditions are met.

Scheduler and asynchronous-job matrix

RECOMMENDED — Every scheduler invocation should record an outcome even when it performs no work: not-eligible, lease-contended, waiting-until, provider-polled, transition-committed, quarantined, or published. A missing invocation and an invocation that correctly deferred work must never look identical.

RECOMMENDED — Scheduler concurrency must be explicitly controlled. Scheduled-job systems can permit overlapping executions by default; for example, Kubernetes CronJobs have explicit Allow, Forbid, and Replace concurrency policies. Regardless of scheduler product, the application still needs one authoritative transition lease and revision check because scheduler-level exclusion does not cover browser or operator actors.

RECOMMENDED — Use a monotonic duration to determine whether a locally held lease needs renewal, but persist an absolute UTC expiry for diagnosis. Every state mutation must include the lease’s fencing token and the expected process revision. If either is stale, the write must fail without side effects.

RECOMMENDED — Reconciliation must be level-triggered rather than edge-triggered. At every invocation, read authoritative state and ask, “What action is now due?” Do not depend on one particular prior wake-up having occurred. This allows a missed cron tick to self-recover while retaining exactly-once state transitions through idempotency and compare-and-swap.

Retry contracts

RECOMMENDED — Retries are safe only when the ambiguity, deduplication mechanism, success condition, stop condition, and retained evidence are explicit. Idempotent operations are the prerequisite for safe automatic retries; backoff and jitter control retry pressure but do not themselves prevent duplicate side effects.

Ranked fault tree and hypothesis analysis

RECOMMENDED — The diagram separates control-plane liveness, provider/evidence progression, publication, and visibility. A branch can coexist with another; for example, an evidence-validation defect can be hidden behind a stale status cache.

flowchart TD
    A[Daily Brief is not current or views disagree]

    A --> B[No authoritative cycle advances]
    A --> C[Provider or evidence lifecycle does not advance]
    A --> D[Composition or publication does not complete]
    A --> E[Publication completed but is not visible]
    A --> F[Multiple authorities or releases exist]

    B --> B1[Scheduler never starts]
    B --> B2[CLI exits before reservation]
    B --> B3[Lease remains held or stale writer is not fenced]
    B --> B4[Retry deadline comparison never becomes due]
    B --> B5[Scheduler and web see different state]

    C --> C1[Provider ID lost]
    C --> C2[Known provider operation never polled]
    C --> C3[Terminal output retrieved but not retained]
    C --> C4[Binder rejects source identities]
    C --> C5[Validation rejects all candidates]
    C --> C6[Accepted evidence discarded between rounds]

    D --> D1[Compose transition never reserved]
    D --> D2[Composition terminal output fails validation]
    D --> D3[Publication write or atomic promote fails]
    D --> D4[Artifact written but supervisor CAS fails]

    E --> E1[Origin/reverse-proxy/CDN cache is stale]
    E --> E2[Browser or service worker is stale]
    E --> E3[Artifact exists but journal link is missing]
    E --> E4[Client displays inferred progress]

    F --> F1[Alternate hostname serves old document root]
    F --> F2[English and Spanish use different state roots]
    F --> F3[Cron and web load different release/config]
    F --> F4[Rollback left incompatible state schema]
    F --> F5[Operator and cron race]

INFERRED — The probability estimates below are investigative priors, not measured frequencies. They are ranked from the August 1 public evidence and should be updated as telemetry arrives.

Detailed falsification logic for major layers

RECOMMENDED — Scheduler: The scheduler hypothesis is strengthened only by a missed expected invocation or a started invocation with no durable outcome. A null public heartbeat is insufficient. It is falsified when owner evidence shows on-time invocations whose actor, release, state path, lease result, and transition result are all recorded.

RECOMMENDED — PHP CLI runtime: The runtime-divergence hypothesis is strengthened by any difference in PHP binary, SAPI, loaded php.ini, scanned INI directory, extension set, open_basedir, timezone, process environment, working directory, user/group, chroot, document root, or application release. It is falsified only when these identities and the authoritative state object are shown to match the web runtime. PHP documentation confirms that FastCGI per-directory settings and FPM pool configuration can create such differences.

RECOMMENDED — Filesystem/database: This hypothesis is strengthened when two actors report different snapshot hashes at the same revision, when inode/mount/database identities differ, or when one actor can read but not atomically promote an artifact. It is falsified by an owner-supplied cross-runtime write/read probe performed in staging or against a dedicated diagnostic object—not against the live edition state.

RECOMMENDED — Supervisor state: This hypothesis is strengthened by revision regressions, mixed fields from different commits, absent compare-and-swap failures, or a terminal artifact that is not referenced by the supervisor. It is falsified by an append-only transition log in which every public snapshot maps to one committed revision and one snapshot hash.

RECOMMENDED — Provider orchestration: This hypothesis is strengthened when a stored operation is never retrieved after it becomes due, when create outcomes are ambiguous, or when terminal provider output lacks a retained application hash. It is falsified when the same operation is periodically retrieved, every result is committed, and terminal output is durably preserved before validation.

RECOMMENDED — Evidence validation: This hypothesis is strengthened by a nonempty provider batch, empty retained docket, stable rejection classes, or accepted evidence disappearing between revisions. It is falsified when the exact retained batch and validator release reproduce a qualified docket in a deterministic replay.

RECOMMENDED — Composition: This hypothesis is strengthened only after thresholds are met but no compose reservation occurs, or after a retained composition repeatedly fails explicit final rules. It is currently lower priority because the sampled process reported zero compose attempts.

RECOMMENDED — Publication write: This hypothesis is strengthened by a passed final validation followed by permission, path, disk, temp-write, rename, or read-back failures. It is falsified by a verified immutable artifact whose hash is referenced by the supervisor.

RECOMMENDED — Caches and service workers: This family is strengthened when the same URL produces different revisions by cache source, network, profile, or worker controller. It is falsified when fresh-origin and ordinary public views carry the same snapshot within the declared propagation objective. Service workers can replace network responses with cached objects, so they must be inspected separately from ordinary Cache-Control.

RECOMMENDED — Deployment drift: This family is strengthened by different release IDs, PHP/runtime fingerprints, process schemas, static-asset hashes, redirects, or state roots across host/language/SAPI. It is falsified only by a deployment manifest proving that all public entry points and scheduler actors resolve to the same immutable release.

Minimum telemetry, incident runbooks, and operator checklist

Minimum additional telemetry from the owner

RECOMMENDED — The owner should supply sanitized, read-only evidence for the affected target and process. Provider identifiers may be replaced with stable hashes; paths may be tokenized consistently; secrets and content requiring protection should be removed.

RECOMMENDED — The public status should expose a sanitized subset sufficient for black-box diagnosis: current UTC server time, status-generated time, scheduler last-start/last-finish/last-result, process ID and revision, lease state without sensitive host details, provider lifecycle and poll sequence, last meaningful transition, exact next due action, artifact existence, and release/runtime fingerprint.

Incident evidence-collection runbook for one publication cycle

RECOMMENDED — The following is a 24-hour read-only run. During daylight saving time, 8:00 a.m. America/Chicago is 13:00 UTC. The runbook should calculate the exact offset from the IANA timezone on the execution date rather than hard-code it.

RECOMMENDED — Sampling should be event-sensitive: low frequency while no action is due, temporarily increased around the known next-check boundary, and reduced after the state becomes terminal. This avoids unauthorized load while still determining whether deadlines cause transitions.

RECOMMENDED — The observer must not press publication or recovery controls, construct undisclosed endpoints, cancel provider operations, clear server locks, or provoke duplicate work. The only browser changes allowed are local cache/worker inspection in isolated profiles.

Confirmation plan across multiple cycles

RECOMMENDED — The 72-hour plan must prove repeatability, not merely one successful recovery.

RECOMMENDED — During the second and third cycles, compare:

• RECOMMENDED — process creation time relative to eligibility;
• RECOMMENDED — provider operation count per process stage;
• RECOMMENDED — poll interval and poll-sequence monotonicity;
• RECOMMENDED — evidence retained across repair rounds;
• RECOMMENDED — time from provider terminal result to state transition;
• RECOMMENDED — time from final validation to readable artifact;
• RECOMMENDED — time from artifact commit to public-language and hostname convergence;
• RECOMMENDED — scheduler heartbeat continuity and runtime fingerprint consistency.

One-page operator checklist

Remediation priorities, acceptance criteria, and unanswered questions

Ranked remediation matrix

RECOMMENDED — Black-box monitoring should be symptom-oriented: it should verify that a current edition becomes publicly readable, not merely that PHP returns 200. Google’s SRE guidance distinguishes black-box monitoring as observation of active user-visible symptoms and recommends synthetic transactions where real traffic may be absent.

Remediation design details

RECOMMENDED — Server-owned liveness: Remove production progress mutations from ordinary public page requests. A scheduler should run frequently enough that one missed tick does not miss the publication window. Each tick should reconcile the current state, not assume it is beginning a new attempt.

RECOMMENDED — One authoritative state transition: Model each mutation as:

transition(
    process_id,
    expected_revision,
    fencing_token,
    actor_id,
    operation_key,
    input_manifest_hash,
    proposed_event
) -> committed_revision | conflict

RECOMMENDED — Durable provider intent: Before a provider create call, commit a state stating that creation is reserved for a specific process, stage, and operation key. After receiving an identifier, commit it immediately. If the response is lost after the provider might have accepted the request, enter create-outcome-ambiguous; do not pretend no operation exists.

RECOMMENDED — Evidence continuity: Never replace a retained evidence set with a later empty set. Store each batch immutably and derive a merged working set by deterministic rules. A repair pass should add or supersede evidence through explicit lineage, not mutate the historical record.

RECOMMENDED — Validation transparency: Every rejected candidate should identify the rule code, relevant source IDs, observed values, expected condition, validator release, and whether the failure is repairable. A summary count of 42 issues is useful for UI display but insufficient for incident reconstruction.

RECOMMENDED — Atomic publication: Construct English and Spanish artifacts and their manifest under temporary names, validate links and language parity, promote them to immutable final names, read them back, and only then commit the supervisor’s terminal state and mutable indexes. If a final file already exists, compare its content hash; an identical artifact is idempotent success, while a different artifact is a conflict requiring quarantine.

RECOMMENDED — Cache separation: Dynamic status should be generated from one current committed snapshot and should not be stored by browsers, service workers, or shared caches. Immutable dated editions may be cached for long periods because their contents do not change. Mutable journal indexes may use bounded cache lifetimes plus validators and explicit invalidation on publication. RFC 9111 provides the relevant cache-control and revalidation semantics.

RECOMMENDED — Canonical deployment: Redirect alternate hosts before loading application publication state. Include the immutable release ID and state-schema version in the status snapshot. The scheduler should verify the same expected release at startup and fail visibly rather than modify state from a mismatched rollback.

RECOMMENDED — Runtime parity: Capture PHP_SAPI, PHP version, loaded configuration path, scanned INI files, extension-manifest hash, timezone, current working directory, effective user/group, application release, and redacted state-store identity at every scheduler start. Do not expose a public phpinfo() page; PHP documentation notes that it reveals extensive environment and server data.

RECOMMENDED — Synthetic monitoring: Run read-only probes from at least two approved external locations. Check the status snapshot age, revision liveness, overdue next action, latest edition date, linked artifact availability, English/Spanish identity, canonical redirects, and cache headers. Synthetic checks must not initiate publication.

Production acceptance criteria

RECOMMENDED — Acceptance must be based on multiple real publication cycles, including at least one restart or missed-tick recovery exercise in a safe environment.

RECOMMENDED — A passing result is not “the progress bar moved.” A passing result is a traceable chain from scheduler invocation to one immutable bilingual edition, with one process lineage, one operation lineage per stage, retained evidence, an atomically verified artifact, and converged public views.

Unanswered questions