Black-Box Incident Reconstruction and Reliability Plan for the InternationalIntelligence.org Daily Brief

Full report

On this page

Incident assessment

Executive finding and most likely failure families

OBSERVED. On August 1, 2026, the public Daily Brief page and public status API showed that production was not completely inert. They exposed an active target-scoped process for August 1, process ID dbp_20260801_c9fb48f6d62f8513, revision 34, an apparently stored provider response identifier, and an active background response in queued state. The same public snapshot also showed that the edition was unpublished, the latest edition remained July 30, five dates were missing, and today_url was null.

OBSERVED. The most important public contradiction is that the process reported “Next check: Due now,” next_check_at was already in the past relative to the status response’s displayed server time, the background response’s poll count was zero, and every public Daily Brief cron-heartbeat field was null. The public page simultaneously said that live status refresh was active and that the same stored response would be polled on the next bounded cycle.

INFERRED. The strongest current failure family is therefore supervisor or scheduler non-advancement after asynchronous work has been reserved, not initial provider connectivity. The evidence is consistent with one or more of these mechanisms:

INFERRED. A second independent problem family may be present even if scheduler reconciliation is repaired: the earlier research result appears to have reached the application but failed to produce any retained docket. The public state reported 120 provider-returned sources, 14 rejected candidates, zero exact source binds, zero candidates with evidence, and a deterministic fallback. That pattern is much more consistent with a data-contract, evidence-identity, date-normalization, corroboration, or validator-policy mismatch than with a provider outage.

UNKNOWN. The sampled background operation was only several minutes old, so the observation does not prove that this particular provider response was permanently stuck. The stronger finding is that the visible supervisor had not yet polled it despite reporting that a check was due, and no cron execution was publicly recorded. A complete publication-cycle trace is required before assigning final root cause.

Known facts, reported facts, assumptions, and unknowns

Production reality versus test success

Why passing deterministic tests can coexist with a broken production feature

INFERRED. Deterministic tests prove that a function or transition can succeed under their fixture, process environment, path layout, clock, and input ordering. They do not prove that production invokes that function, invokes it at the correct time, supplies the same state, persists the output durably, or subsequently reconciles asynchronous completion.

RECOMMENDED. Treat the Daily Brief as a long-running distributed transaction rather than as a single PHP request. Its correctness boundary spans at least scheduler dispatch, lease acquisition, state reservation, provider creation, durable identifier storage, periodic retrieval, validation, retained evidence, composition, publication write, supervisor update, cache invalidation, localized rendering, and browser convergence.

INFERRED. A validator suite can pass while production remains broken through any of these separations:

RECOMMENDED. Production acceptance must therefore assert observed end-to-end state transitions on the deployed host, including timestamps and artifact hashes, rather than merely assert that validators return success.

Symptom taxonomy

Black-box reconstruction

Black-box observation plan and evidence log

RECOMMENDED. The observation program should be passive except for ordinary GET and HEAD requests, normal browser navigation, lawful cache revalidation, and owner-approved fresh private sessions. It must not call publication-control endpoints, manufacture authorization claims, repeatedly force refresh triggers, or send request rates materially above normal use.

RECOMMENDED. Use one coordinated observation matrix per sampling instant:

RECOMMENDED. HTTP cache tests must interpret metadata according to HTTP semantics. Age describes time spent in a cache, Date is not a reliable application revision, ETags are representation validators, and Last-Modified is generally weaker than a content-specific strong validator. no-store is the appropriate directive when reuse of a sensitive or rapidly changing representation must be prevented; revalidation directives alone do not mean that no copy exists.

RECOMMENDED. Capture the following evidence-log record for every observation:

RECOMMENDED. A suitable filename convention is:

20260801T213637.123Z__canonical__en-US__daily-brief__chromium-private__normal-cache__obs-0042.*

RECOMMENDED. Body hashes must be computed over raw bytes before prettification, localization normalization, or JSON key sorting. A second semantic hash may be computed over a canonical subset containing process ID, revision, target, stage, provider status, next check, and publication URL; the raw hash and semantic hash serve different purposes.

UTC chronology reconstruction method

RECOMMENDED. Build one event ledger in UTC and retain every original timestamp and timezone. Never overwrite source timestamps with normalized values.

RECOMMENDED. For every event, record five distinct time concepts:

INFERRED. The current status illustrates why these distinctions matter. Its top-level feed-generation timestamp was earlier than some Daily Brief process timestamps embedded in the same response, while another field exposed a later server time. This may be a legitimate aggregation of state produced at different times, but the public document needs separate field names to prevent investigators from interpreting one timestamp as the age of the entire response.

RECOMMENDED. Sort primarily by source_event_at_utc, secondarily by observed time, and never infer event order solely from wall-clock timestamp when two writers may be active. Use process revision and transition sequence as the causal order. A later wall-clock event with a lower revision must be classified as stale evidence, not as a rollback, unless the authoritative store itself accepted the lower revision.

RECOMMENDED. Flag these chronology violations automatically:

Interface and execution matrices

HTTP, cache, and canonical-origin test matrix

OBSERVED. The public documentation states that the status endpoint should use no-store, while feed and RSS endpoints intentionally support bounded public caching and ETag revalidation. The OpenAPI document identifies the non-www HTTPS origin as canonical.

OBSERVED. A public request to the www Daily Brief route redirected to the non-www route, and the resulting page showed the same release and process information as the canonical request.

UNKNOWN. Raw Cache-Control, Age, Date, ETag, Last-Modified, Vary, CDN cache-status, and origin request-ID headers were not captured in this research environment. The following matrix is therefore a required field investigation, not a claim that those headers are currently correct.

RECOMMENDED. Determine whether two pages show the same or different state as follows:

RECOMMENDED. A service worker must be treated as a separate programmable cache and proxy. Its active worker can continue controlling clients while a new worker is waiting, and registrations persist beyond individual pages. Therefore, “hard reload” is not a sufficient service-worker diagnostic by itself; capture controller identity, registration scope, active/waiting script hashes, and whether “bypass for network” changes the response.

Scheduler and asynchronous-job test matrix

OBSERVED. The August 1 snapshot reported a stored provider response with a recorded ID, provider status queued, poll count zero, a due next check, attempt count four of four, and no cron heartbeat.

RECOMMENDED. Use the following passive matrix over one full intended cycle:

RECOMMENDED. The asynchronous provider reconciliation rule should be explicit:

CREATE operation
  reserve stage operation key
  persist create-intent and operation key
  submit at most once while result is ambiguous
  persist provider response ID before releasing writer ownership

RETRIEVE operation
  use the same stored provider response ID
  poll only when eligible
  while state is queued or in_progress, record observation and schedule next poll
  on terminal completion, persist output hash and advance exactly once
  on terminal failure, record terminal code and apply bounded stage policy

RECOMMENDED. OpenAI’s official background-mode guidance says to retrieve the same background response while it is queued or in_progress and treat departure from those states as terminal. This makes provider polling a reconciliation responsibility of the application; successful creation alone is not completion of the workflow.

RECOMMENDED. Retry behavior must be defined by ambiguity class:

RECOMMENDED. Safe retries require stable request identity and result reconciliation. Idempotency guidance from Stripe and AWS emphasizes that a retry must not create a second side effect and that indeterminate outcomes must be reconciled rather than blindly repeated.

Fault isolation

Ranked fault tree

INFERRED. The current public observations place the likely break between “operation reserved” and “qualified evidence ready,” with especially strong evidence at the reconciliation and validation branches.

flowchart TD
    A[Daily Brief absent or stale] --> B{Active target process exists?}

    B -- No --> C[Eligibility or scheduler branch]
    C --> C1[Scheduler never invoked]
    C --> C2[CLI bootstrap exits]
    C --> C3[Different release, path, user, or state store]
    C --> C4[Lease falsely held]

    B -- Yes --> D{Provider operation recorded?}
    D -- No --> E[Reservation or create branch]
    E --> E1[Create never attempted]
    E --> E2[Create accepted but ID lost]
    E --> E3[Attempt cap blocks creation]

    D -- Yes --> F{Provider operation reconciled?}
    F -- No --> G[Highest-ranked current branch]
    G --> G1[Cron absent or not firing]
    G --> G2[Browser-trigger dependency]
    G --> G3[Retry time never serviced]
    G --> G4[CLI/web state-store split]
    G --> G5[Lease or clock disagreement]

    F -- Yes --> H{Evidence retained?}
    H -- No --> I[Strong second current branch]
    I --> I1[Source-ID binding mismatch]
    I --> I2[Freshness timezone/date rejection]
    I --> I3[Corroboration or regional gates impossible]
    I --> I4[Valid retained evidence overwritten]

    H -- Yes --> J{Composition valid?}
    J -- No --> K[Composition or final-validation defect]

    J -- Yes --> L{Artifact committed?}
    L -- No --> M[Permissions, path, filesystem, or atomic-write failure]

    L -- Yes --> N{Supervisor and public views converge?}
    N -- No --> O[State roll-forward, index, cache, CDN, service worker, or deployment drift]

INFERRED. Probability ratings below are conditional engineering judgments, not statistical measurements. “High” means the hypothesis directly explains multiple current observations; “medium” means plausible but not yet discriminated; “low” means current evidence argues against it but does not eliminate intermittent occurrence.

Request-sequence reconstruction and snapshot discrimination

RECOMMENDED. The target request sequence should be reconstructed as a causally ordered state machine:

sequenceDiagram
    participant S as Scheduler or eligible page request
    participant W as Transition writer
    participant D as Durable supervisor state
    participant P as Provider
    participant V as Validators
    participant F as Publication store
    participant C as Public caches
    participant B as Browser

    S->>W: Start cycle with invocation ID
    W->>D: CAS acquire writer lease and fencing epoch
    W->>D: Reserve target/process/stage operation
    W->>P: Create background operation once
    P-->>W: Response ID and queued state
    W->>D: Persist response ID, revision, next_check_at
    W-->>S: Cycle complete

    S->>W: Later reconciliation cycle
    W->>D: Acquire new writer lease
    W->>D: Load same process and response ID
    W->>P: Retrieve same background response
    P-->>W: queued, in_progress, completed, or failed
    W->>D: Record poll and provider state

    alt Provider completed
        W->>V: Validate and retain evidence
        V-->>W: Immutable vetted manifest or bounded rejection
        W->>P: Compose from immutable manifest if qualified
        P-->>W: Bilingual composition
        W->>V: Final deterministic validation
        W->>F: Write temp in destination filesystem
        W->>F: Atomic rename/create-once commit
        W->>D: CAS published state with artifact hash
        W->>C: Purge or revalidate mutable indexes
        B->>D: GET no-store status
        B->>F: GET immutable edition
    else Provider still active
        W->>D: Set next eligible poll
    else Terminal failure
        W->>D: Record terminal code and bounded retry/quarantine
    end

RECOMMENDED. Every arrow that can change state should carry: invocation ID, process ID, target date, expected revision, new revision, writer identity, lease/fencing epoch, stage operation key, and result classification.

RECOMMENDED. A process should never infer success merely because an earlier function returned without exception. Success must be recognized by the durable postcondition relevant to that stage: stored provider ID, terminal output hash, vetted manifest hash, composed-output hash, committed artifact hash, or published supervisor revision.

Owner telemetry and operational runs

Minimum additional telemetry needed from the owner

RECOMMENDED. The owner need not disclose credentials, prompts, source bodies, private candidate text, or absolute private paths. A sanitized support bundle can confirm nearly every major hypothesis with the following fields:

RECOMMENDED. The most valuable single addition is an append-only, target-scoped transition journal. A mutable status snapshot says only what the system claims now; a journal shows how it got there and whether a later writer overwrote valid earlier state.

RECOMMENDED. The public status should distinguish these times explicitly: response generation time, state snapshot time, last scheduler observation, last meaningful transition, provider update time, next action time, and publication deadline. The current aggregate contains several time domains but not enough naming to prevent ambiguity.

Twenty-four-hour incident evidence-collection runbook

RECOMMENDED. The run begins before the next intended eligibility boundary and uses synchronized UTC clocks.

RECOMMENDED. Sampling should be event-aligned rather than aggressive. The public API asks consumers to honor caching and avoid continuous polling, and production synthetic monitoring should imitate ordinary user behavior rather than apply unauthorized load.

RECOMMENDED. The operator should classify each overdue condition:

Seventy-two-hour confirmation plan and operator checklist

RECOMMENDED. The 72-hour plan must cover at least three target transitions because one successful edition can be a coincidence caused by browser traffic or manual intervention.

RECOMMENDED. Use independent black-box synthetics because black-box monitoring tests the user-visible symptom rather than merely internal component health. Google’s SRE guidance distinguishes symptom-oriented black-box monitoring from white-box internal telemetry and recommends synthetic traffic where real-user traffic may be absent.

One-page operator checklist

Remediation and acceptance

Prioritized remediation recommendations

RECOMMENDED. The order below favors changes that improve both reliability and diagnosability while minimizing the chance of duplicate publication.

RECOMMENDED. Distributed leases alone are insufficient if an expired owner can still write. Kubernetes Lease records illustrate the minimum useful ownership data—holder identity, acquisition, renewal, duration, and transition count—while production leader-election guidance warns that slow execution and pauses can consume the remaining lease time. Every write should therefore include a fencing epoch or compare-and-swap revision that rejects a stale owner even if it resumes later.

RECOMMENDED. Cron should be invoked with an explicit PHP binary, absolute script and state paths, explicit working directory, explicit timezone, and controlled environment. Cron’s default environment is limited, and PHP CLI is a distinct SAPI from web execution; relying on interactive-shell PATH, relative files, or web-server configuration is unsafe.

RECOMMENDED. Publication should use a same-filesystem temporary file in the destination directory, validate its size and cryptographic hash, perform any required durability flushes supported by the platform, and then atomically rename it into the immutable dated path. A successful same-filesystem rename can atomically replace a name, but this must not be generalized to cross-filesystem moves or unsupported storage abstractions.

RECOMMENDED. Mutable journal, home-page, RSS, and status indexes should point to the immutable artifact by hash and edition ID. If index update fails, the system should preserve the artifact and reconcile forward; it should not regenerate the edition.

RECOMMENDED. The browser should never infer authoritative progress from elapsed time. It should display only a server revision and server-declared state. An older tab must ignore lower revisions, and any write-capable browser trigger must include the current process and expected revision so the server can reject stale requests.

Production acceptance criteria and unanswered questions

RECOMMENDED. The feature is accepted only when all of the following are demonstrated in production for three consecutive intended publication cycles:

RECOMMENDED. The acceptance test must deliberately include controlled fault injection in a non-production-equivalent environment and passive confirmation in production: scheduler interruption after provider creation, process restart before ID commit, provider retrieve timeout, validator rejection of part of a batch, composition failure, artifact-write interruption, supervisor-update interruption, stale browser tab, and stale service worker. Production should not be modified merely to stage these failures.

Unanswered questions