Black-Box Diagnostic and Remediation Report: Daily Brief Publication Lifecycle Anomalies

Full report

On this page

1. Executive Finding and Most Likely Failure Families

(INFERRED) The chronic unreliability of the Daily Brief publication system on InternationalIntelligence.org is not rooted in upstream provider unavailability, basic network connectivity failures, or simplistic credential errors. Instead, the architecture is experiencing a complex confluence of state-reconciliation faults, HTTP cache coherence failures, and runtime-isolation boundaries. The system is structurally vulnerable to split-brain scenarios and data-consistency violations caused by differing process environments and the lack of strict concurrency controls over asynchronous operations. (INFERRED) The most critical failure families destabilizing the publication lifecycle are centered around three architectural domains. First, runtime environment divergence is highly probable. The web runtime (PHP-FPM) and the scheduled runtime (PHP CLI/cron) are likely operating in isolated temporary filesystems due to modern operating system service isolation protocols, or they are experiencing bytecode staleness across the Server API (SAPI) boundary1. This divergence causes scheduled tasks and browser-triggered processes to initiate duplicate work, failing to converge on a single authoritative state. Second, the system appears to rely on distributed lease expiration without monotonic fencing. The reported browser trigger lease allows a process to reserve work, but long-running provider API calls can easily outlive this time-based lease due to network latency or execution pauses4. Without monotonic fencing tokens checked at the final storage layer, a delayed response from an expired process can awaken and overwrite a newer, active process state, violating the strict forward-only progression invariant4. Third, aggressive HTTP cache coherence and content negotiation failures are disrupting the client-side view of the system. The divergence between English and Spanish views, as well as cross-tab inconsistencies, strongly indicates that intermediary Content Delivery Network (CDN) or reverse proxy caches are mishandling request variations5. Furthermore, the deployment of asynchronous revalidation directives without coordinated cache invalidation allows edge nodes to serve stale states indefinitely, effectively masking successful background publication transitions from the public domain6. (RECOMMENDED) Systemic resolution requires replacing time-based leases with token-fenced atomic operations, unifying the filesystem and bytecode namespaces between cron and web workers, and strictly defining HTTP caching directives for all stateful API endpoints.

2. Known Facts, Assumptions, Reported Facts, and Unknowns

To establish a rigorous diagnostic foundation, the operating parameters and architectural realities must be explicitly categorized. (REPORTED BY PROJECT OWNER) Prior diagnostics have established that lower-level infrastructure and dependencies are healthy. DNS resolution, TLS verification, PHP cURL functionality, OpenSSL integrity, the system CA bundle, OpenAI credentials, and the configured language models are all operational. Furthermore, the Responses API, required web search integrations, strict structured output generation, and the creation and retrieval of background responses have passed isolated validations. Therefore, diagnostics will not waste resources probing these validated domains. (REPORTED BY PROJECT OWNER) The application architecture has evolved to include specific reliability mechanisms. Earlier releases introduced a canonical host definition, a discrete Daily Brief process identifier, monotonic revision sequencing, cross-tab synchronization for browser clients, browser polling, a browser trigger lease, and a server-owned supervisor concept. (REPORTED BY PROJECT OWNER) Many deterministic, no-network validators have passed in testing environments, yet the production feature remains broken. (OBSERVED) The public Daily Brief system fails reliably in the production environment. The public manifestations of these failures include the system remaining entirely stale, failing to publish a final artifact, showing an outdated target date, remaining permanently stuck in an intermediate stage, or appearing highly inconsistent across different pages, browser tabs, or user sessions. (INFERRED) The system utilizes a PHP-based application server architecture, likely separating web traffic handling (e.g., PHP-FPM behind Nginx or Apache) from background task execution (PHP CLI triggered by a cron daemon). (INFERRED) The infrastructure relies on one or more layers of edge caching, reverse proxies, or CDNs to serve global traffic, which actively mediates and potentially alters the headers and caching semantics between end users and the origin server. (UNKNOWN) The exact CDN provider, the specific PHP version, the explicit reverse proxy configuration, and the final storage medium for the publication artifact (e.g., local disk, Network File System (NFS), or cloud object storage) remain unverified from an external perspective. (UNKNOWN) It is unverified whether the production environment relies on a distributed multi-node cluster behind a load balancer or a single monolithic computing instance, which drastically alters the probability of race conditions.

3. Why Passing Deterministic Tests Coexist with a Broken Production Feature

(INFERRED) Deterministic testing typically verifies sequential logic within a strictly controlled, uniform environment. The failure of the Daily Brief feature in production, despite passing tests, is a classic manifestation of distributed systems anomalies, deployment environment drift, and the fundamental differences between synthetic isolation and live execution. The most prominent discrepancy arises from the Server API (SAPI) boundary and memory isolation. Deterministic tests almost exclusively execute in a single PHP CLI process. In a production environment, the system relies on PHP-FPM for handling web requests and PHP CLI for scheduled cron jobs. These SAPIs do not share the same OPcache memory space3. If opcache.revalidate\_freq is set to 0 or file timestamps are not actively validated, one SAPI will serve outdated bytecode while the other executes newly deployed logic3. This means a manual browser trigger might execute one version of the supervisor logic, while cron executes an incompatible older version, leading to corrupted state transitions. Furthermore, runtime filesystem isolation introduces severe split-brain vulnerabilities. Modern Linux distributions running systemd frequently enforce the PrivateTmp=true directive for daemons like PHP-FPM to enhance security2. If the PHP application utilizes functions like sys\_get\_temp\_dir() to create lock files, sqlite databases, or state flags, the web runtime writes to an isolated namespace (e.g., /tmp/systemd-private-...), while cron writes to the global /tmp directory10. Tests run by a single continuous integration user never encounter this split-brain scenario because they operate within a unified filesystem view. In production, this causes the web interface to report no active process, prompting it to start one, while cron simultaneously runs its own invisible process. Time and asynchrony represent another massive divergence between test and production. Deterministic tests mock clocks and network latency, assuming operations complete linearly. In production, a process might acquire a lease, initiate an asynchronous provider request, and then experience a garbage collection pause, a CPU throttle, or a network delay12. By the time the process awakens to write its data, the time-based lease has expired, and another process has taken over. Without strict fencing tokens, the delayed process overwrites the new process, corrupting the monotonic progression4. Finally, deterministic tests bypass CDN edge nodes and HTTP caching middleware. In production, HTTP caching directives such as stale-while-revalidate or an improperly configured Vary header will cause the edge cache to serve older representations of the process state5. This violates the required invariant that a state cannot visually or logically move backward, as the CDN will actively serve a cached response that contradicts the true origin state.

4. Symptom Taxonomy

(RECOMMENDED) To accurately diagnose the failure without privileged access, we must classify external observations into a highly explicit symptom taxonomy. This taxonomy distinguishes between failures in initialization, execution state corruption, output validation, and presentation layer desynchronization. The taxonomy of initialization failures begins with instances where the scheduler never starts. (INFERRED) If the target date remains perpetually in the past and no new process ID is ever generated, it suggests cron is misconfigured, the cron daemon has halted, or the PHP CLI binary path is invalid in the production environment. Alternatively, the scheduler starts but exits before reserving work. (INFERRED) In this scenario, cron executes but immediately aborts because it reads a stale lock file indicating another process is active, or the cron user lacks the filesystem permissions required to write to the lock directory, leaving the system permanently dormant. A related symptom is when one process is permanently leased. (INFERRED) This occurs when a process crashes mid-execution (e.g., due to an out-of-memory error) without triggering a try/finally block to release the lease. If the supervisor lacks a dead-letter timeout or heartbeat monitor to forcefully expire abandoned leases, the system will never initiate a new brief. State corruption and provider reconciliation failures form the next category. When provider work is created but its identifier is lost, (INFERRED) the process successfully initiated asynchronous provider work (like an OpenAI API call) but crashed or timed out before committing the returned provider job ID to persistent storage. The work completes upstream, but the server has no record of how to retrieve it. Conversely, provider work completes but is never retrieved. (INFERRED) Here, the job ID is saved, but the background retrieval polling mechanism is starving for CPU time, or the callback webhook is being blocked by an aggressive edge firewall or caching layer. Validation and transition anomalies often masquerade as systemic lockups. If provider output is retrieved but evidence validation rejects it, (INFERRED) the application logic is functioning, but the structured output schema is too rigid, or the provider is injecting unexpected formatting (e.g., markdown code blocks surrounding JSON) that the parser cannot sanitize, causing an infinite retry loop. If valid evidence is discarded between attempts, (INFERRED) state files are likely being written to ephemeral storage that is wiped between requests, or a load-balancer using round-robin routing without sticky sessions is sending subsequent polling requests to a cluster node that does not possess the local state files. When composition never starts, (INFERRED) the transition state from "evidence gathered" to "composition" relies on an event trigger that was swallowed by the system, or the supervisor fails to mathematically recognize the completion of all required preceding stages. If composition succeeds but final validation fails, (INFERRED) the composed English and Spanish texts likely contain mismatched data structures, missing required translation fields, or violate strict length invariants, preventing finalization. Terminal publication failures indicate issues at the storage layer. If the publication file write fails, (INFERRED) the PHP worker process lacks write permissions on the target webroot directory, or a Network File System (NFS) flock issue is preventing an atomic file rename, causing the system to abort the write14. If publication succeeds but the supervisor is not updated, (INFERRED) an unhandled exception occurs immediately after the successful file write but before the final database commit, leaving the artifact live but the supervisor perpetually waiting. Presentation layer and caching anomalies are responsible for severe user-facing inconsistencies. If the supervisor updates but public pages remain cached, (INFERRED) the origin server successfully completes the workflow, but the CDN continues serving an old payload because the max-age TTL has not expired and no active cache-purging mechanism is implemented16. When English and Spanish views diverge, (INFERRED) the CDN cache key lacks the Vary: Accept-Language header5. This causes a shared cache to conflate the two localized representations, serving Spanish content to English users or freezing one language's progress while the other advances. If one hostname serves a stale deployment, (INFERRED) the canonical host redirects correctly, but an alternate hostname bypasses the primary CDN configuration or routes traffic directly to an older, isolated backend instance. When browser JavaScript is stale, (INFERRED) the HTML document references an unversioned static asset, prompting the browser to cache it aggressively and execute old polling logic against new API endpoints. If a service worker or CDN serves an old status, (INFERRED) the service worker intercepts the client-side polling request and returns a cached response from the local CacheStorage API, trapping the client in a stale loop regardless of server health. Concurrency and clock anomalies represent the final taxonomy category. When operator action and cron race, (INFERRED) an operator manually triggers the brief via the web UI while cron triggers it simultaneously. Without strict atomic fencing, both processes acquire a localized view of the lock and execute duplicate, conflicting work. If the displayed progress is client-inferred rather than server-authoritative, (INFERRED) the frontend UI advances a visual progress bar based on arbitrary local timeouts rather than parsing true server state, leading to a visual completion that contradicts the backend reality. When the status endpoint is healthy but the edition artifact is missing, (INFERRED) the database reflects a terminal success state, but the actual HTML or JSON artifact was written to a temporary directory or a different node's isolated disk. If the edition artifact exists but is not linked from the public page, (INFERRED) the main index page cache is severely out of sync with the artifact cache, or the symlink atomic swap failed. If the process is waiting for a retry time that is never reached, (INFERRED) the exponential backoff algorithm calculated an integer overflow, or the retry timestamp was recorded in a local timezone while the supervisor continuously compares it against UTC. Finally, if a lock or lease expires according to one clock but not another, (INFERRED) NTP daemon drift between cluster nodes is occurring, or timezone configurations (date.timezone) differ fundamentally between the CLI and FPM runtimes.

5. Black-Box Observation Plan

(RECOMMENDED) To isolate the fault domain without privileged access, we must conduct a structured matrix of non-destructive external observations. The objective is to probe the public endpoints across different dimensions to map cache behavior, state transitions, origin consensus, and canonical enforcement. The observation plan relies entirely on analyzing public HTTP response headers, content negotiation, and temporal state progression. We begin by observing the English and Spanish Daily Brief pages discovered through normal site navigation. By utilizing browser developer tools and command-line utilities like curl, we extract origin response headers, specifically targeting Cache-Control, Age, ETag, Last-Modified, Vary, and Server. This establishes the baseline caching posture of the infrastructure. To map cache behavior comprehensively, we must inject lawful cache-busting query parameters (e.g., ?cb=123456) to bypass edge caching and view the true origin state17. We compare this bypassed state to requests made without cache busters. If the bypassed request shows a higher process revision number than the standard request, we have definitively proven that the edge cache is trapping stale content. We closely monitor the Age header; a continually incrementing Age across polling cycles confirms that requests are not reaching the origin supervisor. Testing across the canonical hostname and publicly known alternate hostnames is critical to identifying deployment drift. We request the Daily Brief from www.internationalintelligence.org, the apex domain, and any exposed direct IP addresses. We observe whether HTTP 301 or 302 redirects properly enforce a single canonical host18. If an alternate hostname returns a 200 OK with an independent, divergent process state, it indicates a split-brain deployment where alternate hostnames route to isolated document roots or older application instances. Fresh private browser sessions and multiple parallel tabs are utilized to passively monitor concurrency controls. By opening multiple tabs during the expected cron trigger window, we monitor the network tab for polling frequencies and XHR responses. If Tab A reports a "Composing" stage while Tab B reports a "Stale" stage, or if they display two different concurrent Provider Operation IDs, we can infer that distributed locking has failed and multiple supervisor views exist. Different browsers and mobile/desktop viewport widths are tested to ensure that Vary: User-Agent is not causing unnecessary cache fragmentation or routing to disparate mobile-specific backends. Requests are executed with JavaScript disabled to verify if the displayed progress is server-authoritative or client-inferred. If the page renders completely differently or fails to show a status without JavaScript, the system lacks progressive enhancement and relies entirely on client-side synchronization, masking true server state. Finally, we conduct repeated observations across at least one complete intended publication cycle (a full 24-hour window). We log the process identity, target date, stage, revision, provider status, last-progress time, next action, and publication date wherever publicly exposed. By tracking these variables chronologically, we map exactly when the state freezes, leaps forward, or maliciously rolls backward.

6. UTC Chronology Reconstruction Method

(RECOMMENDED) Because we lack internal server logs, we must reconstruct the server's internal timeline from the outside using HTTP caching headers. This mathematical reconstruction relies on standard RFC 9111 HTTP semantics to determine exactly when the origin server processed a state change.

1. Establish Baseline Time: The observer records their local time and immediately converts it to Coordinated Universal Time (UTC) at the exact moment an HTTP request is initiated. This is denoted as [Figure omitted from source export].

2. Extract the Date Header: The HTTP Date header represents the time the origin server generated the response19. It is critical to note that CDNs cache the Date header along with the body.

3. Extract the Age Header: The Age header indicates how many seconds the response has resided in a proxy or CDN cache since it was originally generated by the origin server19. A value of 0 indicates a direct origin hit, while a higher value indicates staleness.

4. Calculate Origin Generation Time ([Figure omitted from source export]): The true time the server snapshot was created is calculated as [Figure omitted from source export]. By subtracting the cache age from our strict observation time, we determine exactly when the origin server last successfully generated this specific state snapshot, accounting for clock skew.

5. Detect State Rollbacks: We map [Figure omitted from source export] against the observed Process Revision over time. If observation [Figure omitted from source export] yields a [Figure omitted from source export] of 14:00:00 UTC with Revision 5, and observation [Figure omitted from source export] ten minutes later yields a [Figure omitted from source export] of 13:45:00 UTC with Revision 4, the state has chronologically moved backward. This mathematically proves load-balancer desynchronization, CDN node divergence, or filesystem restoration, violating the forward-only invariant.

Chronology Template

(RECOMMENDED) All observations must be logged in this structured format to build a defensible incident timeline.

7. HTTP, Cache, and Canonical-Origin Test Matrix

(RECOMMENDED) Execute this matrix to determine whether two pages are displaying the same server snapshot, independently cached snapshots, two genuinely different backend processes, localized variants, or entirely different site instances.

Determination Logic:

1. The same current server snapshot: The requests yield an identical ETag, Date, and Body hash.

2. Two independently cached snapshots: The requests yield a different Age and Date (indicating different cache entry times), but possess an identical Process ID and Revision number.

3. Two genuinely different backend processes: The requests display a completely different Process ID, or one target date shows two completely divergent progression paths (e.g., Process A is in "Gather", Process B is in "Failed").

4. The same process with different localized labels: The requests possess an identical Process ID and Revision, display different language strings, and the response headers explicitly include Vary: Accept-Language.

5. Different releases or site instances: The DOM structure, CSS classes, or JavaScript bundle hashes differ entirely between the two requests, indicating load-balancer routing to un-synchronized deployment nodes.

8. Scheduler and Asynchronous-Job Test Matrix

(RECOMMENDED) To safely observe the interaction between the supervisor, the scheduler, and asynchronous providers without altering production state, execute the following passive matrix.

9. Fault Tree with Ranked Hypotheses

To systematically eliminate variables, we organize the suspected fault domains into a rigorous fault tree, ranking hypotheses by their probability of causing the observed invariants to fail.

Code snippet graph TD A\[Daily Brief Failure\] \--\> B(Cache/CDN Layer) A \--\> C(State & Concurrency Layer) A \--\> D(Runtime Isolation Layer)

B \--\> B1\[Stale-While-Revalidate Race\] B \--\> B2\[Accept-Language Cache Collision\]

C \--\> C1\[Lease Expiration without Fencing\] C \--\> C2\[Atomic Rename Failure over NFS\]

D \--\> D1\[PrivateTmp /tmp Split-Brain\] D \--\> D2\[OPcache CLI/FPM Desync\]

Hypothesis 1: Runtime Namespace Isolation (systemd PrivateTmp Split-Brain)

• Observable Symptoms: The public web page triggers a process, but the cron scheduler never advances it. Alternatively, cron succeeds silently in the background, but the web UI insists no brief exists and prompts for a manual trigger.
• Confidence Evidence: State files, filesystem locks, or SQLite databases are configured to reside in sys\_get\_temp\_dir() or /tmp10.
• Falsification Evidence: All state is definitively stored in a centralized, network-accessible database (e.g., MySQL/PostgreSQL) rather than the local filesystem.
• Likely Responsible Layer: PHP CLI runtime; Web PHP runtime; Filesystem; OS (systemd).
• Severity: Critical. Breaks the invariant that cron and web views converge.
• Probability: High. This is standard default behavior on modern Linux distributions (CentOS, Ubuntu) running systemd2.
• Safest Next Diagnostic Action: Check if cache-busted web requests align chronologically with expected cron execution times, or if they appear completely oblivious to the schedule.
• Minimum Telemetry Required: The output of sys\_get\_temp\_dir() from both a web-accessible script and a CLI script executed by the cron user.
• Can it Self-Recover: No.
• Safe Operator Intervention: No. Restarting the web server will merely clear its isolated /tmp directory, abandoning any processes it was tracking.
• Duplicate Work Risk: High. If intervention occurs, cron and FPM will continue to race.

Hypothesis 2: Lease Expiration without Monotonic Fencing Tokens

• Observable Symptoms: The publication process advances, halts abruptly, and suddenly regresses to an older state, or duplicate editions are published simultaneously.
• Confidence Evidence: Revisions numbers jump erratically; long delays exist between OpenAI API responses and state updates.
• Falsification Evidence: The architecture strictly utilizes a single node, and all provider API calls are synchronous, blocking requests that never yield the thread.
• Likely Responsible Layer: Supervisor state; Provider orchestration.
• Severity: High. Violates the invariants of "One immutable edition," "At most one server transition writer," and "State cannot move backward."
• Probability: High. Standard time-based distributed leases (like Redlock) are fundamentally unsafe for correctness in distributed systems experiencing garbage collection pauses or network delays4.
• Safest Next Diagnostic Action: Monitor the public status endpoint for interleaved Revision IDs or duplicated Provider Operation IDs over a 10-minute window.
• Minimum Telemetry Required: Supervisor logs showing the delta between LockAcquired timestamps and StateCommitted timestamps, compared against the lease TTL.
• Can it Self-Recover: Occasionally, if the delayed thread dies before writing.
• Safe Operator Intervention: No.
• Duplicate Work Risk: High risk. If an operator manually un-sticks the process by deleting a lock, the original delayed thread may awaken moments later and write stale data over the operator's new process. Requires fencing tokens to fix4.

Hypothesis 3: HTTP Cache Coherence and Vary Divergence

• Observable Symptoms: English and Spanish views diverge wildly. Users see old target dates unless they apply a cache-busting parameter.
• Confidence Evidence: The HTTP response lacks the Vary: Accept-Language header5. The Age header continually climbs despite the origin successfully updating behind the scenes19.
• Falsification Evidence: Cache-Control: no-store, private is strictly enforced on all API endpoints and HTML documents, and the CDN explicitly honors it24.
• Likely Responsible Layer: CDN; Reverse Proxy; Web PHP Runtime.
• Severity: Moderate. The backend system functions, but the public perceives it as broken.
• Probability: Very High. Cache invalidation is notoriously difficult to synchronize with asynchronous backend workflows.
• Safest Next Diagnostic Action: Compare response headers of the Daily Brief using strict Accept-Language: en and Accept-Language: es variations.
• Minimum Telemetry Required: Exported CDN Cache configuration rules and Page Rules.
• Can it Self-Recover: Yes, once the maximum TTL expires.
• Safe Operator Intervention: Yes. Purging the CDN cache is safe and non-destructive to the origin state.
• Duplicate Work Risk: Low.

Hypothesis 4: Atomic Rename Failure over Network Filesystem (NFS)

• Observable Symptoms: The publication file write fails entirely, leaving the system in a perpetual "Composing" state, or results in zero-byte artifact files being served to the public.
• Confidence Evidence: The infrastructure relies on NFS for shared storage across multiple web nodes, and the codebase utilizes flock() combined with rename() for publication.
• Falsification Evidence: The system uses single-node local NVMe storage or writes directly to an S3-compatible object store.
• Likely Responsible Layer: Filesystem; Publication write.
• Severity: High. Terminal publication failure violating the "immutability" invariant.
• Probability: Moderate. flock() semantics over NFS are notoriously unreliable, often translating to POSIX locks unexpectedly, and atomic renames can fail or block indefinitely15.
• Safest Next Diagnostic Action: Observe if the final artifact URL returns a 404 Not Found while the status endpoint insists the stage is "Complete".
• Minimum Telemetry Required: Storage architecture details (NFS vs Local block storage).
• Can it Self-Recover: No.
• Safe Operator Intervention: Yes, an operator could theoretically manually place the generated file if the payload can be extracted.
• Duplicate Work Risk: Low.

Hypothesis 5: OPcache SAPI Desynchronization

• Observable Symptoms: Code deployments or hotfixes are recognized by the web UI (FPM) but completely ignored by the cron scheduler (CLI), leading to data validation failures because one runtime expects a new schema while the other writes the old schema.
• Confidence Evidence: opcache.revalidate\_freq \= 0 is set in the php.ini, and no manual invalidation hook exists for CLI processes7.
• Falsification Evidence: OPcache is explicitly disabled for CLI via the opcache.enable\_cli=0 directive1.
• Likely Responsible Layer: PHP CLI runtime vs Web PHP runtime; Deployment drift.
• Severity: Moderate.
• Probability: Moderate.
• Safest Next Diagnostic Action: N/A from a black-box perspective, aside from inferring schema validation errors.
• Minimum Telemetry Required: The output of php \-i | grep opcache from both environments.
• Can it Self-Recover: No.
• Safe Operator Intervention: Yes, restarting PHP-FPM and the cron daemon clears the cache.
• Duplicate Work Risk: Low.

10. Minimum Additional Telemetry Needed from the Owner

(RECOMMENDED) To definitively confirm the hypotheses generated by the fault tree without requiring direct access to source code or sensitive credentials, the project owner must provide the following sanitized telemetry:

1. SAPI Temporary Path Outputs: The exact string output of php \-r 'echo sys\_get\_temp\_dir();' executed via SSH as the cron user, alongside the output of a web-accessible PHP script running the exact same command. This confirms or refutes the PrivateTmp split-brain hypothesis.

2. Filesystem Architecture Confirmation: Explicit confirmation of whether shared storage (e.g., NFS, AWS EFS, GlusterFS) is utilized for state files and artifacts, or if the system relies on local disk block storage.

3. OPcache Configuration Values: The configured values of opcache.enable\_cli, opcache.revalidate\_freq, and opcache.validate\_timestamps3.

4. Distributed Locking Mechanism Description: A high-level description of how the Supervisor acquires the browser trigger lease. Is it using filesystem flock, a MySQL GET\_LOCK, a Redis Redlock pattern, or a SQLite transaction? Do the writes include a monotonically increasing integer check?

5. CDN Page Rules: A sanitized export of the CDN rules dictating if edge nodes are overriding origin Cache-Control headers, specifically looking for forced edge TTLs or stripped Vary headers.

11. A 24-Hour Incident Evidence-Collection Runbook

(RECOMMENDED) Execute this runbook to safely map the Daily Brief cycle from the outside, utilizing synthetic monitoring emulation to trace the exact moment of failure. Prerequisites:

• A machine running a terminal with curl and jq (if JSON parsing is needed).
• No authenticated session to ensure true black-box observation.

Phase 1: Pre-Cycle Baseline (T-minus 1 hour from scheduled start)

1. Execute the following command to capture the baseline state of the English endpoint:curl \-svo /dev/null \-H "Accept-Language: en-US" https://internationalintelligence.org/daily-brief 2\>&1 | grep \-E "\< HTTP|\< Cache|\< Vary|\< ETag|\< Age|\< Date"

2. Log the output into the UTC Chronology Template.

3. Repeat the exact request using \-H "Accept-Language: es-ES".

4. Compare the ETag and Age values to establish if the languages share a cache key incorrectly.

Phase 2: Transition Observation (T-zero to T+2 hours)

1. Set up an automated polling script (cron job or while loop) on the observer machine to poll the status API (or scrape the HTML page) exactly every 60 seconds.

2. Critical Directive: Append a unique cache-buster (e.g., ?req\_id=$(date \+%s)) on alternating requests. This allows you to compare the Edge Cache view (without the parameter) against the True Origin view (with the parameter).

3. Parse the response for the Process ID, Target Date, Revision, and Stage. Append this data to the evidence log.

Phase 3: Post-Cycle Verification (T+12 hours)

1. Request the final published artifact URL. Check if the ETag matches between the English and Spanish structural envelopes.

2. Attempt to fetch the status endpoint. Verify if the state indicates a terminal completion or if it is stuck waiting for a retry that has passed.

Recommended Evidence-Log Format

(RECOMMENDED) Store this log in a structured format (JSON or CSV) to allow for chronological sorting and delta analysis.

• Observation timestamp in UTC: 2026-08-01T14:00:00Z
• Observer timezone: America/Chicago
• URL: https://internationalintelligence.org/brief
• HTTP method: GET
• Redirect chain: 301 \-\> 200
• Status code: 200
• Response headers: { "Cache-Control": "max-age=60", "Age": "12", "Vary": "Accept-Encoding" }
• Body hash: sha256:a1b2c3...
• Visible process ID: DB-20260801-A
• Visible revision: 4
• Visible target date: 2026-08-01
• Visible stage: Awaiting\_Provider
• Visible provider status: Polling
• Last meaningful progress time: 2026-08-01T13:55:00Z
• Next eligible cycle: 2026-08-01T14:05:00Z
• Publication date: null
• Browser and version: curl/7.81.0
• Device and viewport: Headless
• JavaScript enabled or disabled: Disabled
• Cache state: HIT (Age: 12\)
• Screenshot filename: N/A
• Raw-header filename: headers\_140000.txt
• Observer notes: Missing Vary: Accept-Language observed. Age header is climbing.
• Observed/Reported/Inferred: OBSERVED

12. A 72-Hour Confirmation Plan

To validate the hypotheses derived from the 24-hour runbook, a three-day confirmation sequence is required.

• Day 1 (Observation and Timeline Construction): Execute the 24-Hour Evidence-Collection Runbook. At the conclusion of the day, analyze the chronological data to identify the exact second invariants are violated. Look specifically for moments where the Revision number drops (e.g., from 5 to 4), indicating a fencing failure or cache rollback.
• Day 2 (Falsification Testing via Header Injection): Request the project owner to inject a benign, invisible diagnostic header into the HTTP response (e.g., X-SAPI-Env: FPM or X-Temp-Dir: /tmp). Resume observation. Observe if the API polling requests suddenly shift namespaces or environments during the publication cycle. This conclusively proves or refutes the Runtime Namespace Isolation hypothesis.
• Day 3 (Controlled Intervention and Domain Isolation): Request the owner to disable the CDN temporarily (or implement a strict cache bypass rule for a specific testing IP address). Run a manual trigger of the Daily Brief. If the system flawlessly succeeds without the CDN, the entire fault domain is proven to be Edge Cache Coherence16. If the system still fails identically, the caching layer is exonerated, and the fault domain isolates strictly to Runtime/Concurrency layer anomalies.

13. Prioritized Remediation Recommendations

(RECOMMENDED) Remediation actions are ranked by their structural Impact, our Confidence in the diagnosis, the Effort required to implement, the Dependency on specific infrastructure, and the inherent Rollback Risk.

Priority 1: Implement Monotonic Fencing Tokens (Concurrency Layer)

• Action: When the supervisor creates a lease, it must generate a strictly monotonically increasing integer (the fencing token). Every subsequent write, state transition, or file creation must include this token in the payload. The storage mechanism (Database or Filesystem) must actively reject any write where the provided token is less than the highest previously seen token4.
• Rationale: Time-based leases are fundamentally unsafe. Fencing guarantees that if a process goes to sleep, loses its lease, and awakens later, its attempt to write stale data will be hard-rejected, preserving the monotonic progression invariant4.

Priority 2: Unify the Temporary Namespace (Runtime Layer)

• Action: Immediately cease the use of sys\_get\_temp\_dir() or /tmp for storing application state, supervisor lock files, or SQLite databases. Configure the application to utilize a persistent, unified directory (e.g., /var/lib/intlint/state) configured with strict 0770 permissions, owned by a shared user group containing both the FPM web user and the CLI cron user11.
• Rationale: This eliminates the PrivateTmp split-brain condition, ensuring both runtimes converge on the exact same view of the supervisor state.

Priority 3: Fix Cache Coherence and Content Negotiation (Cache Layer)

• Action: Add the Vary: Accept-Language header to all localized endpoints to prevent CDN cache poisoning5. For active Supervisor status endpoints that report dynamic progress, strictly emit Cache-Control: no-store, private to prevent shared caches from trapping the state26. Remove stale-while-revalidate from state-transition endpoints, as it masks the exact state the frontend needs to make accurate polling decisions6.

Priority 4: Re-architect Retries with Explicit Idempotency

(RECOMMENDED) Generic advice to "add retries" exacerbates distributed system failures by causing retry storms28. Retries must be engineered with explicit idempotency constraints.

• Which ambiguity it addresses: Transient network loss or timeout occurring after an asynchronous provider response has been generated, but before the server successfully downloads it.
• How duplicate work is prevented: The system must send a deterministic Idempotency-Key header (derived mathematically from the Process ID and the current Revision) to the OpenAI API or provider29.
• How success is recognized: The provider recognizes the idempotency key and returns an HTTP 200 OK with the exact same payload as the initial successful generation, rather than burning credits to generate a new response.
• When retrying stops: Retries must unconditionally stop when a terminal state is reached. This includes receiving an HTTP 400 Bad Request (indicating a malformed prompt), failing structural evidence validation 3 consecutive times, or reaching a maximum exponential backoff threshold of 2 hours.
• What evidence is recorded: The supervisor must commit an evidence log to disk stating: Retry Attempt X, Backoff Ys, Reason: \[HTTP Code/Timeout\].
• How an operator distinguishes waiting from failure: The status endpoint must emit Status: Awaiting\_Retry alongside a precisely calculated Next\_Eligible\_Timestamp. If the current UTC time exceeds the Next\_Eligible\_Timestamp and the status remains Awaiting\_Retry, the operator definitively knows the scheduler is dead, rather than just waiting in a backoff queue.

Priority 5: Atomic Publication Writes

• Action: The system must not write bytes directly to the final public artifact path. It must write the payload to a temporary file (e.g., brief\_new.json) and utilize rename('brief\_new.json', 'brief.json'). In POSIX systems, rename() is an atomic operation. If operating over NFS, use link() and unlink() to simulate atomicity, as rename can cause locking conflicts15. Never use flock() over NFS14.

14. Production Acceptance Criteria

(RECOMMENDED) To mathematically prove the Daily Brief feature works in the production environment—rather than merely appearing to work under ideal conditions—the system must pass the following rigorous acceptance criteria under live traffic:

1. Concurrency Resilience: Triggering the process simultaneously via the background cron scheduler and three independent browser sessions must result in exactly one active supervisor process. This is verified by observing a single, unbroken monotonic Process ID sequence and zero duplicate API requests sent to the upstream provider.

2. Cache Convergence: Disabling JavaScript and navigating to the Daily Brief in English and Spanish across two different devices (or curl sessions) must perfectly reflect the exact same internal server revision number at any given second, proving Vary headers are functioning.

3. Fenced Degradation: Injecting an artificial 120-second sleep into the provider retrieval script (deliberately exceeding the browser lease time) must result in a safe lease timeout, an orderly recovery by the next cron cycle, and the strict rejection of the delayed 120-second payload via a fencing token validation failure.

4. Immutability: Once the final artifact is published and the supervisor enters a terminal state, subsequent synthetic network delays, old cache responses, or manual trigger attempts cannot revert the supervisor to an active state or overwrite the artifact.

Operator Action Checklist

• \[ \] Verify HTTP Status: Run curl \-I https://.../brief and check for the presence of Vary: Accept-Language.
• \[ \] Check Cache Staleness: Note the Age header on the status endpoint. Does it reset to 0 when expected, or continually climb?
• \[ \] Monitor Revision Drift: Open two browser tabs simultaneously. Do the UI Revision numbers match exactly at all times?
• \[ \] Validate Temp Directory: Execute php \-r "echo sys\_get\_temp\_dir();" as both www-data and the root/cron user. Are the paths identical?
• \[ \] Check Fencing: Ask engineering: "Does the database or state file strictly reject writes if the lock ID is older than the current lock ID?"
• \[ \] Review Retries: Ensure the status endpoint shows Next\_Eligible\_Timestamp for delayed retries, preventing operator panic and manual interference.

Unanswered Questions for Further Investigation

• Is the provider asynchronous callback mechanism web-hook based (push) or cron-polling based (pull)? Push mechanisms passing through a CDN often strip authorization headers or get aggressively cached by edge nodes.
• What is the specific locking storage mechanism currently in use? (e.g., Redis Redlock, MySQL GET\_LOCK, SQLite transactions, or Filesystem flock).
• Does the deployment pipeline clear OPcache via opcache\_reset() on the web-nodes post-deployment?30 If not, PHP code executing the state transitions may be physically out of sync with the database schema expectations, leading to silent failures.

Code snippet sequenceDiagram participant C as Cron (CLI) participant W as Web Browser (FPM) participant E as Edge CDN participant S as Supervisor / Storage participant P as Provider (OpenAI)

Note over C,W: Split-Brain / Fencing Failure Scenario C-\>\>S: Request Lease (Token 1\) S--\>\>C: Granted C-\>\>P: Dispatch Async Job Note over C: GC Pause / Network Delay (Lease Expires)

W-\>\>E: User clicks "Trigger" E-\>\>S: Request Lease S--\>\>E: Granted (Token 2\) E-\>\>P: Dispatch Async Job (Duplicate Work)

P--\>\>C: Job 1 Completes C-\>\>S: Write State (No Fencing Token checked\!) Note over S: State Corrupted. Older process overwrote newer process.

Works cited

1. Fine-Tune Your OPcache Configuration to Avoid Caching Surprises \- Tideways, https://tideways.com/profiler/blog/fine-tune-your-opcache-configuration-to-avoid-caching-suprises

2. Fedora 17 New Security Feature part II \- PrivateTmp \- Dan Walsh, https://danwalsh.livejournal.com/51459.html

3. Runtime Configuration \- Manual \- PHP, https://www.php.net/manual/en/opcache.configuration.php

4. How to do distributed locking \- Martin Kleppmann, https://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html

5. How does http caching work in combination with the accept-language header, https://stackoverflow.com/questions/70125825/how-does-http-caching-work-in-combination-with-the-accept-language-header

6. Origin Cache Control \- Cloudflare Docs, https://developers.cloudflare.com/cache/concepts/cache-control/

7. Updating addons via CLI doesn't invalidate opcode cache | XenForo community, https://xenforo.com/community/threads/updating-addons-via-cli-doesnt-invalidate-opcode-cache.208574/latest

8. Adjusting PHP OPCache for WordPress \- Managing WP, https://managingwp.io/2022/12/14/adjusting-php-opcache-for-wordpress/

9. PHP-FPM security setting "PrivateTmp=true" writes PHP session files to /tmp instead of session.save\_path \- cPanel Support, https://support.cpanel.net/hc/en-us/articles/1500001962582-PHP-FPM-security-setting-PrivateTmp-true-writes-PHP-session-files-to-tmp-instead-of-session-save-path

10. PHP: sys\_temp\_dir is not the same as upload\_tmp\_dir, it stays /tmp \- Claudio Kuenzler, https://www.claudiokuenzler.com/blog/680/php-sys\_temp\_dir-not-same-upload\_tmp\_dir-always-tmp

11. systemd \- private temp directories and permissions \- Server Fault, https://serverfault.com/questions/749069/systemd-private-temp-directories-and-permissions

12. Distributed Locks: Redlock, Fencing Tokens & Advisory \- AppScale Blog, https://appscale.blog/en/blog/distributed-locks-redlock-fencing-tokens-postgres-advisory-2026

13. 1.2.3 • stale-while-revalidate-reference • testland • Registry \- Tessl, https://tessl.io/registry/testland/stale-while-revalidate-reference/1.2.3

14. flock(2) \- Linux man page, https://linux.die.net/man/2/flock

15. Simple Atomic File Locking in Linux \- GitHub Gist, https://gist.github.com/pwillis-els/b01b22f1b967a228c31db3cf2789ee13

16. HTTP Caching Explained: Browser, CDN, and Server (2026) | Crosscheck, https://crosscheck.cloud/blogs/http-caching-explained-browser-cdn-server/

17. HTTP Caching explained, https://http.dev/caching

18. HTTP response status codes \- MDN Web Docs \- Mozilla, https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status

19. Age \- Expert Guide to HTTP headers, https://http.dev/age

20. RFC 9111 \- HTTP Caching \- IETF Datatracker, https://datatracker.ietf.org/doc/html/rfc9111

21. If-None-Match \- Expert Guide to HTTP headers, https://http.dev/if-none-match

22. RFC 9110: HTTP Semantics, https://www.rfc-editor.org/rfc/rfc9110.html

23. Cache-Status \- Expert Guide to HTTP headers, https://http.dev/cache-status

24. Cache-Control \- Expert Guide to HTTP headers, https://http.dev/cache-control

25. Locking NFS files in PHP \- Stack Overflow, https://stackoverflow.com/questions/218451/locking-nfs-files-in-php

26. HTTP caching \- MDN Web Docs \- Mozilla, https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/Caching

27. Cache-Control header \- HTTP \- MDN Web Docs, https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Cache-Control

28. Cascading Failures: Reducing System Outage \- Google SRE, https://sre.google/sre-book/addressing-cascading-failures/

29. Background mode | OpenAI API, https://developers.openai.com/api/docs/guides/background

30. Boosting PHP performance with OPCache \- HiBit, https://www.hibit.dev/posts/175/boosting-php-performance-with-opcache

31. PHP OPcache important settings and revalidation simplified \- Tech Semicolon, https://techsemicolon.github.io/blog/2019/10/05/php-fpm-and-opcache-settings-and-invalidation-simplified/