Incident Investigation Report: Black-Box Diagnostics of the Daily Brief Publication System

Full report

On this page

1. Executive Finding and Most Likely Failure Families

The persistent failure of the Daily Brief publication system on InternationalIntelligence.org represents a complex class of distributed systems anomalies manifesting within a monolithic or pseudo-monolithic deployment architecture \[INFERRED\]. The underlying platform utilizes asynchronous-provider reconciliation and a server-owned supervisor concept, which inherently subjects the application to the race conditions, clock skew, and state divergence typically associated with large-scale distributed microservices1. Because prior diagnostics have successfully verified lower-level provider connectivity, DNS resolution, and automated deterministic logic, the ongoing production failures are conclusively not localized to basic syntax errors or external API outages \[REPORTED BY PROJECT OWNER\]. Instead, the investigation isolates the root cause to environmental boundaries that fracture the unified state of the application \[INFERRED\]. The most highly probable failure families include configuration drift between the PHP Command Line Interface (CLI) and FastCGI Process Manager (FPM) runtimes, POSIX cross-device link (EXDEV) failures during atomic file operations, advisory file locking degradation, and HTTP edge-caching policies that induce state reversion3. Specifically, the implementation of stale-while-revalidate caching directives or 24-hour Service Worker byte-for-byte update caps likely trap the browser in a stale loop, violating the invariant that state cannot move backward7. Addressing these failures requires replacing legacy operational paradigms with strict state reconciliation loops, mathematically unique idempotency keys, and rigorous origin cache controls \[RECOMMENDED\].

2. Known Facts, Assumptions, Reported Facts, and Unknowns

To enforce diagnostic rigor and prevent confirmation bias, all variables concerning the InternationalIntelligence.org infrastructure must be strictly partitioned into authoritative facts, observed behaviors, and critical unknowns. The system relies on a PHP-based bilingual publication framework utilizing a multi-stage server-side workflow \[REPORTED BY PROJECT OWNER\]. The foundational network transport layers, including DNS resolution, TLS verification, PHP cURL, and OpenSSL, are fully operational \[REPORTED BY PROJECT OWNER\]. Furthermore, dependencies on external generative models, required web search capabilities, and the creation of background responses have been verified as functional \[REPORTED BY PROJECT OWNER\]. The architecture introduces a canonical host, a monotonic process-revision sequence, browser polling mechanisms, and a browser trigger lease integrated with a server-owned supervisor concept \[REPORTED BY PROJECT OWNER\]. Ten core invariants govern the system, primarily dictating that only one active process may exist per target date, revisions must be strictly monotonic, and localized views (English and Spanish) must converge on the exact same authoritative state \[REPORTED BY PROJECT OWNER\]. Conversely, the visible symptoms directly contradict these invariants. The Daily Brief remains stale, occasionally fails to publish entirely, displays outdated target dates, stalls indefinitely within intermediate stages, or presents inconsistent data across independent user sessions \[OBSERVED\]. Critical infrastructural specifics remain inaccessible under the black-box methodology. The exact separation of memory allocation and OPcache validation rules between the scheduled cron runtime and the web-facing runtime is unverified \[UNKNOWN\]. The precise mount topology of the host filesystem—specifically whether the temporary directory used for atomic staging shares a block device with the final publication document root—has not been exposed \[UNKNOWN\]. The origin server's injection of Cache-Control extensions, such as s-maxage or must-revalidate, remains obscured behind potential Content Delivery Network (CDN) layers \[UNKNOWN\]. Finally, the internal mechanisms for preventing concurrent lease acquisition, such as the use of POSIX flock() or database-backed row locks, are entirely hidden from public inspection \[UNKNOWN\].

3. Why Passing Deterministic Tests Can Coexist with a Broken Production Feature

The dissonance between perfectly passing automated tests and a fundamentally broken production feature is a hallmark of stateful environmental drift \[INFERRED\]. Deterministic tests are engineered to strip away the complexities of time, concurrency, and distributed state. They typically execute synchronously within a single PHP CLI thread, interacting with an isolated in-memory or localized SQLite database, and mocking network latency to zero \[INFERRED\]. In stark contrast, the production environment shatters this controlled isolation across multiple dimensions. First, FPM and CLI runtime isolation creates invisible state partitions. Modern PHP deployments frequently utilize FPM for serving web requests and CLI for executing scheduled cron jobs. To maximize throughput, production FPM configurations often disable opcache.validate\_timestamps5. This instructs the PHP engine to cache the compiled Zend opcodes indefinitely until a manual reload is triggered12. However, the CLI runtime spawns a fresh process for each cron execution, instantly reading the newest deployed code from disk14. If a deployment occurs and the FPM service is not gracefully restarted, the scheduled background tasks will write state transitions using the logic of Release [Figure omitted from source export], while the web application attempts to deserialize and render that state using the cached bytecode of Release [Figure omitted from source export] \[INFERRED\]. This structural mismatch allows deterministic tests to pass entirely within Release [Figure omitted from source export], while the live site fractures. Second, the assumption of filesystem atomicity frequently collapses in production. To satisfy the invariant that the system must publish one immutable edition or remain resumable, developers typically write the compiled document to a temporary file and execute a rename() system call to move it to the public directory \[REPORTED BY PROJECT OWNER\]. While POSIX guarantees that rename() is atomic, this guarantee strictly requires both the source and destination paths to reside on the identical mounted filesystem3. In production architectures, temporary directories (like /tmp) are often mounted as volatile RAM disks (tmpfs), while the document root (/var/www/html) resides on a persistent block storage volume (like an AWS EBS volume or an NFS mount) \[INFERRED\]. Attempting a rename() across these boundaries triggers an EXDEV (Invalid cross-device link) error4. If the application does not explicitly handle EXDEV, the publication simply aborts, or worse, silently falls back to a non-atomic copy-and-delete operation that violates concurrent read invariants \[INFERRED\]. Third, concurrency models tested via local mocks fail against true asynchronous races. If the server-owned supervisor limits active operations using advisory file locks via PHP's flock(), the protection is illusory if accessed over distributed filesystems or executed by mixed users18. Deterministic tests assert that locking logic returns the correct boolean, but they do not simulate a scenario where an authorized operator clicks a web trigger at the exact millisecond the cron scheduler awakens, causing both the web user www-data and the root cron user to attempt simultaneous LOCK\_EX acquisitions \[INFERRED\].

4. Symptom Taxonomy

To conduct a rigorous black-box investigation, the vaguely defined "broken" status must be atomized into a precise taxonomy of failures \[RECOMMENDED\]. Distinguishing these granular symptoms is paramount for reconstructing the underlying sequence of failures without accessing internal logs.

5. Black-Box Observation Plan

To rigorously identify the specific faults without utilizing privileged access, an exhaustive black-box observation matrix must be executed. This matrix probes the network perimeter to deduce internal architectural states. The observation protocol dictates querying the English and Spanish Daily Brief pages simultaneously, alongside the canonical hostname and any publicly exposed asynchronous status endpoints \[RECOMMENDED\]. Requests must be instantiated across fresh private browser sessions, utilizing multiple concurrent tabs to verify synchronization, and alternating between mobile and desktop user agents to detect device-specific caching \[RECOMMENDED\]. Testing must include iterations with JavaScript entirely disabled to isolate server-rendered artifacts from client-side DOM manipulations \[RECOMMENDED\]. Furthermore, requests must be systematically executed both normally and with lawful cache-bypassing headers (e.g., Pragma: no-cache and Cache-Control: no-cache) to determine whether the intermediate Content Delivery Network respects client-directed revalidation commands26. These repeated observations must span at least one complete intended publication cycle to capture temporal drifts \[RECOMMENDED\]. During every request, the observer must explicitly capture all origin HTTP response headers, focusing specifically on Age, Date, ETag, Last-Modified, Vary, Cache-Control, and request identifiers \[RECOMMENDED\]. Additionally, all publicly exposed metadata—including process identity, target date, stage progression, revision sequences, provider status, and last-progress times—must be documented directly alongside the cryptographic hash of the HTTP response body to ensure tracking fidelity \[RECOMMENDED\].

6. UTC Chronology Reconstruction Method

In a distributed environment obscured by caching proxies, the timestamps generated by the observer's local machine are useless for diagnosing synchronization faults \[INFERRED\]. A unified UTC chronology must be mathematically reconstructed relying exclusively on authoritative HTTP semantic headers defined in RFC 9110 and RFC 911127. The fundamental metric required is the Origin Generation Time ([Figure omitted from source export]), which represents the exact millisecond the backend PHP runtime compiled the state payload before it was swallowed by edge caches. To reconstruct this chronology:

1. Extract the Date Header: The Date header indicates the clock time of the origin server when the response was initially born. Convert this to a standard UTC UNIX timestamp ([Figure omitted from source export]) \[RECOMMENDED\].

2. Extract the Age Header: The Age header indicates the number of seconds the response has been held in a shared or private cache28. If missing, assume [Figure omitted from source export] (accounting for network latency) \[RECOMMENDED\].

3. Calculate [Figure omitted from source export]: Subtract the Age from the Date:

[Figure omitted from source export]

4. Extract Embedded Application Timestamps: Extract the "last-progress time" or "next eligible cycle" explicitly printed inside the JSON or HTML body by the PHP application \[RECOMMENDED\].

By comparing these variables, the investigator can reliably determine the precise relationship between any two observed pages \[RECOMMENDED\].

• The same current server snapshot: Two responses possess identical ETag values and identical calculated [Figure omitted from source export] timestamps.
• Two independently cached snapshots: The body hashes match, but the calculated [Figure omitted from source export] values differ slightly, indicating that two separate CDN edge nodes fetched the exact same process state at slightly different times \[INFERRED\].
• Two genuinely different backend processes: The embedded Process ID differs entirely between requests, or the Monotonic Revision has increased, accompanied by a sequentially newer [Figure omitted from source export].
• The same process with different localized labels: The English and Spanish variants present distinct ETags and body hashes, but share the identical Process ID, Revision, and "last-progress time" within the payload payload.
• Different releases or site instances: The Server or X-Powered-By headers fluctuate across requests, and the calculated [Figure omitted from source export] behaves non-linearly (jumping backwards), proving that a load balancer is routing traffic across mismatched deployment configurations \[INFERRED\].

7. HTTP, Cache, and Canonical-Origin Test Matrix

This testing matrix aims to exploit standard RFC 9111 caching mechanisms to reveal hidden proxy layers and Service Worker traps30.

8. Scheduler and Asynchronous-Job Test Matrix

This matrix observes temporal boundaries to deduce the reliability of the cron execution and external provider reconciliation loops.

9. Fault Tree with Ranked Hypotheses

The following fault tree logically stratifies the most probable failure architectures, ranking them based on invariants and observed symptom profiles \[RECOMMENDED\].

Code snippet graph TD A\[Daily Brief Fails to Advance\] \--\> B(Frontend / Cache Domain) A \--\> C(Backend / State Domain)

B \--\> B1\[Stale-While-Revalidate Trap\] B \--\> B2\[Service Worker Byte-for-Byte Freeze\]

C \--\> C1\[CLI vs FPM OPcache Drift\] C \--\> C2\[Advisory File Locking Failure\] C \--\> C3\[EXDEV Atomic Rename Failure\]

C1 \--\> C1a(Cron updates state schema, FPM crashes reading it) C3 \--\> C3a(Document written to tmpfs, fails to mount on /var/www)

Hypothesis 1: Origin Cache Stale-While-Revalidate Trap

• Responsible Layer: Origin Cache / CDN.
• Observable Symptoms: The UI progress halts, jumps forward, and occasionally jumps backward across page reloads. The process appears entirely corrupted to the public.
• Confidence Evidence: Age headers continue to climb while the Cache-Control header exposes stale-while-revalidate=X8.
• Falsification Evidence: All requests contain Cache-Control: no-cache or must-revalidate, explicitly forbidding stale serving29.
• Severity: High. (Destroys user trust and violates monotonic invariant).
• Probability: High. (Very common CDN misconfiguration for dynamic APIs).
• Safest Next Diagnostic Action: Inspect raw HTTP response headers for cache extensions.
• Minimum Telemetry Required: Network HAR files from the client browser.
• Self-Recovering: Yes, once the asynchronous revalidation completes and the SWR window expires, the state snaps back to reality.
• Operator Intervention / Duplicate Risk: Safe. Forcing a CDN cache purge resolves the visual anomaly without creating duplicate backend jobs.

Hypothesis 2: PHP CLI vs. Web Runtime OPcache Drift

• Responsible Layer: PHP CLI Runtime & Web PHP Runtime.
• Observable Symptoms: The scheduler successfully generates the brief in the background, but the web UI remains broken, displaying a 500 Server Error or a completely stalled state payload.
• Confidence Evidence: A recent deployment occurred, but the FPM service was not restarted. Cron runs a new codebase; web runs an old codebase5.
• Falsification Evidence: A synchronized reboot of the entire virtual machine does not resolve the divergence.
• Severity: Critical. (Completely fractures the application state views).
• Probability: Medium. (Common in modern rapid-deployment PHP architectures).
• Safest Next Diagnostic Action: Trigger a manual PHP-FPM reload via the system operator.
• Minimum Telemetry Required: Output of php \-v and OPcache status arrays from both FPM and CLI environments5.
• Self-Recovering: No. The memory pools will remain divergent indefinitely if opcache.validate\_timestamps=011.
• Operator Intervention / Duplicate Risk: Safe. Restarting the FPM process drops the opcode cache but preserves the persistent database state.

Hypothesis 3: EXDEV Cross-Device Link Failure on Publish

• Responsible Layer: Filesystem / Publication Write.
• Observable Symptoms: The process advances successfully through composition. The supervisor reports the task as "Complete", but the physical artifact returns a 404 Not Found.
• Confidence Evidence: The temporary writing directory is mounted on a separate partition (e.g., /tmp on tmpfs) from the document root. The POSIX rename() function silently fails or throws an unhandled exception3.
• Falsification Evidence: Test scripts confirm that rename() succeeds natively between the temporary path and the public path.
• Severity: Critical. (Fails the primary directive of publishing the edition).
• Probability: Medium. (Highly dependent on specific Linux mounting topologies).
• Safest Next Diagnostic Action: Request filesystem layout metrics from the systems team.
• Minimum Telemetry Required: Output of df \-h and PHP error logs searching for Invalid cross-device link.
• Self-Recovering: No.
• Operator Intervention / Duplicate Risk: Safe. Redefining the sys\_temp\_dir to share the volume of the document root prevents the anomaly without duplicating provider tasks38.

Hypothesis 4: Concurrent Lease Failure via Advisory Locking

• Responsible Layer: Supervisor state.
• Observable Symptoms: Provider work is dispatched, but its identifier is immediately lost. The system becomes deadlocked waiting for a callback linked to an orphaned identifier.
• Confidence Evidence: The application relies on flock($fp, LOCK\_EX | LOCK\_NB) to ensure single-process execution, but fails to handle race conditions where advisory locks are ignored across network mounts or disparate execution users6.
• Falsification Evidence: The architecture strictly utilizes ACID-compliant database row-level locking (e.g., SELECT ... FOR UPDATE) to manage state transitions.
• Severity: High. (Orphans jobs and burns OpenAI credits uselessly).
• Probability: Low to Medium.
• Safest Next Diagnostic Action: Rapidly poll the process state exactly at the initiation minute to detect fluttering Process IDs.
• Minimum Telemetry Required: Source code snippet detailing the exact mutual exclusion acquisition logic.
• Self-Recovering: Unlikely. Usually requires manual database truncation of the deadlocked state.
• Operator Intervention / Duplicate Risk: Dangerous. Manually killing active PHP processes may corrupt intermediate state files if not done safely.

10. Minimum Additional Telemetry Needed from the Owner

To definitively confirm these hypotheses without compromising the production state, the investigator requires the following sanitized artifacts from the project owner \[RECOMMENDED\]:

11. A 24-Hour Incident Evidence-Collection Runbook

Objective: Safely orchestrate the collection of irrefutable black-box evidence across one entire intended publication lifecycle.

Phase 1: Baseline Establishment (T-Minus 2 Hours)

The investigator clears all local DNS caches and browser storage. Utilizing curl and browser DevTools, the investigator probes the canonical hostname to establish the baseline ETag and Age header behaviors \[RECOMMENDED\]. A test is executed with JavaScript disabled to confirm whether the loading state is embedded in the initial HTML payload or retrieved via subsequent asynchronous fetch operations.

Phase 2: Active Monitoring (T-Minus 15 Minutes to T-Plus 15 Minutes)

The investigator initiates an automated script to poll the status endpoint every 10 seconds. The script meticulously parses the response, populating the chronology template below. If the monotonic revision jumps backward, the investigator immediately logs the specific Age header of that exact payload to prove the existence of an edge-cache trap \[RECOMMENDED\].

Phase 3: Post-Completion Artifact Validation (T-Plus 30 Minutes)

The investigator attempts to access the final English and Spanish artifacts via direct URL manipulation. If the API claims completion but the files return a 404, the investigator documents the EXDEV atomicity failure \[RECOMMENDED\].

Recommended Evidence-Log Chronology Template

12. A 72-Hour Confirmation Plan

Following the 24-hour baseline collection, the diagnostic sequence transitions into a targeted isolation protocol spread across three distinct daily publication cycles. Day 1: Isolation of Caching Proxies To confirm or deny Hypothesis 1 (SWR Cache Trap), the operator modifies their local workstation's hosts file to explicitly map www.internationalintelligence.org to the backend origin IP address, bypassing Cloudflare/Fastly entirely. If the UI flawlessly increments monotonically via the direct IP, but stutters and jumps backward for the general public, the CDN stale-while-revalidate misconfiguration is conclusively proven \[RECOMMENDED\]. Day 2: Isolation of OPcache Deployment Drift To confirm or deny Hypothesis 2 (CLI vs FPM Memory Partitions), the system administrator is instructed to perform a graceful restart of the PHP-FPM service (systemctl reload php-fpm) exactly 10 minutes prior to the target initiation time. If the FPM restart magically cures the UI serialization crashes and allows the brief to publish smoothly, the deployment configuration drift is conclusively proven \[RECOMMENDED\]. Day 3: Isolation of Filesystem Atomicity Failures To confirm or deny Hypothesis 3 (EXDEV Rename Errors), the project owner must temporarily override the PHP sys\_temp\_dir environment variable. Instead of writing temporary files to /tmp, the application is configured to write them to /var/www/html/tmp/. If this single configuration change allows the final stage to successfully write the public artifact, the cross-mount atomicity failure is conclusively proven \[RECOMMENDED\].

13. Prioritized Remediation Recommendations

The following structural remediations are ranked meticulously by their impact on systemic reliability, diagnostic confidence, engineering effort, system dependencies, and potential rollback risk \[RECOMMENDED\].

Mandatory Requirements for Retry Mechanisms

Generic "add retries" logic will catastrophically exacerbate distributed systems failures. Any retry implementation surrounding the external provider orchestration must adhere strictly to the IETF HTTP Idempotency-Key standard36. \[RECOMMENDED\]

• Ambiguity Addressed: Safely identifies whether a 504 Gateway Timeout occurred before the external provider began processing the job, or after the job completed but before the response reached the server.
• Preventing Duplicate Work: The PHP application must generate a unique UUID and a cryptographic fingerprint of the payload36. This key is transmitted via the Idempotency-Key header. The external provider caches the response against this key.
• Recognizing Success: If a retry request is issued with the identical key, the provider intercepts the request and instantly returns the cached HTTP 200 OK containing the original task identifier, entirely skipping re-execution37.
• When Retrying Stops: Retrying MUST immediately cease if the provider returns a 422 Unprocessable Content (indicating a fingerprint mismatch), a 400 Bad Request, or if 3 sequential 500-level errors occur.
• Evidence Recorded: The exact UUID, the SHA-256 payload hash, the UTC dispatch timestamp, and the final HTTP status code must be written to the local supervisor database prior to transmitting the curl request.
• Distinguishing Waiting from Failure: The application must parse a 409 Conflict as a mandate to wait (the job is currently processing), whereas a 422 or 401 indicates absolute failure requiring the stage to quarantine37.

14. Production Acceptance Criteria

To empirically verify that the underlying engineering faults have been resolved, the application must demonstrably satisfy the following strict criteria during live production execution. A successful test run in a local staging environment is explicitly insufficient \[RECOMMENDED\].

Proof of Functional Integrity

1. Strict Monotonicity Proof: Automated polling of the status API at 10Hz during the cron execution window yields a sequence of Revision IDs that increase strictly monotonically. The sequence never stalls and never decreases.

2. Cache Integrity Proof: Polling the status API via the public CDN never yields an HTTP response where the calculated [Figure omitted from source export] is mathematically older than a previously received [Figure omitted from source export].

3. Atomic Publication Proof: Executing high-frequency GET requests against the final artifact URL during the transition phase results in exactly one state change from 404 Not Found to 200 OK. Zero 500 Internal Server Error or partial-read responses occur.

4. Bilingual Convergence Proof: The ETag and metadata payloads for both the English and Spanish documents reliably resolve to the exact same underlying Monotonic Revision and Target Date upon process termination.

5. Idempotent Dispatch Proof: Simulating a network drop during provider dispatch (via artificial iptables dropping) forces the supervisor to retry. The retry loop successfully retrieves the original provider job ID via Idempotency-Key reconciliation, guaranteeing zero duplicate tasks are spawned externally.

One-Page Operator Diagnostics Checklist

Request-Sequence Diagram: Idempotent Provider Orchestration

Code snippet sequenceDiagram participant Cron as PHP CLI Supervisor participant DB as Local Database participant API as External Provider

Cron-\>\>DB: Generate UUID & Payload Hash DB--\>\>Cron: Lock Acquired, Idempotency-Key Saved Cron-\>\>API: POST /task (Idempotency-Key: UUID) Note over API: Job Starts API--xCron: Connection Timeout (504) Note over Cron: Wait 30s (Exponential Backoff) Cron-\>\>API: POST /task (Idempotency-Key: UUID) Note over API: Key Recognized\! API--\>\>Cron: 200 OK (Cached Original Response & Job ID) Cron-\>\>DB: Save Job ID, Update Stage

Works cited

1. DriftlessAF: Introducing Chainguard Factory 2.0, https://www.chainguard.dev/unchained/driftlessaf-introducing-chainguard-factory-2-0

2. GitOps Prescription: Curing the Configuration Drift Epidemic \- BridgePhase, https://bridgephase.com/insights/drift-detection

3. January 2018 \- Giedrius Statkevičius, https://giedrius.blog/2018/01/

4. Is mv atomic operation between two file systems? \- Unix & Linux Stack Exchange, https://unix.stackexchange.com/questions/452620/is-mv-atomic-operation-between-two-file-systems

5. OPcache Configuration for Laravel: The Free Performance Boost You're Ignoring, https://deploynix.io/blog/opcache-configuration-for-laravel-the-free-performance-boost-youre-ignoring

6. Does flock lock the file across processes? \- Stack Overflow, https://stackoverflow.com/questions/55556706/does-flock-lock-the-file-across-processes

7. Preventing server-forced updates · Issue \#822 · w3c/ServiceWorker \- GitHub, https://github.com/w3c/ServiceWorker/issues/822

8. Does stale-while-revalidate work? \- Application Performance \- Cloudflare Community, https://community.cloudflare.com/t/does-stale-while-revalidate-work/24161

9. HTTP Caching: Cache Assets Forever, Keep HTML Fresh — PandaStack Blog, https://pandastack.io/blog/http-caching-headers-guide

10. Consider relying on eTags (or other headers) for service worker dependencies to check for updates · Issue \#839 · w3c/ServiceWorker \- GitHub, https://github.com/w3c/ServiceWorker/issues/839

11. Fine-Tune Your OPcache Configuration to Avoid Caching Surprises \- Tideways, https://tideways.com/profiler/blog/fine-tune-your-opcache-configuration-to-avoid-caching-suprises

12. Runtime Configuration \- Manual \- PHP, https://www.php.net/manual/en/opcache.configuration.php

13. Boosting PHP performance: mastering OPcache optimization with Blackfire, https://blog.blackfire.io/boosting-php-performance-mastering-opcache-optimization-with-blackfire.html

14. What is the difference between php-cli and php-fpm, why 2 php variants and why choose one or the other? \- DevOps Stack Exchange, https://devops.stackexchange.com/questions/9505/what-is-the-difference-between-php-cli-and-php-fpm-why-2-php-variants-and-why-c

15. PHP CLI vs FPM \- Medium, https://medium.com/@mamreezaa/php-cli-vs-fpm-562395ecfba3

16. Atomically swapping two directories | The FreeBSD Forums, https://forums.freebsd.org/threads/atomically-swapping-two-directories.74948/

17. Quota management | W E K A, https://docs.weka.io/weka-filesystems-and-object-stores/quota-management

18. PHP flock \- File Locking in PHP \- ZetCode, https://zetcode.com/php-filesystem/flock/

19. Locking files with flock() \- Hacking with PHP, http://www.hackingwithphp.com/8/11/0/locking-files-with-flock

20. REST Guidelines \- Belgif, https://www.belgif.be/specification/rest/api-guide/

21. content/files/en-us/web/api/service\_worker\_api/index.md at main \- GitHub, https://github.com/mdn/content/blob/main/files/en-us/web/api/service\_worker\_api/index.md?plain=1

22. Understanding Distributed Systems 2nd Edition 1838430210 | PDF | Transmission Control Protocol | Computer Network \- Scribd, https://www.scribd.com/document/572213004/Understanding-distributed-systems-2nd-edition-1838430210

23. Database Reliability Engineering \- The Swiss Bay, https://theswissbay.ch/pdf/Books/Computer%20science/databasereliabilityengineering.pdf

24. Distributed Systems Practitioners Dimos Raptis Raspoznan | PDF \- Scribd, https://www.scribd.com/document/683054179/Distributed-Systems-Practitioners-Dimos-Raptis-Raspoznan

25. PHP CLI vs PHP-FPM version mismatches \- UniSyn, https://unisyn.tech/resources/php-cli-vs-fpm-version-mismatch/

26. Cache-Control \- Expert Guide to HTTP headers, https://http.dev/cache-control

27. RFC 9110: HTTP Semantics, https://www.rfc-editor.org/rfc/rfc9110.html

28. RFC 9111 \- HTTP Caching \- IETF Datatracker, https://datatracker.ietf.org/doc/html/rfc9111

29. Cache-Control header \- HTTP \- MDN Web Docs, https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Cache-Control

30. Caching Best Practices in REST API Design \- Speakeasy, https://www.speakeasy.com/api-design/caching/

31. Origin Cache Control \- Cloudflare Docs, https://developers.cloudflare.com/cache/concepts/cache-control/

32. If-None-Match \- Expert Guide to HTTP headers, https://http.dev/if-none-match

33. ETag \- Expert Guide to HTTP headers, https://http.dev/etag

34. HTTP ETag Header: What It Is, How It Works and How to Implement It \- Traficxo, https://www.traficxo.com/blog/etag-what-it-is-how-it-works-and-how-to-use-it-correctly

35. Progressive Web Apps with Angular: Create Responsive, Fast and Reliable PWAs Using Angular \[1st ed.\] 978-1-4842-4447-0 \- dokumen.pub, https://dokumen.pub/progressive-web-apps-with-angular-create-responsive-fast-and-reliable-pwas-using-angular-1st-ed-978-1-4842-4447-0978-1-4842-4448-7.html

36. draft-ietf-httpapi-idempotency-key-header-07, https://datatracker.ietf.org/doc/html/draft-ietf-httpapi-idempotency-key-header

37. Idempotency-Key header \- HTTP \- MDN Web Docs, https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Idempotency-Key

38. Fail to save configuration.php when the temporary system directory (/tmp) is protected · Issue \#48010 · joomla/joomla-cms \- GitHub, https://github.com/joomla/joomla-cms/issues/48010

39. designing data intensive applications \- GitHub Pages, https://0-lucas.github.io/digital-garden/99.-Books/Martin-Kleppmann---Designing-Data-Intensive-Applications\_-O%E2%80%99Reilly-Media-(2017).pdf

40. draft-idempotency-header-01 \- IETF Datatracker, https://datatracker.ietf.org/doc/html/draft-idempotency-header-01