Black-box diagnosis of the optional OpenAI consequence-narrative service on IARPA.org Planetary Atlas

Full report

On this page

Executive summary

Verified evidence. The public IARPA.org Planetary Atlas describes the OpenAI component as an optional, bounded narrative layer for fictional events only. The site’s public copy states that the deterministic browser-local simulation is authoritative, that OpenAI may supply narrative only, that the model cannot change numeric state, that exact coordinates and private notes are omitted from the OpenAI request, that the API key is never exposed in the browser, and that a deterministic local fallback exists. The public UI also exposes controls consistent with that design, including Generate consequence brief and Use deterministic brief, and a status text that says “AI consequences on” and “Checking bounded consequence service…”.

Not verified in this session. I did not directly observe a live successful GET to /api/simulation-consequences.php, did not capture the endpoint’s actual response JSON or headers, and did not directly observe a real POST transaction from the browser after creating a fictional event. Because of that, I cannot truthfully assert the current live values of enabled, configured, available, reason, release, or generation, and I cannot claim that the OpenAI-backed service is currently healthy. I also did not observe any successful provider response.

Most important conclusion. Based on public evidence, the architecture and trust boundary appear intentionally designed to keep OpenAI subordinate to deterministic simulation, which matches the contract you supplied. However, the runtime capability status is currently unknown from my vantage point. The strongest evidence today is about design intent and publicly declared safety boundaries, not about live endpoint health.

Most plausible failure classes if users are seeing fallback or narrative unavailability. Without a live endpoint trace, the most plausible classes are: server feature disabled, key not visible to the actual PHP runtime, key-file path rejected by local safety checks, provider authentication/quota failure, or a Responses API request / Structured Outputs mismatch on the server side. The PHP-FPM environment-handling issue deserves particular attention because PHP-FPM clears worker environments by default unless variables are explicitly passed through pool configuration. OpenAI’s current documentation also matters: for the Responses API, Structured Outputs have moved from response_format to text.format, and Responses are stored by default unless store: false is set.

Test scope and environment

This investigation was performed on July 27, 2026 in the user’s preferred locale en-US, with the user timezone context America/Chicago. The test used publicly reachable web surfaces only. No source code, repository history, secrets, shell access on the origin, hosting control panel, deployment pipeline, ZIPs, or previous conversations were used.

The practical environment constraints matter:

The homepage itself explicitly says the fully interactive globe needs JavaScript, while the non-JavaScript publication remains available. That distinction explains why public copy was inspectable, but live runtime XHR behavior was not fully observable here.

Verified public evidence

Site contract and public UI

The public homepage and related pages are unusually explicit about the boundary between deterministic simulation and optional OpenAI narrative. The strongest publicly verifiable claims are these:

The site states that consequence briefs are fictional narrative only, that OpenAI may supply bounded narrative only, that the deterministic engine remains authoritative, and that a brief cannot change numeric state, target a real place, rank a group, or become factual research. It also says historical research events are never rewritten by the model.

The fictional-event UI says the deterministic browser-local simulation creates the outcome first and that OpenAI may then explain consequences without changing numeric state. The same section states that the clicked point becomes an invented synthetic cell, that exact coordinates are neither displayed nor sent to the consequence service, and that the private browser-local note is optional. In the AI consequences help text, the site further states that the request contains only a synthetic cell identifier and reviewed abstract choices, and omits exact coordinates and private notes.

The site explicitly says that the model never receives the API key in the browser and that there is a deterministic local fallback. That public statement aligns with OpenAI’s own official guidance that API keys are secrets, must not be exposed in client-side code such as browsers or apps, and should be loaded server-side from an environment variable or key-management system.

Separately, the privacy page says the site can be read without an account and does not include advertising, analytics, remote fonts, or embedded third-party forms. The about page also describes the project as an independent educational planetary simulation, read-only for current statistics and fictional for simulation earth, with no account or analytics required. Those privacy statements are relevant because they make it more plausible that a “private browser-local note” is intended to remain local rather than being transmitted outward.

Endpoint evidence

The user asked for exact GET evidence for /api/simulation-consequences.php. Here is the most rigorous version of what I can and cannot claim.

The public UI wording strongly suggests a capability check exists, but that is still only inference about application behavior, not a captured network transaction. The difference matters.

Browser network evidence

Because I could not run a full DevTools network session against the unpublished endpoint from this environment, the browser-network portion of the investigation has to be split into verified UI evidence and unverified transport evidence.

Current capability status

The current capability status is unknown.

I can verify that the site advertises an optional bounded consequence service and a capability-check UI. I cannot verify that the current live endpoint is returning the healthy state you described: enabled=true, configured=true, available=true, reason=ready. I also cannot exclude the opposite. Therefore the only defensible status line is:

Current capability status: unknown from direct endpoint evidence; feature is publicly present, but live health was not observed.

That is materially different from saying the service is working.

OpenAI API behavior relevant to this service

OpenAI’s official documentation says the Responses API is the recommended API surface for new projects, and that it is the primary interface for direct model requests. It also says authentication is via a bearer credential and explicitly warns that API keys must not be exposed in browser or app client code.

For Structured Outputs, OpenAI’s current documentation says that Structured Outputs ensure adherence to a supplied JSON Schema, and that in the Responses API the schema definition belongs under text.format, not the older Chat Completions-style response_format. OpenAI also says Responses are stored by default unless store: false is set. Those details are especially important for this service because a request-body mismatch or an unintentional storage default would both be operationally meaningful under your contract.

Failure-path analysis

Verified stages

Verified stage: deterministic simulation happens locally first. The public event-creation copy states that the deterministic browser-local simulation creates the outcome first, and only afterwards may OpenAI explain consequences. This means the application is intentionally designed so that event existence and simulated state do not depend on OpenAI.

Verified stage: OpenAI is contractually limited to narrative. Public copy states that OpenAI may supply only bounded narrative, cannot alter numeric state, cannot target a real place, and receives neither exact coordinates nor private notes. That strongly supports the contract that the deterministic simulation is authoritative.

Verified stage: local fallback is part of the design. The public UI includes a deterministic-brief option, and the help text says a deterministic local fallback exists. That means a failure of the optional provider path should not be allowed to break the main simulation flow.

Unverified stages

Everything below remains unverified in this session because I did not directly capture the capability endpoint or a POST trace:

• Whether the browser actually made the capability GET in this session.
• Whether the capability GET returned healthy JSON.
• Whether a POST was attempted after a fictional event.
• Whether failure, if any, occurred before POST, during same-origin processing, in PHP configuration, during outbound DNS, during outbound TLS, during provider authentication, during provider request validation, or during Structured Outputs validation.
• Whether the browser immediately fell back after a failed POST.
• Whether the endpoint emitted release/generation headers or Cache-Control: no-store.
• Whether the server used store: false on the OpenAI Responses request.

Root-cause hypotheses with confidence levels

The hypotheses below are deliberately separated from verified evidence.

Hypothesis with medium confidence: if users are seeing deterministic fallback instead of an OpenAI brief, the problem is more likely server-side than browser-side. The browser-side/public contract already says the API key stays on the server, the provider gets only abstracted inputs, and fallback is deterministic. Nothing public suggests that the browser is intended to own provider authentication.

Hypothesis with medium confidence: one especially plausible failure class is “key exists somewhere on the host, but not in the actual PHP runtime.” This is common on PHP-FPM because clear_env defaults to Yes, which clears the worker environment before adding explicitly allowed variables from the pool configuration. Apache also distinguishes between its own internal per-request environment and true OS environment. So an operator can easily prove a variable exists in an interactive shell yet still fail to expose it to the PHP worker serving requests.

Hypothesis with low-to-medium confidence: if the service now uses OpenAI’s Responses API but the request body still follows older Chat Completions structure for Structured Outputs, the outbound call could fail validation. OpenAI’s current migration guide says that in Responses, Structured Outputs moved from response_format to text.format. That is exactly the kind of integration mismatch that would produce provider-side request errors or local schema-handling failures.

Hypothesis with low confidence: if the request reaches OpenAI but fails with 401 or 429, likely causes are invalid authentication, organization/project mismatch, rate limit, or quota/spend exhaustion. OpenAI’s official error guide explicitly documents those classes.

Hypothesis with low confidence: if the app emits a local reason such as provider_transport_failed, provider_http_error, structured_output_missing, or structured_output_rejected, then the problem is already past same-origin PHP entry and inside either the outbound transport, provider response, or local schema-validation stage. That is a useful conceptual distinction, but I did not observe the live reason code.

Hypothesis with very low confidence: outbound DNS or TLS from the server to OpenAI is broken. I can verify only that inbound DNS/TLS to iarpa.org worked for public pages during this session; I cannot verify outbound DNS/TLS from the application host to api.openai.com.

Diagnostic matrices and operator checklists

Failure-code matrix

The matrix below combines your expected local reason taxonomy with official OpenAI semantics and PHP/Apache runtime behavior. Where a row refers to OpenAI authentication, rate limits, request shape, or schema behavior, it is grounded in current official OpenAI docs. Where a row refers to PHP runtime visibility, it is grounded in PHP and Apache docs.

Hosting-specific configuration guidance

The highest-risk operator misunderstanding is confusing shell environment visibility with web-request PHP runtime visibility.

On Apache with mod_php, Apache can set variables using SetEnv, and can pass process environment variables with PassEnv, but Apache’s docs also note that its “environment variables” are often internal httpd structures and become true OS environment variables mainly when provided to CGI/SSI. That means an operator should validate inside the actual PHP request path, not by echoing environment variables from a shell.

On PHP-FPM, the official PHP manual says workers clear the environment by default with clear_env = yes. FPM pools can also run with different environments. In practice, this means setting OPENAI_API_KEY in a login shell or even in the web-server service environment may still leave it invisible to the PHP process unless the pool configuration explicitly passes it through.

On CGI/FastCGI, PHP’s getenv() documentation warns that SAPI-provided environment handling can differ, and Apache’s docs explain that variables passed to CGI scripts are a distinct part of request processing. That again argues for a same-SAPI diagnostic page rather than a CLI sanity check.

On shared hosting, environment-variable controls are often limited or mediated through hosting panels. In that setting, the key-file pattern you described is often safer operationally than assuming shell-style environment injection will reach the request worker. This point is partly inference, but it follows directly from the way FPM pools and CGI handoffs work.

On containerized hosting, environment variables are common, but FPM still needs them visible to the worker pool. A bind-mounted private key file outside the document root is a legitimate alternative when the runtime’s environment propagation is uncertain. FPM’s own documentation highlights that pools can be started with different environments.

Would a restart be required after changing environment variables? Usually, yes, for the process that actually serves PHP requests. That is an inference from the official architecture: Apache only passes what its running process has, and FPM workers start with pool-defined environment. Changing a shell variable after the service is already running does not retroactively update long-lived worker processes. In practice this usually means:

• mod_php / Apache-owned env: reload or restart Apache.
• PHP-FPM env or pool config: reload or restart PHP-FPM, and sometimes also reload the fronting web server if its FastCGI mapping changed.
• Container env changes: recreate or restart the container.

Outbound network and TLS checklist

This checklist is operational guidance rather than a claim that these failures occurred.

To rule out outbound DNS / transport / TLS failures from the application host, the operator should verify from the same PHP runtime or the same container/network namespace that serves the app:

• DNS resolution for api.openai.com.
• Outbound TCP to port 443.
• Successful HTTPS handshake with certificate validation enabled.
• A valid CA trust store available to cURL/OpenSSL.
• Correct system clock, to avoid certificate-validation anomalies.
• No corporate proxy, WAF, host firewall, shared-hosting egress rule, or container policy blocking the request.
• Sensible timeouts and retry policy.
• Private logging of transport error numbers, not just a generic “provider_transport_failed”.

OpenAI’s own docs recommend logging request IDs for production troubleshooting, and expose x-request-id along with processing and rate-limit headers. That means once outbound requests are actually working, the application should capture those identifiers server-side.

Provider authentication and quota checklist

OpenAI’s official API docs state that application requests use bearer authentication, and their error guide documents distinct classes for invalid authentication, incorrect API key, rate-limit reached, and quota exceeded. For this service, the operator checklist should therefore include:

• Confirm that the server sends Authorization: Bearer ... from a server-side secret, never from browser JavaScript.
• Confirm the right organization/project context if the deployment relies on project-scoped access. OpenAI documents optional OpenAI-Organization and OpenAI-Project headers for multi-org / legacy-key cases.
• If a provider call fails, capture HTTP status and x-request-id privately on the server.
• Treat 401 as an auth/configuration problem until proven otherwise.
• Treat 429 as either rate limiting or quota/spend exhaustion, using OpenAI’s distinctions.
• Treat 5xx / overload as transient provider availability issues and immediately use deterministic fallback.

Request-body and structured-output checklist

OpenAI’s current official documentation implies the following checklist for this service:

The request should use the Responses API, which OpenAI currently recommends for new projects. The minimal shape includes a model and input.

Authentication must be bearer auth, with the key held server-side only.

If the service expects schema-constrained narrative output, then for Responses the schema should be placed under text.format, not the older response_format form that applies to Chat Completions. OpenAI’s migration guide is explicit about that movement.

If schema adherence is required, OpenAI says Structured Outputs are the feature that ensures the model adheres to the supplied JSON Schema, and it recommends Structured Outputs over plain JSON mode when possible.

If the system contract forbids storing provider-side results, the server should explicitly send store: false, because OpenAI says Responses are stored by default unless disabled. That point is easy to overlook and is materially relevant to your stores_provider_response / privacy expectations.

If a model is configured that is not actually available to the project or not appropriate for the requested feature set, the failure should be treated as a provider request validation / model availability problem and surfaced as deterministic fallback, not as a simulation failure. OpenAI’s current model docs say the latest models are available via Responses API, and their docs emphasize checking model availability and compatibility.

Privacy, fallback, and user messaging

Privacy and secret-handling audit

Verified against public site text: the site publicly states that the request to OpenAI omits exact coordinates and private notes, uses only a synthetic cell identifier plus reviewed abstract choices, never exposes the API key in the browser, and keeps deterministic simulation authoritative.

Verified against OpenAI docs: API keys must not be exposed in client-side code such as browsers or apps, and should be loaded on the server from environment variables or similar secret management.

Privacy implication from OpenAI docs: if this service uses the Responses API and the operator wants provider-side retention minimized, the request should explicitly set store: false, because Responses are stored by default. That is not a conjecture; it is a current official OpenAI behavior.

Audit conclusion. The publicly declared privacy boundary is good, but one runtime question remains especially important and currently unverified: whether the server actually sets store: false on the outbound Responses API request.

Recommended nonsecret health diagnostics

A production-safe capability endpoint should expose state, not secrets. The endpoint you described is directionally correct. I recommend the capability GET emit only these nonsecret items:

• enabled
• configured
• available
• reason
• mode
• release
• generation
• state_authority
• stores_provider_response
• real_world_targeting
• optional booleans such as has_env_flag, has_api_key_env, has_key_file_path, key_file_resolved, key_file_readable, key_file_outside_docroot
• optional timing such as capability_checked_ms
• optional provider_request_id_present boolean on the POST path, rather than a raw provider ID in public mode

I also recommend the capability endpoint send:

• Content-Type: application/json
• Cache-Control: no-store
• a public release header
• a public generation header
• preferably X-Robots-Tag: noindex, noarchive

I did not verify those headers on the live endpoint in this session. This is a recommendation based on the endpoint’s operational purpose.

Recommended user-facing status messages

Because the deterministic simulation is authoritative, user-facing messages should never suggest that OpenAI decides whether the event exists or whether the visual appears. Recommended messages:

These messages preserve the contract: simulation first, narrative optional, fallback always available.

Deterministic fallback requirements

The public copy already points in the right direction, and the production behavior should match it:

• The deterministic simulation must create the event before any provider round-trip.
• Event existence, numeric state, and visuals must remain authoritative even if provider calls fail.
• The browser should never block the main interaction on an optional narrative path.
• On any provider error, the UI should fall back immediately to the deterministic brief, ideally with a short non-alarming status note.
• Historical research events must never be rewritten by model output.

Production acceptance, rollback, and remaining unknowns

Production acceptance procedure

A robust acceptance procedure for this service should prove each of the following cases end to end, without revealing secrets.

One successful provider request. Use a known fictional event that satisfies the site’s abstract-input contract. Confirm that the deterministic event exists first, the browser never receives a provider key, the server returns a successful narrative payload, and the resulting text does not mutate numeric state. Privately log the OpenAI x-request-id server-side for this one request. OpenAI recommends production logging of request IDs for troubleshooting.

One deliberately disabled provider request. Set the server feature gate off in a controlled environment. Confirm that the capability GET reports a disabled reason, that no provider POST is attempted, and that the browser immediately exposes the deterministic brief path.

One invalid-key test in nonproduction. Use a nonproduction environment only. Introduce an intentionally invalid key or invalid file contents. Confirm that the capability GET or POST path returns the appropriate nonsecret local reason, that no credential-shaped data is exposed, and that the user sees deterministic fallback. OpenAI’s documented 401 classes make this test meaningful.

One outbound-network failure test. Block egress from the app host to api.openai.com:443 or simulate failure in a controlled network namespace. Confirm that the local reason maps to a transport failure, not a generic disabled state, and that fallback is immediate.

One structured-output rejection test. In nonproduction, intentionally tighten or mismatch the local schema to force a local schema rejection while keeping transport/auth healthy. Confirm distinction between provider_response_invalid, structured_output_missing, and structured_output_rejected. OpenAI’s current Structured Outputs docs make this especially important because the Responses API shape differs from older patterns.

Given When Then acceptance tests

The table below provides more than the requested minimum. These are acceptance tests to run against the actual application, not claims that they have already passed.

Rollback procedure

A safe rollback should preserve the deterministic simulation and remove only the optional provider dependency. Recommended order:

First, disable the feature gate for narrative generation at the server level so the capability GET reports a deliberate disabled state rather than a broken state. That keeps user messaging clean and preserves deterministic behavior.

Second, if the deployment uses env-based secrets, remove or invalidate the narrative feature configuration in a controlled change window and reload or restart the relevant serving process—Apache, PHP-FPM, or the container runtime—depending on hosting mode. The need for restart/reload is usually an operational inference from how these processes inherit environment.

Third, confirm that the application now presents deterministic fallback immediately and does not attempt outbound provider calls.

Fourth, preserve private server logs for the failing release before changing anything else, especially local reason codes and any captured provider request IDs.

Fifth, once the provider path is repaired in staging, re-enable it there first, not in production, and replay the acceptance procedure.

Remaining unknowns

Several important questions remain unresolved because they require a live endpoint or server runtime that I was not able to observe directly in this session:

• The actual current GET status code, body, and headers for /api/simulation-consequences.php.
• The current live values of enabled, configured, available, reason, mode, release, and generation.
• Whether the browser currently attempts the capability GET on page load in a live JS session.
• Whether the browser currently attempts the narrative POST after a fictional event.
• The actual current provider model ID.
• Whether the server currently uses OpenAI Responses with the correct text.format schema shape.
• Whether the server explicitly sets store:false.
• Whether the actual hosting stack is mod_php, PHP-FPM, CGI/FastCGI, shared hosting, or containerized.
• Whether environment variables that may exist in a shell are truly visible to the PHP worker runtime.
• Whether outbound DNS/TLS to api.openai.com from the origin host is healthy.
• Whether user-observed failure, if any, is before POST, in same-origin PHP, or after outbound provider contact.

The public web evidence is strong enough to assess the intended architecture and identify the most likely failure classes. It is not strong enough to certify current endpoint health. The honest bottom line is therefore: the design contract appears sound in public, but the live capability state remains unverified, and I did not observe a successful provider response.