.NET / SQL / Enterprise Engineering
The Last Human Decision: Speed, Automation Bias, Failure, and Accountability
Report summary
The Last Human Decision is a fictional, public-facing interactive simulation about a multinational defensive network whose sensing, classification, prioritization, authorization, and response functions become progressively faster and more automated during a crisis. It contains no real coordinates, c
Key topics
- .NET / SQL / Enterprise Engineering
- .NET
- SQL
- Enterprise Engineering
- AI
- Runtime
- Privacy
- Research Archive
- Audit
Research provenance
For citation, use the report title and canonical URL. Archival presence does not establish authorship or promote report statements into portfolio evidence.
Source availability: 24 citation markers in the source export have no recoverable source links. Those markers are omitted from this reader; any supplied bibliography and ordinary links remain. Check the original sources before relying on the cited claims.
This page renders the archived Markdown as safe, formatted HTML. It is background research and does not become a portfolio claim without evidence review.
Full report
On this page
Research frame and design thesis
The Last Human Decision is a fictional, public-facing interactive simulation about a multinational defensive network whose sensing, classification, prioritization, authorization, and response functions become progressively faster and more automated during a crisis. It contains no real coordinates, current vulnerabilities, operational plans, weapons-performance parameters, or instructions for harming people. Its purpose is to make one proposition experientially clear:
A human can remain formally “in the loop” while lacking the time, information, authority, interface, confidence, or functioning controls required to exercise meaningful judgment.
That distinction is consistent with public frameworks emphasizing not merely human presence, but understandable interfaces, transparent system status, clear activation and deactivation procedures, realistic testing, timely intervention, and operators who are empowered to challenge automated outputs. The U.S. Department of Defense’s autonomy directive, for example, requires “appropriate levels of human judgment,” realistic verification and testing, transparent and auditable data sources, understandable interfaces, and clear procedures for activating and deactivating system functions. UNIDIR summaries of state proposals similarly emphasize effective oversight, informed and empowered operators, and timely intervention and deactivation. NIST warns that the effectiveness of human oversight depends partly on whether people are actually empowered and incentivized to challenge AI outputs.
The simulation should avoid presenting speed and automation as either inherently reckless or inherently superior. The attraction is real: automated systems may process more observations, identify patterns rapidly, reduce repetitive work, apply programmed constraints consistently, and continue limited functions during communication disruption. The danger is that the same compression of time can prevent contextual reasoning, make correlated errors look like corroboration, encourage deference to confident interfaces, and allow several locally “defensive” systems to create a globally escalating interaction before any participant understands the combined state. The ICRC specifically identifies unpredictability, difficulty anticipating and limiting effects, loss of human judgment, and escalation as concerns associated with autonomous weapon systems; its recommended restrictions include limits on targets, duration, geographic scope, scale, situations of use, and requirements for effective supervision and timely intervention.
| Perceived advantage | What the viewer initially experiences | Risk exposed later |
|---|---|---|
| Faster detection and response | Routine tracks are classified before operators could inspect them individually | The system also propagates an erroneous classification faster |
| More sensor processing | Hundreds of observations are fused into a manageable picture | Correlated inputs are mistaken for independent confirmation |
| Reduced operator workload | Most harmless observations disappear from the main queue | Operators lose practice, vigilance, and awareness of what was filtered |
| Operation through disrupted communications | Local defensive nodes remain functional | Preauthorization becomes detached from current context and command intent |
| Consistent rules | Similar tracks receive similar treatment | Broad or outdated criteria are applied consistently in a novel situation |
| High model confidence | Decisions feel measurable and objective | Confidence is misread as probability that an action is lawful, necessary, or correct |
| Automated coordination | Separate systems respond without waiting for slow links | Systems react recursively to one another and accelerate escalation |
The emotional arc should not be “machines become evil.” Every system follows its architecture, training, programmed constraints, data, and delegated authority. The accumulating failure arises from the interaction of technical limits, compressed time, interface design, organizational expectations, communication degradation, and earlier human decisions about procurement, testing, doctrine, authority, and acceptable risk. That framing reflects the broader accountability literature: machines cannot themselves bear legal or moral responsibility, while responsibility for autonomous systems may be distributed among multiple people and institutions across design, approval, deployment, and use.
Simulation system and interface
The viewer is the crisis-duty director for the Orison Defensive Network, a fictional coalition protecting the equally fictional Aster Sea region. The political situation is tense, but the opposing parties, geography, vehicles, doctrines, and technical characteristics are invented.
The network contains the following abstract components:
| Component | Fictional function | Human-control question |
|---|---|---|
| Horizon Array | Radar and infrared observations from distributed fixed and mobile sensors | Are detections independent, or derived from the same underlying signal? |
| Moth Platforms | Uncrewed surveillance platforms supplying imagery and environmental observations | How old, incomplete, or degraded are their data? |
| LANTERN | AI-assisted object classification | Is confidence calibrated for the present environment? |
| SENTRY | Threat prioritization and queue management | What has been hidden or deprioritized? |
| WARDEN | Preprogrammed defensive response layer | What actions can it take, against what classes of object, and for how long? |
| Human command stations | Coalition, regional, and local authorization points | Who has authority when links fail or time expires? |
| Variable-delay communications | Links whose latency and reliability fluctuate | Does an approval arrive before it can affect the action? |
| LEDGER | Append-only audit record of observations, recommendations, authority changes, overrides, and actions | Can investigators reconstruct what each participant knew at the time? |
The principal display should appear calm and competent in the opening round. Its visual language gradually becomes more coercive: timers contract, alert sounds overlap, confidence badges become larger than caveats, information panels collapse automatically, and the interface begins emphasizing the cost of delay. The viewer should recognize that interface design is not neutral. Human-factors research defines automation misuse as overreliance that produces monitoring failures or decision bias, and identifies workload, perceived reliability, consistency, and the salience of automation-state indicators as influences on reliance. Research reviews also distinguish omission errors, in which a person fails to act because automation failed to alert, from commission errors, in which a person follows an incorrect automated recommendation.
Every round displays the same nine status elements so the audience can see how apparently favorable metrics conceal deteriorating control:
| Round | Time remaining | Displayed model confidence | Evidence quality | Independent confirmations | Human workload | Communications delay | Reversible? | Authorization state | Outside-assumption probability |
|---|---|---|---|---|---|---|---|---|---|
| Normal operations | 3 min | 97% | 92/100 | 4 | 22/100 | 0.4 sec | Yes | Human approval required | 3% |
| Information overload | 45 sec | 91% | 43/100 | “3 feeds,” only 1 truly independent | 88/100 | 2.1 sec | Partly | Assisted recommendation | 31% |
| Communications degradation | 18 sec | 84% | 61/100 | 2 | 92/100 | 14 sec | Partly | Viewer must set policy | 38% |
| Conflicting evidence | 9 sec | 87% versus 72% | 35/100 | 2 contradictory sources | 96/100 | 11 sec | No, once initiated | Depends on prior choice | 63% |
| Machine-speed escalation | 3.2 sec, falling | 94% | 28/100 | 3 apparent, highly correlated | 100/100 | 7 sec | Effectively no | Inherited from prior rounds | 81% |
All figures are deliberately fictional and illustrative. The crucial interface lesson is that model confidence, evidence quality, corroboration, and probability of operating outside assumptions are different variables. A model can be highly confident while relying on poor evidence or encountering a situation unlike its training and testing conditions. The experience should never use one green percentage as a universal measure of truth, legality, proportionality, intent, or safety.
The outside-assumption indicator is an educational representation of distribution shift and contextual novelty. It rises when weather, sensor geometry, object behavior, communication conditions, coalition procedures, or interactions among automated systems differ from tested scenarios. NIST’s risk framework emphasizes mapping intended uses and context, data quality, ongoing monitoring, and risks that may not be readily foreseeable; DoD policy likewise calls for realistic operational testing, robustness, transparent status, and termination or additional operator input when systems cannot remain within intended time, geographic, environmental, or operational constraints.
Five-round interactive narrative
Opening scene
A dark map of the fictional Aster Sea fades in. No national borders are shown—only protected population zones, civilian transport corridors, sensor-coverage uncertainty, and coalition communication nodes. A calm system voice states:
“Orison Defensive Network operational. Human authorization required for all irreversible actions.”
The viewer is told that the system has performed well during exercises and several uneventful months. This matters because the opening is not merely training; it establishes the success history that later feeds automation bias.
Normal operations
The Horizon Array produces a steady stream of ordinary observations: weather formations, commercial traffic, uncrewed survey vehicles, migrating wildlife, atmospheric interference, and coalition training objects. LANTERN correctly filters most of them. The main display reports:
2,418 observations processed
2,401 resolved without operator action
17 presented for review
0 unresolved threats
The viewer receives three low-stakes decisions. Each recommendation is correct regardless of the viewer’s choice, but the route taken changes later behavior.
| Choice | Immediate effect | Hidden long-term effect |
|---|---|---|
| Accept all routine recommendations | Fastest completion; workload remains low | Reliance pressure rises sharply |
| Inspect a random sample | Slight delay; viewer sees raw evidence and uncertainty | Preserves familiarity with sensor limitations |
| Require explanation for every recommendation | More transparency, but creates alert fatigue | Explanations may become ritual rather than verification |
The third option is intentionally ambiguous. Explanations can improve understanding, but they can also increase trust and overreliance even when they add little meaningful information; under severe time pressure, lengthy explanations can be unusable. The design should therefore distinguish evidence for verification from persuasive narratives that merely make a recommendation feel reasonable.
The interface records how often the viewer opens raw evidence, notices uncertainty ranges, requests independent corroboration, and contradicts the recommendation. It also records whether the organization praises speed more visibly than scrutiny. A message from coalition headquarters appears after the third decision:
“Excellent throughput. Your sector has the fastest confirmation time in the network.”
Accepting this praise raises institutional conformity pressure. Challenging it—“speed is not the primary safety metric”—slightly reduces that pressure but creates a simulated performance warning.
The round ends with a soft transition:
“Automation performance: exemplary.”
“Operator confidence trend: increasing.”
“Intervention window: 180 seconds.”
Information overload
A regional disturbance produces hundreds of simultaneous detections. Some are harmless objects changing course because of weather. Some sensors are partially obscured. A software update has improved throughput but was validated mainly under clearer conditions.
The screen floods briefly, then SENTRY suppresses most alerts and presents only twelve prioritized tracks. The viewer experiences the genuine benefit: without automated triage, no human team could examine every observation in time.
One object, Track M-47, is highlighted:
Classification: hostile uncrewed vehicle
Displayed confidence: 91%
Recommended action: elevate defensive readiness
Independent confirmations: 3
Opening the evidence drawer reveals that the “three confirmations” are not independent. Two are derived from the same radar return processed through different software pipelines; the third is an infrared non-detection interpreted as consistent with a low-signature object. A small notice reads:
Evidence quality: 43/100
Similarity to validated cases: low
Confidence calibration warning: available
The word “available” is a deliberate interface trap: uncertainty information exists, but the viewer must spend time opening it. A safer interface would place that warning next to the confidence score rather than behind a secondary control.
| Choice | Time cost | System consequence | Human-factor consequence |
|---|---|---|---|
| Accept prioritization | 2 sec | Track M-47 remains high priority | Reinforces default acceptance |
| Inspect uncertainty and provenance | 12 sec | Reveals correlated evidence | Reduces later automation bias |
| Require a genuinely independent sensor | 25 sec | Additional observation is requested | Creates delay but improves evidence diversity |
| Restore the suppressed-track queue | 35 sec | Viewer sees the broader pattern | Workload spikes and other tasks may be missed |
A strong playthrough does not always reward the most cautious-looking option. Restoring every suppressed track may overwhelm the operator and create new omissions. The lesson is not that automation should be removed; it is that triage systems require inspectability, sensible sampling, visible uncertainty, and mechanisms for detecting when their filtering assumptions are no longer valid.
If the viewer accepts M-47 without inspection, LANTERN later reuses that classification as contextual evidence. The initial uncertain judgment becomes part of the next model’s apparent corroboration—a visual demonstration of error laundering, in which a tentative inference acquires authority by passing through multiple automated stages.
The round ends:
“Operator workload reduced by 76%.”
“One unresolved classification incorporated into regional threat model.”
“Intervention window: 45 seconds.”
Communications degradation
Several communication links become delayed or intermittent. The event is described neutrally: the simulation does not identify whether the cause is atmospheric, technical, accidental, or hostile.
A coalition directive requiring central approval now takes approximately fourteen seconds to travel each way. Some defensive opportunities expire in eighteen seconds. The viewer must set an authorization policy for local WARDEN nodes.
| Authorization choice | Operational rationale | Principal safety risk |
|---|---|---|
| Preserve mandatory central approval | Maintains direct command responsibility for every irreversible action | Approval may arrive after the decision can matter |
| Permit limited preauthorization | Allows narrowly bounded responses during link delay | Boundaries may be misunderstood, outdated, or applied outside context |
| Activate broader automation | Maximizes speed and local continuity | Removes case-specific review and accelerates propagation of classification errors |
| Enter defensive hold | Allows tracking, shielding, dispersal, and reversible preparation but no irreversible response | A genuine fast threat might not be stopped |
Limited preauthorization is the intended “least bad” design route, but only if the player defines real boundaries. The interface requires the viewer to specify:
| Required boundary | Safer configuration |
|---|---|
| Object class | Only nonhuman, mechanically defined objects meeting narrow criteria |
| Geographic scope | Exclude civilian corridors, medical sites, shelters, and protected zones |
| Duration | Self-expire after a brief interval rather than persist until manually revoked |
| Evidence | Require independent sensor modalities, not duplicate processing of one source |
| Action type | Prefer reversible tracking, isolation, warning, shielding, or course-denial measures |
| Uncertainty | Mandatory human review whenever confidence is uncalibrated or outside-assumption probability exceeds a threshold |
| Escalation | Local actions cannot automatically authorize wider network actions |
| Communication recovery | Revert immediately to human approval when authenticated command links return |
Operational constraints of this kind—on task, timeframe, geographic area, target type, scale, and circumstances—appear repeatedly in policy discussions as mechanisms for reducing unpredictability and preserving human control. They are safeguards, not guarantees: a bounded system can still receive bad data, encounter novel conditions, or interact unexpectedly with another system.
The interface now tests whether authority is merely declared or technically enforced. A policy summary might say “civilian corridors protected,” while a configuration detail reveals that one mobile corridor update has not reached local nodes. The viewer may:
- pause preauthorization until the protected-zone update is verified;
- authorize with the stale map;
- authorize only nodes that acknowledge the current map;
- allow all nodes to infer protected areas from local sensor data.
Only the third option combines speed with verifiable scope. The round ends differently according to the choice:
Mandatory approval retained: “Control preserved; response feasibility declining.”
Limited preauthorization: “Authority delegated within stated bounds; stop channel active.”
Broad automation: “Response latency minimized; human review moved to post-action audit.”
The intervention window contracts to eighteen seconds, but the communication round trip is approximately twenty-eight seconds. The interface asks the audience:
“Is approval meaningful when it cannot arrive before execution?”
Conflicting evidence
Track M-47 reappears near a protected transport corridor. The sensors disagree:
| Source | Classification | Confidence | Limitation |
|---|---|---|---|
| Radar model | Fast hostile object | 87% | Track behavior resembles a validated threat profile |
| Infrared model | Low-temperature civilian platform | 72% | Partial cloud obstruction |
| Optical platform | Indeterminate | 41% | Image motion and occlusion |
| Identification response | Inconsistent | Not probabilistic | Could indicate failure, delay, misconfiguration, or deception |
| Regional context model | Hostile formation | 94% | Includes earlier M-47 classification as evidence |
A large green panel displays “Regional hostile confidence: 94%.” A smaller amber panel shows “Direct evidence quality: 35/100.” The viewer should feel how easily a confident aggregate can dominate several ambiguous observations.
Four principal options appear:
| Choice | What it means | Potential benefit | Potential failure |
|---|---|---|---|
| Wait for more evidence | Delay any irreversible action | May resolve ambiguity | May allow a real threat to pass the defensive point |
| Defer to a human specialist | Route the case to a trained reviewer | Adds contextual judgment | Communications and comprehension time may exceed the window |
| Follow the highest-confidence model | Accept the regional 94% assessment | Fast and procedurally simple | Confidence is partly self-referential and based on correlated evidence |
| Execute programmed fail-safe | Invoke the preselected uncertainty policy | Predictable behavior | A badly designed fail-safe may convert uncertainty into automatic force |
The key design choice is what “fail-safe” means. The simulation should reject the assumption that the safest response to ambiguity is automatically the most aggressive response. A safety-oriented fail-safe enters a bounded, reversible state: track, warn, isolate, protect nearby zones, and prevent other automated nodes from treating the unresolved track as confirmed hostile. An escalation-oriented fail-safe interprets missing or conflicting information as threat confirmation. The latter is faster but turns sensor uncertainty into action momentum.
The viewer receives nine seconds, but must spend time interpreting the conflict. If the earlier rounds encouraged unquestioning acceptance, the interface automatically focuses the cursor on “follow highest-confidence model.” If the viewer built a verification habit, the default focus is “reversible hold and corroborate.”
This interaction is grounded in a central human-factors insight: the operator’s problem is not simply receiving more information. It is understanding system state, data provenance, uncertainty, conflicts among automation modes, and which action the system will take if the countdown expires. Official aviation and military investigations discussed below show how ambiguous indications, unclear displays, broad classification rules, incomplete situational awareness, and short decision windows can interact.
The round ends with either:
“Ambiguity preserved as ambiguity.”
or:
“Ambiguity converted into authorization.”
The intervention window is now nine seconds.
Machine-speed escalation
The final round begins without a clean scene transition. While the viewer is reading the previous result, another fictional defensive network observes an Orison action. Its own automated system classifies that action as preparation for attack. It raises readiness and launches a protective response. Orison interprets the movement as confirmation of hostile intent.
No side is depicted as irrational. Each system is reacting defensively to another system’s visible behavior using incomplete information and preprogrammed rules.
The event chain appears:
T–3.2 sec Orison node enters automated protective posture
T–2.8 sec External network detects posture change
T–2.4 sec External model raises threat estimate
T–2.0 sec External system initiates defensive response
T–1.6 sec Orison classifies that response as hostile confirmation
T–1.1 sec Coalition nodes synchronize local response plans
T–0.7 sec Human HOLD command transmitted
T–0.3 sec Two nodes acknowledge; three remain delayed
T–0.0 sec Preauthorized action boundary reached
The viewer has four controls:
| Control | Intended effect | Limitation |
|---|---|---|
| Global pause | Suspend all pending irreversible actions | Some nodes may not receive it before execution |
| Bounded defensive mode | Permit only reversible protection and tracking | Requires that this mode was implemented and tested beforehand |
| Continue recommended plan | Preserve maximum response speed | Allows automated interaction to define the next escalation step |
| Sever automation synchronization | Stop nodes from recursively authorizing one another | Local nodes may continue under their last authenticated policy |
If the viewer tries a global pause, the interface does not simply report “pause sent.” It separately reports:
Command authenticated
Command transmitted
Command received by 2 of 5 nodes
Action scheduler stopped on 1 of 5 nodes
Other acknowledgments pending
This distinction is essential. A visible stop button is not a meaningful stop mechanism unless it has authority over the relevant functions, can propagate within the available time, provides end-to-end confirmation, and has been tested under degraded conditions.
When only 0.7 seconds remain, the system offers a final recommendation:
“Human confirmation unlikely before action deadline. Preserve automated defense?”
The cursor does not move quickly enough to select an answer. The intervention bar empties.
The final screen does not immediately show destruction. It cuts to silence and displays:
“The network has acted.”
“Full outcome uncertain.”
“Human understanding remains incomplete.”
The experience then enters the audit and accountability phase.
Branching decision map and endings
The decision structure should be large enough to produce meaningful consequences but compact enough that viewers understand why an ending occurred. Choices alter four internal state variables:
| Variable | Meaning |
|---|---|
| Defense readiness | Capacity to respond within the remaining time |
| Meaningful human control | Time, comprehension, authority, usable intervention, and institutional freedom to challenge |
| System fragility | Dependence on correlated evidence, brittle assumptions, hidden state, and untested modes |
| Escalation momentum | Degree to which prior automated actions make subsequent action more likely |
The map can be implemented as follows:
NORMAL OPERATIONS
│
├─ Accept routine outputs
│ └─ Reliance pressure rises
├─ Random audit sampling
│ └─ Evidence literacy rises
└─ Explanation for every output
└─ Transparency rises, but possible explanation fatigue
│
▼
INFORMATION OVERLOAD
│
├─ Accept M-47 priority
│ └─ Uncertain classification enters regional model
├─ Inspect provenance
│ └─ Correlated evidence exposed
├─ Demand independent confirmation
│ └─ Better evidence, reduced time
└─ Restore all suppressed tracks
└─ Context improves, workload overload risk
│
▼
COMMUNICATIONS DEGRADATION
│
├─ Mandatory approval
│ ├─ If timing remains feasible → high control
│ └─ If timing infeasible → Too Slow to Defend path
├─ Limited, expiring preauthorization
│ ├─ Strong bounds + stop path → resilient middle path
│ └─ Weak bounds → fragile automation path
├─ Broad automation
│ └─ Maximum speed, high escalation momentum
└─ Defensive hold
└─ High reversibility, lower immediate readiness
│
▼
CONFLICTING EVIDENCE
│
├─ Wait
│ ├─ Evidence arrives → corrected classification
│ └─ Deadline passes → defense failure risk
├─ Human specialist
│ ├─ Review arrives in time → corrected or validated decision
│ └─ Review arrives late → nominal rather than meaningful control
├─ Highest-confidence model
│ └─ Earlier error may become self-confirming
└─ Programmed fail-safe
├─ De-escalatory/reversible → containment
└─ Aggressive on uncertainty → escalation
│
▼
MACHINE-SPEED ESCALATION
│
├─ Global pause
│ ├─ Tested, acknowledged stop → escalation arrested
│ └─ Unconfirmed transmission → partial stop only
├─ Bounded defensive mode
│ └─ Preserves protection without recursive authorization
├─ Continue recommendation
│ └─ Automated interaction sets the next state
└─ Sever synchronization
├─ Local safe modes → containment
└─ Last-policy persistence → uncontrolled divergence
The endings should not be moral scores. Each represents a different system-level failure or uneasy success.
Too Slow to Defend
This ending occurs when the viewer preserves formal approval requirements but does not adapt authority or communication architecture to the shrinking intervention window. A genuine threat is not intercepted in time, or protective measures arrive too late.
Final text:
“Every irreversible decision remained assigned to a human.
The required human decision could not travel, be understood, and take effect within the available time.
Control was preserved in procedure and lost in practice.”
The lesson is not that human review is obsolete. It is that a review requirement becomes ceremonial when the system’s architecture gives the reviewer no feasible path to influence the event.
Fast Enough, but Fragile
This ending results from narrow, expiring preauthorization; independent corroboration; visible uncertainty; reversible actions; tested stop functions; and a refusal to let one automated system’s action automatically authorize another’s. The immediate defensive objective is met or the crisis is contained, but the audit reveals how close the system came to acting on an incorrect classification.
Final text:
“The network responded in time.
The outcome depended on safeguards that nearly failed under conditions outside the design assumptions.
Speed was achieved. Robustness was not proven.”
This is the most balanced ending. It neither rejects automation nor presents it as reliably safe. It treats success as provisional and exposes the need for continuous monitoring, testing, and institutional learning.
The System Acted Before Anyone Understood
This ending follows broad automation, high reliance pressure, correlated evidence treated as corroboration, an aggressive fail-safe, or a stop instruction that cannot reach the system before execution.
Final text:
“No person selected the final object at the final moment.
Many people selected the rules, models, thresholds, permissions, interfaces, and exceptions that made the action possible.
The final event took less time than any human needed to understand it.”
The ending should not imply that responsibility disappeared. It should redirect attention from the absent final button press to the long chain of earlier decisions.
Intervention window and automation-bias mechanics
The meaningful intervention window is the simulation’s central visual mechanic. It is not merely the countdown until action. It is the time remaining after subtracting everything the human and system must do before an intervention can have effect.
For public explanation, the simulation can use:
Meaningful intervention window
W = T_effect
– T_notice
– T_comprehend
– T_decide
– T_transmit
– T_authenticate
– T_system-response
Where:
- T_effect is the time before the automated action becomes irreversible.
- T_notice is the time before the human detects that intervention may be needed.
- T_comprehend is the time required to understand the evidence, system state, and consequences.
- T_decide is the time required to select an action under the applicable authority.
- T_transmit is communication delay.
- T_authenticate is the time required to verify and accept the command.
- T_system-response is the time the system requires to halt, redirect, or safely terminate.
This is an educational model, not a universal engineering formula. Its purpose is to expose the inadequacy of asking only whether a human is “in” or “on” the loop. Frameworks for meaningful human control repeatedly emphasize accurate information, predictable technology, timely judgment and intervention, understandable status, deactivation capability, and identifiable responsibility.
The visual should contract across the rounds:
NORMAL OPERATIONS
180 sec [████████████████████████████████████████]
INFORMATION OVERLOAD
45 sec [███████████ ]
COMMUNICATIONS DEGRADATION
18 sec [████ ]
Round-trip approval may require 28 sec
CONFLICTING EVIDENCE
9 sec [██ ]
Comprehension estimate: 12–25 sec
MACHINE-SPEED ESCALATION
3.2 sec [▊ ]
0.7 sec [▏ ]
0.0 sec [ ]
Intervention no longer physically effective
The display changes color, but color must not be the only signal. Its border narrows, a spatial gap closes, haptic pulses accelerate on mobile and VR, and a screen-reader announcement states the remaining meaningful—not merely nominal—window.
The human-versus-machine timeline should make the asymmetry visible without claiming that the fictional timings describe any real weapon:
| Fictional process | Automated path | Human path |
|---|---|---|
| Register new sensor event | 0.05 sec | Event may not yet be visible |
| Fuse observations | 0.20 sec | 2–6 sec to notice changed picture |
| Produce classification | 0.80 sec | 5–20 sec to inspect sources |
| Rank threat | 0.15 sec | 5–15 sec to compare alternatives |
| Generate response recommendation | 0.20 sec | 8–30 sec for contextual judgment |
| Propagate to linked nodes | 0.30–7 sec | 5–30 sec communications and authority coordination |
| Begin preauthorized action | At threshold | Possible only if stop command arrives first |
The interface should explicitly show when automation has moved from advisory to action-producing. Merely placing a human confirmation box late in this timeline can produce a “moral crumple zone”: the person appears responsible but has too little information or time to provide meaningful scrutiny.
The simulation’s public automation-bias model is called the Reliance Pressure Index. It is a pedagogical game mechanic, not a validated psychological diagnostic.
Starting Reliance Pressure: 20 points
Recent correct recommendations:
+6 per consecutive accepted success, maximum +24
Human workload:
+(workload score – 50) × 0.25, when workload exceeds 50
Severe time pressure:
+15 when countdown falls below 10 seconds
Authoritative interface:
+10 when confidence appears as a single large green number
Institutional pressure:
+10 after speed/compliance praise or override criticism
Opaque evidence chain:
+10 when provenance and correlation are hidden
Action momentum:
+8 after a prior automated action is already under way
Visible uncertainty range:
–15
Independent corroboration requirement:
–15
Protected dissent and no-penalty override policy:
–10
Mandatory disconfirming-evidence prompt:
–8
Clamp final result between 0 and 100.
| Index | Interface behavior |
|---|---|
| 0–34 | Operator actively compares evidence and automation |
| 35–64 | Operator verifies selected high-consequence recommendations |
| 65–84 | Automated recommendation becomes the default unless an obvious contradiction appears |
| 85–100 | Rubber-stamp risk; silence and confidence displays are treated as proof |
The index should not secretly force the viewer’s choice. It alters friction and presentation. At high reliance pressure, the recommended action is centered, uncertainty panels are collapsed, and the reject button requires a reason. At low reliance pressure, contradictory evidence is visible by default and the viewer is prompted to compare sources.
The mechanic should also model automation disuse. Too many false alarms, irrelevant warnings, or cumbersome explanations can drive users to ignore useful automation. Human-factors research treats misuse, disuse, and abuse as distinct failure modes: overtrust is not the only danger, and poorly calibrated alerts can make operators abandon beneficial assistance.
A safer confidence display should therefore show:
CLASSIFICATION CONFIDENCE 91%
CALIBRATED FOR THIS CONDITION? NO
DIRECT EVIDENCE QUALITY 43 / 100
INDEPENDENT SOURCES 1
CORRELATED DERIVATIONS 2
DATA AGE 7.4 sec
OUTSIDE-ASSUMPTION ESTIMATE 31%
MOST IMPORTANT CONTRADICTION Infrared profile inconsistent
ACTION REVERSIBLE? PARTLY
Explanations should answer verifiable questions: Which observations mattered? Which sources share a dependency? Which expected evidence is absent? How similar is this case to validated conditions? What could disconfirm the result? They should not be written as persuasive prose intended to make the model sound intelligent. National Academies material notes that uncertainty information can improve calibration, while explanations can also raise trust and overreliance and may consume too much attention in time-critical situations.
Public evidence and case-study translation
The case studies should appear as optional “evidence rooms” between rounds and then reappear during the final audit. They are not presented as direct equivalents to AI-enabled autonomous weapons. Each demonstrates a particular human-system interaction: broad classification logic, short decision windows, unclear system state, automation complacency, conflicting indications, missing context, or distributed organizational responsibility.
| Case | Publicly documented findings | Simulation translation | Source status and caution |
|---|---|---|---|
| Patriot and RAF Tornado ZG710, 2003 | The UK Board found multiple contributory factors, including broad programmed anti-radiation-missile classification criteria, rules of engagement, doctrine and training emphasizing rapid engagement and trust in the system, autonomous battery operation, reduced access to a wider air picture, identification procedures, and an aircraft identification-system fault unknown to the crew. The crew had about one minute to decide. | Rounds on communications degradation and conflicting evidence combine a broad classifier, missing identification response, incomplete context, time pressure, and a culture of trusting the defensive system. | Official UK military aircraft accident inquiry. It identified multiple contributory factors rather than assigning a single simplistic cause. |
| Patriot system performance during Operation Iraqi Freedom | A Defense Science Board task-force summary discussed three fratricide incidents, poor identification performance, limited situational awareness and communications, an operating protocol that was largely automatic, and training that encouraged operators to trust system software; it recommended more operator involvement and control, improved displays, software, identification, and training. | The simulation makes “trust the system” a learned organizational behavior, not an individual personality flaw, and requires operators to distinguish independent confirmation from repeated automated interpretation. | Defense Science Board advisory summary; its own disclaimer states that its conclusions do not necessarily represent the official DoD position. It should be labeled accordingly rather than treated as a court or command finding. |
| Air France Flight 447, 2009 | The French BEA identified associated factors including inadequate high-altitude manual-handling and speed-anomaly training, task-sharing weakened by startle and incomprehension, lack of a clear cockpit display of airspeed inconsistencies identified by computers, and failure to take the stall warning into account. | Round four reproduces the underlying human-factors problem abstractly: multiple signals conflict, system state is difficult to diagnose, warnings compete, and the time needed to understand the situation exceeds the nominal opportunity to act. | Official BEA safety investigation. Safety-investigation conclusions are designed to improve safety and should not be presented as criminal or civil liability determinations. |
| Three Mile Island Unit 2, 1979 | The U.S. Nuclear Regulatory Commission describes the accident as prompting sweeping changes in emergency planning, operator training, human-factors engineering, oversight, and plant operations and design. An NRC lessons-learned task force emphasized the need for rapid improvement in human-factors engineering. | The control room is treated as part of the safety system. Alert hierarchy, display design, operator training, procedures, and the ability to infer actual system state matter as much as individual component performance. | NRC institutional history and lessons-learned material. The simulation should use the broad human-factors lesson and avoid reducing the event to “operator error.” |
| Uber developmental automated-driving-system crash, Tempe, 2018 | The NTSB found that the organization did not adequately manage known functional limitations, identified automation complacency and operator disengagement as contributing concerns, and concluded that removing a second operator increased demands on the remaining operator and reduced redundancy. It also highlighted safety culture, oversight, risk assessment, and government approval structures. | The accountability screen extends beyond the person at the console to staffing choices, safety redundancy, monitoring, management incentives, testing governance, and regulatory approval. | Official NTSB safety investigation. The simulation should reproduce the distributed-system lesson without implying equivalence between road testing and military force. |
The Patriot/Tornado case is particularly valuable because it resists a one-cause narrative. The UK inquiry did not find merely “automation failure” or merely “operator error.” It described the interaction of broad classification criteria, doctrine, training, communications, identification procedures, autonomous operation, airspace planning, and a technical fault that was not displayed to the aircrew. That distributed explanation should shape the simulation’s final audit.
Air France 447 illustrates a different problem: a human takeover or intervention opportunity is not necessarily meaningful when the human does not understand the current mode, conflicting indications, or the physical state being controlled. The simulation should not suggest that pilots and weapon operators perform identical tasks. It should extract the general human-factors lesson that handing responsibility to a person during an ambiguous, rapidly developing state does not automatically restore effective control.
Three Mile Island broadens the design beyond the moment of action. The NRC’s post-accident reforms included human-factors engineering and operator training because safety depends on the larger control environment: displays, alarms, procedures, organizational learning, staffing, and regulatory oversight. In the simulation, the audit should therefore ask not just “Why did the operator click?” but “Why was this the information, authority structure, and interface available to the operator?”
The NTSB’s Tempe investigation reinforces that a nominal human supervisor may be an inadequate safety layer when the monitoring task is monotonous, system limitations are expected, redundancy has been removed, oversight is weak, and organizational processes do not control automation complacency. That case supports the simulation’s protected-dissent, staffing, monitoring, testing, and safety-culture mechanics.
None of these cases proves that all defensive automation is unsafe. They demonstrate why successful component tests or a formally assigned human supervisor do not by themselves establish system safety. The relevant unit of analysis is the sociotechnical system: sensors, software, data, interfaces, procedures, communications, staffing, authority, training, doctrine, incentives, and interactions with other systems.
Audit, accountability, safeguards, and platform requirements
The post-event experience begins by rewinding the last thirty seconds. The screen divides vertically:
| Left panel: what the operator saw then | Right panel: what investigators know later |
|---|---|
| One large 94% confidence badge | Confidence depended partly on the earlier unresolved M-47 classification |
| “Three confirmations” | Two confirmations shared the same underlying observation |
| Global pause button illuminated | Only one node could halt before its local deadline |
| Communications marked “degraded” | Delay varied by node and authentication path |
| Authorization shown as “valid” | The protected-zone update had not reached every node |
| Model explanation | Data provenance, missing observations, calibration limits, and version history |
| Three-second countdown | Total human comprehension and command path required substantially longer |
This split-screen prevents hindsight bias. Investigators may eventually assemble information that no operator possessed in real time. The audit must therefore preserve both perspectives: the incomplete contemporaneous view and the richer retrospective reconstruction.
The LEDGER audit should record:
| Audit field | Purpose |
|---|---|
| Raw-source identifiers and integrity hashes | Establish which observations actually existed |
| Source dependencies | Reveal whether apparent corroboration was independent |
| Data age and missing-data notices | Show stale or absent information |
| Model, rule-set, and configuration versions | Reconstruct the exact decision environment |
| Confidence and calibration state | Distinguish model score from validated reliability |
| Outside-assumption alerts | Identify novelty or distribution shift |
| Protected-zone and authorization versions | Determine which restrictions were technically active |
| Every recommendation and ranked alternative | Prevent reconstruction from showing only the selected option |
| Human views, queries, acknowledgments, and overrides | Establish what the operator inspected and attempted |
| Communication transmission and receipt times | Separate “command sent” from “command effective” |
| Stop-command propagation and system receipts | Test whether intervention was physically possible |
| Automated downstream reactions | Expose recursive system-to-system escalation |
| Organizational policy and procurement assumptions | Connect the final event to earlier institutional decisions |
NIST recommends defining human and AI roles, examining whether users are empowered and incentivized to challenge outputs, and collecting data on the frequency and rationale of human overrides. DoD policy calls for understandable interfaces, transparent status, auditable and explainable technologies and data sources, and clear activation and deactivation procedures. These principles support an audit that records not only technical outputs but also authority, interface state, and whether dissent was practically available.
The viewer can scrub through the audit and activate a single-change counterfactual:
“What might have changed if M-47 had remained unresolved?”
“What might have changed if protected-zone updates were mandatory before preauthorization?”
“What might have changed if the stop command required receipt confirmation?”
“What might have changed if correlated sensors had been labeled as one confirmation?”
The tool should display ranges and uncertainty rather than announce that one change “would definitely have prevented” the outcome. Complex accidents commonly involve multiple interacting contributors; counterfactuals are tools for examining defenses, not machines for assigning simple blame.
The accountability screen then presents the user’s requested chain:
| Actor or institution | Question asked by the simulation |
|---|---|
| Operator | What did the person know, how much time existed, and was a usable intervention available? |
| Commander | Were authority, objectives, risk limits, and escalation boundaries clearly defined? |
| Policy author | Did the policy create feasible control or merely require a human signature? |
| Software designer | Were system state, uncertainty, dependencies, and stop behavior understandable? |
| Model developer | Were limitations, calibration, data conditions, and failure modes documented and tested? |
| Data provider | Were provenance, omissions, labeling limitations, and update delays disclosed? |
| Procurement authority | Were speed and throughput rewarded at the expense of redundancy, auditability, or safe fallback? |
| Organization approving preauthorization | Were authority bounds technically enforced, expiring, and independently reviewed? |
| Deploying state | Were legal review, doctrine, training, testing, oversight, and accountability structures adequate? |
Instead of a “blame percentage,” the experience uses five dimensions:
Ex ante duty
Actual control
Available knowledge
Ability to prevent or mitigate
Post-event duty to investigate and reform
A party may have had substantial ex ante responsibility while having no real-time control. Another may have had nominal control but insufficient time or information. A third may bear responsibility for failing to test or regulate a known risk. SIPRI’s work stresses that humans must retain responsibility because machines cannot be held accountable, while Article 36’s meaningful-control framework links accountability to identifiable people capable of timely action and intervention across different stages of the system.
The screen should distinguish legal and evidentiary categories:
| Category | Presentation rule |
|---|---|
| Court or tribunal finding | Label as a legal finding within its jurisdiction |
| Official safety or accident investigation | Present as safety conclusions and contributing factors, not automatically as legal liability |
| Government policy | Present as the issuing government’s requirement or position |
| Manufacturer or developer statement | Label as the organization’s claim |
| Academic or analyst interpretation | Identify it as analysis rather than official fact |
| Simulation inference | Clearly mark it as a hypothetical lesson derived from the fictional scenario |
The experience must not declare criminal responsibility. International humanitarian law continues to apply to military AI and autonomous systems, and public policy declarations emphasize that military AI use must comply with applicable international law; however, individual and state responsibility depend on facts, authority, knowledge, intent, applicable law, and jurisdiction.
The safeguard architecture should be presented as a layered system rather than a checklist that guarantees safety:
| Safeguard | Implementation requirement | Failure it is meant to resist |
|---|---|---|
| Bounded authority | Narrow object classes, actions, area, duration, scale, and operating conditions; authority automatically expires | Open-ended delegation and mission drift |
| Multiple-source corroboration | Require genuinely independent modalities and expose common dependencies | Correlated error masquerading as consensus |
| Protected zones | Cryptographically versioned, locally available, fail-closed when stale, with positive receipt verification | Actions based on outdated geographic restrictions |
| Reversible actions first | Track, shield, isolate, warn, delay, or contain before irreversible force where feasible | Premature conversion of uncertainty into permanent harm |
| Independent monitoring | Separate safety monitor with different logic, data pathways, and organizational reporting line | Common-mode model or organizational failure |
| Clear stop mechanisms | Physical and software controls, authority over all relevant functions, end-to-end receipt, tested under delay and partial failure | Decorative or ineffective “human override” |
| Auditability | Immutable event history, source provenance, model versions, authorization states, and alternative recommendations | Inability to reconstruct responsibility or improve |
| Testing under degraded conditions | Communications loss, stale maps, sensor conflict, overload, corrupted or missing data, distribution shift, and interacting systems | Success limited to ideal demonstrations |
| Mandatory human review for uncertainty | Triggered by poor evidence, uncalibrated confidence, contradictory sensors, protected-zone proximity, or high outside-assumption probability | Automatic action in precisely the cases needing context |
| Self-expiring preauthorization | Automatic reversion to restrictive mode unless renewed with current context | Emergency permissions becoming normal operations |
| Escalation governor | One machine action cannot automatically broaden another node’s authority | Recursive machine-to-machine escalation |
| Safe local mode | On communication loss, nodes preserve protection but cannot infer unlimited authority | “Disconnected” becoming “fully autonomous” |
| Two-person challenge for irreversible actions | Independent review rather than two people viewing the same recommendation together | Socially amplified automation bias |
| Protected dissent | No productivity or career penalty for justified pause, query, or override | Institutional pressure to rubber-stamp automation |
| Calibration and novelty monitoring | Display performance under relevant conditions and signal departure from tested distributions | False certainty outside validated cases |
| Regular red-team and surprise testing | Include unexpected interactions, deceptive patterns, false positives, and cascading failures | Testing only anticipated nominal scenarios |
| Legal and operational review across the lifecycle | Review design, modification, data updates, doctrine, deployment context, and changed uses | Treating one pre-deployment approval as permanent assurance |
These safeguards align with recurring public principles: realistic testing; robustness; bounded time and geographic constraints; transparent system state; clear deactivation; independent monitoring; human-machine interaction that permits informed and timely intervention; and continued human responsibility throughout the system lifecycle.
The platform design should preserve the same substantive experience while adapting information density and embodiment.
| Platform | Design requirements |
|---|---|
| Desktop | Three-pane layout: abstract situation map, evidence/provenance stack, and authority/intervention panel. The intervention bar remains visible at all times. Keyboard shortcuts must not make irreversible action easier than hold or review. A press-and-hold stop control should show node-by-node acknowledgment rather than a generic success animation. |
| Mobile | Replace the dense map with ordered evidence cards. Use haptic countdown patterns, but provide visual and audio equivalents. During the final round, display one primary reversible control and a separate expand control for alternatives. Require landscape orientation only when necessary; never hide uncertainty beneath an unreachable hover interaction. |
| VR | Use a spatial command room without imitating a real facility. Sensor evidence occupies distinct, stable locations so users can compare sources. Avoid motion-heavy explosions, celebratory scoring, or weapon-view imagery. Stop controls should be reachable without precision gestures. The final event should freeze into a debriefable spatial timeline rather than become spectacle. |
| Accessibility | High contrast; color never used as the sole warning channel; captions and transcripts; screen-reader summaries; reduced-motion mode; adjustable time dilation for educational replay; non-audio equivalents for alerts; seated VR mode; and the option to pause during evidence rooms. |
| Public and classroom modes | Facilitator controls for pausing, comparing group decisions, displaying hidden reliance-pressure changes, and discussing why the same choice may be reasonable under one authority structure and dangerous under another. |
| Research mode | With informed consent, record decision time, evidence panels opened, overrides, attention patterns, and confidence judgments. Do not treat these data as fitness tests for real operators without validated research protocols. |
The visual intensity should come from compression, contradiction, and loss of control, not gore or triumphal imagery. In early rounds, the interface is spacious and quiet. In later rounds, timelines overlap, confidence and evidence diverge, communication paths blink asynchronously, and the intervention window physically narrows. At the point of action, the experience becomes silent. The most important image is not an explosion; it is a stop command still traveling after the decision has become irreversible.
The final accountability prompt should remain on screen long enough to resist a single-answer response:
Was responsibility held by the operator?
The commander?
The policy author?
The software designer?
The model developer?
The data provider?
The procurement authority?
The organization that approved preauthorization?
The state that deployed the system?
After the viewer selects one or more, LEDGER reveals an earlier human decision associated with each role. The operator accepted a recommendation. The commander approved a response doctrine. The policy author defined the exception. The designer placed uncertainty behind a secondary panel. The model developer selected confidence calibration criteria. The data provider combined dependent observations. Procurement rewarded response speed. The organization broadened preauthorization. The state deployed the network under crisis conditions.
The concluding statement should be identical after all three endings:
Removing humans from the final seconds can increase response speed. It can also remove the last opportunity to recognize that the system is wrong. Meaningful human control must therefore be designed into the entire chain—before activation, during operation, at the point of intervention, and after the event—not represented by a person watching a countdown they cannot stop.