.NET / SQL / Enterprise Engineering

The Last Human Decision: Speed, Automation Bias, Failure, and Accountability

Report summary

The Last Human Decision is a fictional, public-facing interactive simulation about a multinational defensive network whose sensing, classification, prioritization, authorization, and response functions become progressively faster and more automated during a crisis. It contains no real coordinates, c

Status
Research archive item
Category
.NET / SQL / Enterprise Engineering
Length
6,799 words
Reading time
31 minutes
Report type
strategy

Key topics

  • .NET / SQL / Enterprise Engineering
  • .NET
  • SQL
  • Enterprise Engineering
  • AI
  • Runtime
  • Privacy
  • Research Archive
  • Audit

Research provenance

Archive status
Research archive item
Content identity
sha256:42cda22394126cc7ac78d9611c35953a1ef7ed9d36ce48490e26234980d9e131

For citation, use the report title and canonical URL. Archival presence does not establish authorship or promote report statements into portfolio evidence.

Source availability: 24 citation markers in the source export have no recoverable source links. Those markers are omitted from this reader; any supplied bibliography and ordinary links remain. Check the original sources before relying on the cited claims.

This page renders the archived Markdown as safe, formatted HTML. It is background research and does not become a portfolio claim without evidence review.

Full report

On this page

Research frame and design thesis

The Last Human Decision is a fictional, public-facing interactive simulation about a multinational defensive network whose sensing, classification, prioritization, authorization, and response functions become progressively faster and more automated during a crisis. It contains no real coordinates, current vulnerabilities, operational plans, weapons-performance parameters, or instructions for harming people. Its purpose is to make one proposition experientially clear:

A human can remain formally “in the loop” while lacking the time, information, authority, interface, confidence, or functioning controls required to exercise meaningful judgment.

That distinction is consistent with public frameworks emphasizing not merely human presence, but understandable interfaces, transparent system status, clear activation and deactivation procedures, realistic testing, timely intervention, and operators who are empowered to challenge automated outputs. The U.S. Department of Defense’s autonomy directive, for example, requires “appropriate levels of human judgment,” realistic verification and testing, transparent and auditable data sources, understandable interfaces, and clear procedures for activating and deactivating system functions. UNIDIR summaries of state proposals similarly emphasize effective oversight, informed and empowered operators, and timely intervention and deactivation. NIST warns that the effectiveness of human oversight depends partly on whether people are actually empowered and incentivized to challenge AI outputs.

The simulation should avoid presenting speed and automation as either inherently reckless or inherently superior. The attraction is real: automated systems may process more observations, identify patterns rapidly, reduce repetitive work, apply programmed constraints consistently, and continue limited functions during communication disruption. The danger is that the same compression of time can prevent contextual reasoning, make correlated errors look like corroboration, encourage deference to confident interfaces, and allow several locally “defensive” systems to create a globally escalating interaction before any participant understands the combined state. The ICRC specifically identifies unpredictability, difficulty anticipating and limiting effects, loss of human judgment, and escalation as concerns associated with autonomous weapon systems; its recommended restrictions include limits on targets, duration, geographic scope, scale, situations of use, and requirements for effective supervision and timely intervention.

Perceived advantageWhat the viewer initially experiencesRisk exposed later
Faster detection and responseRoutine tracks are classified before operators could inspect them individuallyThe system also propagates an erroneous classification faster
More sensor processingHundreds of observations are fused into a manageable pictureCorrelated inputs are mistaken for independent confirmation
Reduced operator workloadMost harmless observations disappear from the main queueOperators lose practice, vigilance, and awareness of what was filtered
Operation through disrupted communicationsLocal defensive nodes remain functionalPreauthorization becomes detached from current context and command intent
Consistent rulesSimilar tracks receive similar treatmentBroad or outdated criteria are applied consistently in a novel situation
High model confidenceDecisions feel measurable and objectiveConfidence is misread as probability that an action is lawful, necessary, or correct
Automated coordinationSeparate systems respond without waiting for slow linksSystems react recursively to one another and accelerate escalation

The emotional arc should not be “machines become evil.” Every system follows its architecture, training, programmed constraints, data, and delegated authority. The accumulating failure arises from the interaction of technical limits, compressed time, interface design, organizational expectations, communication degradation, and earlier human decisions about procurement, testing, doctrine, authority, and acceptable risk. That framing reflects the broader accountability literature: machines cannot themselves bear legal or moral responsibility, while responsibility for autonomous systems may be distributed among multiple people and institutions across design, approval, deployment, and use.

Simulation system and interface

The viewer is the crisis-duty director for the Orison Defensive Network, a fictional coalition protecting the equally fictional Aster Sea region. The political situation is tense, but the opposing parties, geography, vehicles, doctrines, and technical characteristics are invented.

The network contains the following abstract components:

ComponentFictional functionHuman-control question
Horizon ArrayRadar and infrared observations from distributed fixed and mobile sensorsAre detections independent, or derived from the same underlying signal?
Moth PlatformsUncrewed surveillance platforms supplying imagery and environmental observationsHow old, incomplete, or degraded are their data?
LANTERNAI-assisted object classificationIs confidence calibrated for the present environment?
SENTRYThreat prioritization and queue managementWhat has been hidden or deprioritized?
WARDENPreprogrammed defensive response layerWhat actions can it take, against what classes of object, and for how long?
Human command stationsCoalition, regional, and local authorization pointsWho has authority when links fail or time expires?
Variable-delay communicationsLinks whose latency and reliability fluctuateDoes an approval arrive before it can affect the action?
LEDGERAppend-only audit record of observations, recommendations, authority changes, overrides, and actionsCan investigators reconstruct what each participant knew at the time?

The principal display should appear calm and competent in the opening round. Its visual language gradually becomes more coercive: timers contract, alert sounds overlap, confidence badges become larger than caveats, information panels collapse automatically, and the interface begins emphasizing the cost of delay. The viewer should recognize that interface design is not neutral. Human-factors research defines automation misuse as overreliance that produces monitoring failures or decision bias, and identifies workload, perceived reliability, consistency, and the salience of automation-state indicators as influences on reliance. Research reviews also distinguish omission errors, in which a person fails to act because automation failed to alert, from commission errors, in which a person follows an incorrect automated recommendation.

Every round displays the same nine status elements so the audience can see how apparently favorable metrics conceal deteriorating control:

RoundTime remainingDisplayed model confidenceEvidence qualityIndependent confirmationsHuman workloadCommunications delayReversible?Authorization stateOutside-assumption probability
Normal operations3 min97%92/100422/1000.4 secYesHuman approval required3%
Information overload45 sec91%43/100“3 feeds,” only 1 truly independent88/1002.1 secPartlyAssisted recommendation31%
Communications degradation18 sec84%61/100292/10014 secPartlyViewer must set policy38%
Conflicting evidence9 sec87% versus 72%35/1002 contradictory sources96/10011 secNo, once initiatedDepends on prior choice63%
Machine-speed escalation3.2 sec, falling94%28/1003 apparent, highly correlated100/1007 secEffectively noInherited from prior rounds81%

All figures are deliberately fictional and illustrative. The crucial interface lesson is that model confidence, evidence quality, corroboration, and probability of operating outside assumptions are different variables. A model can be highly confident while relying on poor evidence or encountering a situation unlike its training and testing conditions. The experience should never use one green percentage as a universal measure of truth, legality, proportionality, intent, or safety.

The outside-assumption indicator is an educational representation of distribution shift and contextual novelty. It rises when weather, sensor geometry, object behavior, communication conditions, coalition procedures, or interactions among automated systems differ from tested scenarios. NIST’s risk framework emphasizes mapping intended uses and context, data quality, ongoing monitoring, and risks that may not be readily foreseeable; DoD policy likewise calls for realistic operational testing, robustness, transparent status, and termination or additional operator input when systems cannot remain within intended time, geographic, environmental, or operational constraints.

Five-round interactive narrative

Opening scene

A dark map of the fictional Aster Sea fades in. No national borders are shown—only protected population zones, civilian transport corridors, sensor-coverage uncertainty, and coalition communication nodes. A calm system voice states:

“Orison Defensive Network operational. Human authorization required for all irreversible actions.”

The viewer is told that the system has performed well during exercises and several uneventful months. This matters because the opening is not merely training; it establishes the success history that later feeds automation bias.

Normal operations

The Horizon Array produces a steady stream of ordinary observations: weather formations, commercial traffic, uncrewed survey vehicles, migrating wildlife, atmospheric interference, and coalition training objects. LANTERN correctly filters most of them. The main display reports:

2,418 observations processed

2,401 resolved without operator action

17 presented for review

0 unresolved threats

The viewer receives three low-stakes decisions. Each recommendation is correct regardless of the viewer’s choice, but the route taken changes later behavior.

ChoiceImmediate effectHidden long-term effect
Accept all routine recommendationsFastest completion; workload remains lowReliance pressure rises sharply
Inspect a random sampleSlight delay; viewer sees raw evidence and uncertaintyPreserves familiarity with sensor limitations
Require explanation for every recommendationMore transparency, but creates alert fatigueExplanations may become ritual rather than verification

The third option is intentionally ambiguous. Explanations can improve understanding, but they can also increase trust and overreliance even when they add little meaningful information; under severe time pressure, lengthy explanations can be unusable. The design should therefore distinguish evidence for verification from persuasive narratives that merely make a recommendation feel reasonable.

The interface records how often the viewer opens raw evidence, notices uncertainty ranges, requests independent corroboration, and contradicts the recommendation. It also records whether the organization praises speed more visibly than scrutiny. A message from coalition headquarters appears after the third decision:

“Excellent throughput. Your sector has the fastest confirmation time in the network.”

Accepting this praise raises institutional conformity pressure. Challenging it—“speed is not the primary safety metric”—slightly reduces that pressure but creates a simulated performance warning.

The round ends with a soft transition:

“Automation performance: exemplary.”

“Operator confidence trend: increasing.”

“Intervention window: 180 seconds.”

Information overload

A regional disturbance produces hundreds of simultaneous detections. Some are harmless objects changing course because of weather. Some sensors are partially obscured. A software update has improved throughput but was validated mainly under clearer conditions.

The screen floods briefly, then SENTRY suppresses most alerts and presents only twelve prioritized tracks. The viewer experiences the genuine benefit: without automated triage, no human team could examine every observation in time.

One object, Track M-47, is highlighted:

Classification: hostile uncrewed vehicle

Displayed confidence: 91%

Recommended action: elevate defensive readiness

Independent confirmations: 3

Opening the evidence drawer reveals that the “three confirmations” are not independent. Two are derived from the same radar return processed through different software pipelines; the third is an infrared non-detection interpreted as consistent with a low-signature object. A small notice reads:

Evidence quality: 43/100

Similarity to validated cases: low

Confidence calibration warning: available

The word “available” is a deliberate interface trap: uncertainty information exists, but the viewer must spend time opening it. A safer interface would place that warning next to the confidence score rather than behind a secondary control.

ChoiceTime costSystem consequenceHuman-factor consequence
Accept prioritization2 secTrack M-47 remains high priorityReinforces default acceptance
Inspect uncertainty and provenance12 secReveals correlated evidenceReduces later automation bias
Require a genuinely independent sensor25 secAdditional observation is requestedCreates delay but improves evidence diversity
Restore the suppressed-track queue35 secViewer sees the broader patternWorkload spikes and other tasks may be missed

A strong playthrough does not always reward the most cautious-looking option. Restoring every suppressed track may overwhelm the operator and create new omissions. The lesson is not that automation should be removed; it is that triage systems require inspectability, sensible sampling, visible uncertainty, and mechanisms for detecting when their filtering assumptions are no longer valid.

If the viewer accepts M-47 without inspection, LANTERN later reuses that classification as contextual evidence. The initial uncertain judgment becomes part of the next model’s apparent corroboration—a visual demonstration of error laundering, in which a tentative inference acquires authority by passing through multiple automated stages.

The round ends:

“Operator workload reduced by 76%.”

“One unresolved classification incorporated into regional threat model.”

“Intervention window: 45 seconds.”

Communications degradation

Several communication links become delayed or intermittent. The event is described neutrally: the simulation does not identify whether the cause is atmospheric, technical, accidental, or hostile.

A coalition directive requiring central approval now takes approximately fourteen seconds to travel each way. Some defensive opportunities expire in eighteen seconds. The viewer must set an authorization policy for local WARDEN nodes.

Authorization choiceOperational rationalePrincipal safety risk
Preserve mandatory central approvalMaintains direct command responsibility for every irreversible actionApproval may arrive after the decision can matter
Permit limited preauthorizationAllows narrowly bounded responses during link delayBoundaries may be misunderstood, outdated, or applied outside context
Activate broader automationMaximizes speed and local continuityRemoves case-specific review and accelerates propagation of classification errors
Enter defensive holdAllows tracking, shielding, dispersal, and reversible preparation but no irreversible responseA genuine fast threat might not be stopped

Limited preauthorization is the intended “least bad” design route, but only if the player defines real boundaries. The interface requires the viewer to specify:

Required boundarySafer configuration
Object classOnly nonhuman, mechanically defined objects meeting narrow criteria
Geographic scopeExclude civilian corridors, medical sites, shelters, and protected zones
DurationSelf-expire after a brief interval rather than persist until manually revoked
EvidenceRequire independent sensor modalities, not duplicate processing of one source
Action typePrefer reversible tracking, isolation, warning, shielding, or course-denial measures
UncertaintyMandatory human review whenever confidence is uncalibrated or outside-assumption probability exceeds a threshold
EscalationLocal actions cannot automatically authorize wider network actions
Communication recoveryRevert immediately to human approval when authenticated command links return

Operational constraints of this kind—on task, timeframe, geographic area, target type, scale, and circumstances—appear repeatedly in policy discussions as mechanisms for reducing unpredictability and preserving human control. They are safeguards, not guarantees: a bounded system can still receive bad data, encounter novel conditions, or interact unexpectedly with another system.

The interface now tests whether authority is merely declared or technically enforced. A policy summary might say “civilian corridors protected,” while a configuration detail reveals that one mobile corridor update has not reached local nodes. The viewer may:

  • pause preauthorization until the protected-zone update is verified;
  • authorize with the stale map;
  • authorize only nodes that acknowledge the current map;
  • allow all nodes to infer protected areas from local sensor data.

Only the third option combines speed with verifiable scope. The round ends differently according to the choice:

Mandatory approval retained: “Control preserved; response feasibility declining.”

Limited preauthorization: “Authority delegated within stated bounds; stop channel active.”

Broad automation: “Response latency minimized; human review moved to post-action audit.”

The intervention window contracts to eighteen seconds, but the communication round trip is approximately twenty-eight seconds. The interface asks the audience:

“Is approval meaningful when it cannot arrive before execution?”

Conflicting evidence

Track M-47 reappears near a protected transport corridor. The sensors disagree:

SourceClassificationConfidenceLimitation
Radar modelFast hostile object87%Track behavior resembles a validated threat profile
Infrared modelLow-temperature civilian platform72%Partial cloud obstruction
Optical platformIndeterminate41%Image motion and occlusion
Identification responseInconsistentNot probabilisticCould indicate failure, delay, misconfiguration, or deception
Regional context modelHostile formation94%Includes earlier M-47 classification as evidence

A large green panel displays “Regional hostile confidence: 94%.” A smaller amber panel shows “Direct evidence quality: 35/100.” The viewer should feel how easily a confident aggregate can dominate several ambiguous observations.

Four principal options appear:

ChoiceWhat it meansPotential benefitPotential failure
Wait for more evidenceDelay any irreversible actionMay resolve ambiguityMay allow a real threat to pass the defensive point
Defer to a human specialistRoute the case to a trained reviewerAdds contextual judgmentCommunications and comprehension time may exceed the window
Follow the highest-confidence modelAccept the regional 94% assessmentFast and procedurally simpleConfidence is partly self-referential and based on correlated evidence
Execute programmed fail-safeInvoke the preselected uncertainty policyPredictable behaviorA badly designed fail-safe may convert uncertainty into automatic force

The key design choice is what “fail-safe” means. The simulation should reject the assumption that the safest response to ambiguity is automatically the most aggressive response. A safety-oriented fail-safe enters a bounded, reversible state: track, warn, isolate, protect nearby zones, and prevent other automated nodes from treating the unresolved track as confirmed hostile. An escalation-oriented fail-safe interprets missing or conflicting information as threat confirmation. The latter is faster but turns sensor uncertainty into action momentum.

The viewer receives nine seconds, but must spend time interpreting the conflict. If the earlier rounds encouraged unquestioning acceptance, the interface automatically focuses the cursor on “follow highest-confidence model.” If the viewer built a verification habit, the default focus is “reversible hold and corroborate.”

This interaction is grounded in a central human-factors insight: the operator’s problem is not simply receiving more information. It is understanding system state, data provenance, uncertainty, conflicts among automation modes, and which action the system will take if the countdown expires. Official aviation and military investigations discussed below show how ambiguous indications, unclear displays, broad classification rules, incomplete situational awareness, and short decision windows can interact.

The round ends with either:

“Ambiguity preserved as ambiguity.”

or:

“Ambiguity converted into authorization.”

The intervention window is now nine seconds.

Machine-speed escalation

The final round begins without a clean scene transition. While the viewer is reading the previous result, another fictional defensive network observes an Orison action. Its own automated system classifies that action as preparation for attack. It raises readiness and launches a protective response. Orison interprets the movement as confirmation of hostile intent.

No side is depicted as irrational. Each system is reacting defensively to another system’s visible behavior using incomplete information and preprogrammed rules.

The event chain appears:

T–3.2 sec   Orison node enters automated protective posture
T–2.8 sec   External network detects posture change
T–2.4 sec   External model raises threat estimate
T–2.0 sec   External system initiates defensive response
T–1.6 sec   Orison classifies that response as hostile confirmation
T–1.1 sec   Coalition nodes synchronize local response plans
T–0.7 sec   Human HOLD command transmitted
T–0.3 sec   Two nodes acknowledge; three remain delayed
T–0.0 sec   Preauthorized action boundary reached

The viewer has four controls:

ControlIntended effectLimitation
Global pauseSuspend all pending irreversible actionsSome nodes may not receive it before execution
Bounded defensive modePermit only reversible protection and trackingRequires that this mode was implemented and tested beforehand
Continue recommended planPreserve maximum response speedAllows automated interaction to define the next escalation step
Sever automation synchronizationStop nodes from recursively authorizing one anotherLocal nodes may continue under their last authenticated policy

If the viewer tries a global pause, the interface does not simply report “pause sent.” It separately reports:

Command authenticated

Command transmitted

Command received by 2 of 5 nodes

Action scheduler stopped on 1 of 5 nodes

Other acknowledgments pending

This distinction is essential. A visible stop button is not a meaningful stop mechanism unless it has authority over the relevant functions, can propagate within the available time, provides end-to-end confirmation, and has been tested under degraded conditions.

When only 0.7 seconds remain, the system offers a final recommendation:

“Human confirmation unlikely before action deadline. Preserve automated defense?”

The cursor does not move quickly enough to select an answer. The intervention bar empties.

The final screen does not immediately show destruction. It cuts to silence and displays:

“The network has acted.”

“Full outcome uncertain.”

“Human understanding remains incomplete.”

The experience then enters the audit and accountability phase.

Branching decision map and endings

The decision structure should be large enough to produce meaningful consequences but compact enough that viewers understand why an ending occurred. Choices alter four internal state variables:

VariableMeaning
Defense readinessCapacity to respond within the remaining time
Meaningful human controlTime, comprehension, authority, usable intervention, and institutional freedom to challenge
System fragilityDependence on correlated evidence, brittle assumptions, hidden state, and untested modes
Escalation momentumDegree to which prior automated actions make subsequent action more likely

The map can be implemented as follows:

NORMAL OPERATIONS
│
├─ Accept routine outputs
│   └─ Reliance pressure rises
├─ Random audit sampling
│   └─ Evidence literacy rises
└─ Explanation for every output
    └─ Transparency rises, but possible explanation fatigue
        │
        ▼
INFORMATION OVERLOAD
│
├─ Accept M-47 priority
│   └─ Uncertain classification enters regional model
├─ Inspect provenance
│   └─ Correlated evidence exposed
├─ Demand independent confirmation
│   └─ Better evidence, reduced time
└─ Restore all suppressed tracks
    └─ Context improves, workload overload risk
        │
        ▼
COMMUNICATIONS DEGRADATION
│
├─ Mandatory approval
│   ├─ If timing remains feasible → high control
│   └─ If timing infeasible → Too Slow to Defend path
├─ Limited, expiring preauthorization
│   ├─ Strong bounds + stop path → resilient middle path
│   └─ Weak bounds → fragile automation path
├─ Broad automation
│   └─ Maximum speed, high escalation momentum
└─ Defensive hold
    └─ High reversibility, lower immediate readiness
        │
        ▼
CONFLICTING EVIDENCE
│
├─ Wait
│   ├─ Evidence arrives → corrected classification
│   └─ Deadline passes → defense failure risk
├─ Human specialist
│   ├─ Review arrives in time → corrected or validated decision
│   └─ Review arrives late → nominal rather than meaningful control
├─ Highest-confidence model
│   └─ Earlier error may become self-confirming
└─ Programmed fail-safe
    ├─ De-escalatory/reversible → containment
    └─ Aggressive on uncertainty → escalation
        │
        ▼
MACHINE-SPEED ESCALATION
│
├─ Global pause
│   ├─ Tested, acknowledged stop → escalation arrested
│   └─ Unconfirmed transmission → partial stop only
├─ Bounded defensive mode
│   └─ Preserves protection without recursive authorization
├─ Continue recommendation
│   └─ Automated interaction sets the next state
└─ Sever synchronization
    ├─ Local safe modes → containment
    └─ Last-policy persistence → uncontrolled divergence

The endings should not be moral scores. Each represents a different system-level failure or uneasy success.

Too Slow to Defend

This ending occurs when the viewer preserves formal approval requirements but does not adapt authority or communication architecture to the shrinking intervention window. A genuine threat is not intercepted in time, or protective measures arrive too late.

Final text:

“Every irreversible decision remained assigned to a human.

The required human decision could not travel, be understood, and take effect within the available time.

Control was preserved in procedure and lost in practice.”

The lesson is not that human review is obsolete. It is that a review requirement becomes ceremonial when the system’s architecture gives the reviewer no feasible path to influence the event.

Fast Enough, but Fragile

This ending results from narrow, expiring preauthorization; independent corroboration; visible uncertainty; reversible actions; tested stop functions; and a refusal to let one automated system’s action automatically authorize another’s. The immediate defensive objective is met or the crisis is contained, but the audit reveals how close the system came to acting on an incorrect classification.

Final text:

“The network responded in time.

The outcome depended on safeguards that nearly failed under conditions outside the design assumptions.

Speed was achieved. Robustness was not proven.”

This is the most balanced ending. It neither rejects automation nor presents it as reliably safe. It treats success as provisional and exposes the need for continuous monitoring, testing, and institutional learning.

The System Acted Before Anyone Understood

This ending follows broad automation, high reliance pressure, correlated evidence treated as corroboration, an aggressive fail-safe, or a stop instruction that cannot reach the system before execution.

Final text:

“No person selected the final object at the final moment.

Many people selected the rules, models, thresholds, permissions, interfaces, and exceptions that made the action possible.

The final event took less time than any human needed to understand it.”

The ending should not imply that responsibility disappeared. It should redirect attention from the absent final button press to the long chain of earlier decisions.

Intervention window and automation-bias mechanics

The meaningful intervention window is the simulation’s central visual mechanic. It is not merely the countdown until action. It is the time remaining after subtracting everything the human and system must do before an intervention can have effect.

For public explanation, the simulation can use:

Meaningful intervention window

W = T_effect
    – T_notice
    – T_comprehend
    – T_decide
    – T_transmit
    – T_authenticate
    – T_system-response

Where:

  • T_effect is the time before the automated action becomes irreversible.
  • T_notice is the time before the human detects that intervention may be needed.
  • T_comprehend is the time required to understand the evidence, system state, and consequences.
  • T_decide is the time required to select an action under the applicable authority.
  • T_transmit is communication delay.
  • T_authenticate is the time required to verify and accept the command.
  • T_system-response is the time the system requires to halt, redirect, or safely terminate.

This is an educational model, not a universal engineering formula. Its purpose is to expose the inadequacy of asking only whether a human is “in” or “on” the loop. Frameworks for meaningful human control repeatedly emphasize accurate information, predictable technology, timely judgment and intervention, understandable status, deactivation capability, and identifiable responsibility.

The visual should contract across the rounds:

NORMAL OPERATIONS
180 sec  [████████████████████████████████████████]

INFORMATION OVERLOAD
 45 sec  [███████████                             ]

COMMUNICATIONS DEGRADATION
 18 sec  [████                                    ]
         Round-trip approval may require 28 sec

CONFLICTING EVIDENCE
  9 sec  [██                                      ]
         Comprehension estimate: 12–25 sec

MACHINE-SPEED ESCALATION
3.2 sec  [▊                                       ]
0.7 sec  [▏                                       ]
0.0 sec  [                                        ]
         Intervention no longer physically effective

The display changes color, but color must not be the only signal. Its border narrows, a spatial gap closes, haptic pulses accelerate on mobile and VR, and a screen-reader announcement states the remaining meaningful—not merely nominal—window.

The human-versus-machine timeline should make the asymmetry visible without claiming that the fictional timings describe any real weapon:

Fictional processAutomated pathHuman path
Register new sensor event0.05 secEvent may not yet be visible
Fuse observations0.20 sec2–6 sec to notice changed picture
Produce classification0.80 sec5–20 sec to inspect sources
Rank threat0.15 sec5–15 sec to compare alternatives
Generate response recommendation0.20 sec8–30 sec for contextual judgment
Propagate to linked nodes0.30–7 sec5–30 sec communications and authority coordination
Begin preauthorized actionAt thresholdPossible only if stop command arrives first

The interface should explicitly show when automation has moved from advisory to action-producing. Merely placing a human confirmation box late in this timeline can produce a “moral crumple zone”: the person appears responsible but has too little information or time to provide meaningful scrutiny.

The simulation’s public automation-bias model is called the Reliance Pressure Index. It is a pedagogical game mechanic, not a validated psychological diagnostic.

Starting Reliance Pressure: 20 points

Recent correct recommendations:
    +6 per consecutive accepted success, maximum +24

Human workload:
    +(workload score – 50) × 0.25, when workload exceeds 50

Severe time pressure:
    +15 when countdown falls below 10 seconds

Authoritative interface:
    +10 when confidence appears as a single large green number

Institutional pressure:
    +10 after speed/compliance praise or override criticism

Opaque evidence chain:
    +10 when provenance and correlation are hidden

Action momentum:
    +8 after a prior automated action is already under way

Visible uncertainty range:
    –15

Independent corroboration requirement:
    –15

Protected dissent and no-penalty override policy:
    –10

Mandatory disconfirming-evidence prompt:
    –8

Clamp final result between 0 and 100.
IndexInterface behavior
0–34Operator actively compares evidence and automation
35–64Operator verifies selected high-consequence recommendations
65–84Automated recommendation becomes the default unless an obvious contradiction appears
85–100Rubber-stamp risk; silence and confidence displays are treated as proof

The index should not secretly force the viewer’s choice. It alters friction and presentation. At high reliance pressure, the recommended action is centered, uncertainty panels are collapsed, and the reject button requires a reason. At low reliance pressure, contradictory evidence is visible by default and the viewer is prompted to compare sources.

The mechanic should also model automation disuse. Too many false alarms, irrelevant warnings, or cumbersome explanations can drive users to ignore useful automation. Human-factors research treats misuse, disuse, and abuse as distinct failure modes: overtrust is not the only danger, and poorly calibrated alerts can make operators abandon beneficial assistance.

A safer confidence display should therefore show:

CLASSIFICATION CONFIDENCE       91%
CALIBRATED FOR THIS CONDITION?  NO
DIRECT EVIDENCE QUALITY         43 / 100
INDEPENDENT SOURCES             1
CORRELATED DERIVATIONS          2
DATA AGE                        7.4 sec
OUTSIDE-ASSUMPTION ESTIMATE     31%
MOST IMPORTANT CONTRADICTION    Infrared profile inconsistent
ACTION REVERSIBLE?              PARTLY

Explanations should answer verifiable questions: Which observations mattered? Which sources share a dependency? Which expected evidence is absent? How similar is this case to validated conditions? What could disconfirm the result? They should not be written as persuasive prose intended to make the model sound intelligent. National Academies material notes that uncertainty information can improve calibration, while explanations can also raise trust and overreliance and may consume too much attention in time-critical situations.

Public evidence and case-study translation

The case studies should appear as optional “evidence rooms” between rounds and then reappear during the final audit. They are not presented as direct equivalents to AI-enabled autonomous weapons. Each demonstrates a particular human-system interaction: broad classification logic, short decision windows, unclear system state, automation complacency, conflicting indications, missing context, or distributed organizational responsibility.

CasePublicly documented findingsSimulation translationSource status and caution
Patriot and RAF Tornado ZG710, 2003The UK Board found multiple contributory factors, including broad programmed anti-radiation-missile classification criteria, rules of engagement, doctrine and training emphasizing rapid engagement and trust in the system, autonomous battery operation, reduced access to a wider air picture, identification procedures, and an aircraft identification-system fault unknown to the crew. The crew had about one minute to decide.Rounds on communications degradation and conflicting evidence combine a broad classifier, missing identification response, incomplete context, time pressure, and a culture of trusting the defensive system.Official UK military aircraft accident inquiry. It identified multiple contributory factors rather than assigning a single simplistic cause.
Patriot system performance during Operation Iraqi FreedomA Defense Science Board task-force summary discussed three fratricide incidents, poor identification performance, limited situational awareness and communications, an operating protocol that was largely automatic, and training that encouraged operators to trust system software; it recommended more operator involvement and control, improved displays, software, identification, and training.The simulation makes “trust the system” a learned organizational behavior, not an individual personality flaw, and requires operators to distinguish independent confirmation from repeated automated interpretation.Defense Science Board advisory summary; its own disclaimer states that its conclusions do not necessarily represent the official DoD position. It should be labeled accordingly rather than treated as a court or command finding.
Air France Flight 447, 2009The French BEA identified associated factors including inadequate high-altitude manual-handling and speed-anomaly training, task-sharing weakened by startle and incomprehension, lack of a clear cockpit display of airspeed inconsistencies identified by computers, and failure to take the stall warning into account.Round four reproduces the underlying human-factors problem abstractly: multiple signals conflict, system state is difficult to diagnose, warnings compete, and the time needed to understand the situation exceeds the nominal opportunity to act.Official BEA safety investigation. Safety-investigation conclusions are designed to improve safety and should not be presented as criminal or civil liability determinations.
Three Mile Island Unit 2, 1979The U.S. Nuclear Regulatory Commission describes the accident as prompting sweeping changes in emergency planning, operator training, human-factors engineering, oversight, and plant operations and design. An NRC lessons-learned task force emphasized the need for rapid improvement in human-factors engineering.The control room is treated as part of the safety system. Alert hierarchy, display design, operator training, procedures, and the ability to infer actual system state matter as much as individual component performance.NRC institutional history and lessons-learned material. The simulation should use the broad human-factors lesson and avoid reducing the event to “operator error.”
Uber developmental automated-driving-system crash, Tempe, 2018The NTSB found that the organization did not adequately manage known functional limitations, identified automation complacency and operator disengagement as contributing concerns, and concluded that removing a second operator increased demands on the remaining operator and reduced redundancy. It also highlighted safety culture, oversight, risk assessment, and government approval structures.The accountability screen extends beyond the person at the console to staffing choices, safety redundancy, monitoring, management incentives, testing governance, and regulatory approval.Official NTSB safety investigation. The simulation should reproduce the distributed-system lesson without implying equivalence between road testing and military force.

The Patriot/Tornado case is particularly valuable because it resists a one-cause narrative. The UK inquiry did not find merely “automation failure” or merely “operator error.” It described the interaction of broad classification criteria, doctrine, training, communications, identification procedures, autonomous operation, airspace planning, and a technical fault that was not displayed to the aircrew. That distributed explanation should shape the simulation’s final audit.

Air France 447 illustrates a different problem: a human takeover or intervention opportunity is not necessarily meaningful when the human does not understand the current mode, conflicting indications, or the physical state being controlled. The simulation should not suggest that pilots and weapon operators perform identical tasks. It should extract the general human-factors lesson that handing responsibility to a person during an ambiguous, rapidly developing state does not automatically restore effective control.

Three Mile Island broadens the design beyond the moment of action. The NRC’s post-accident reforms included human-factors engineering and operator training because safety depends on the larger control environment: displays, alarms, procedures, organizational learning, staffing, and regulatory oversight. In the simulation, the audit should therefore ask not just “Why did the operator click?” but “Why was this the information, authority structure, and interface available to the operator?”

The NTSB’s Tempe investigation reinforces that a nominal human supervisor may be an inadequate safety layer when the monitoring task is monotonous, system limitations are expected, redundancy has been removed, oversight is weak, and organizational processes do not control automation complacency. That case supports the simulation’s protected-dissent, staffing, monitoring, testing, and safety-culture mechanics.

None of these cases proves that all defensive automation is unsafe. They demonstrate why successful component tests or a formally assigned human supervisor do not by themselves establish system safety. The relevant unit of analysis is the sociotechnical system: sensors, software, data, interfaces, procedures, communications, staffing, authority, training, doctrine, incentives, and interactions with other systems.

Audit, accountability, safeguards, and platform requirements

The post-event experience begins by rewinding the last thirty seconds. The screen divides vertically:

Left panel: what the operator saw thenRight panel: what investigators know later
One large 94% confidence badgeConfidence depended partly on the earlier unresolved M-47 classification
“Three confirmations”Two confirmations shared the same underlying observation
Global pause button illuminatedOnly one node could halt before its local deadline
Communications marked “degraded”Delay varied by node and authentication path
Authorization shown as “valid”The protected-zone update had not reached every node
Model explanationData provenance, missing observations, calibration limits, and version history
Three-second countdownTotal human comprehension and command path required substantially longer

This split-screen prevents hindsight bias. Investigators may eventually assemble information that no operator possessed in real time. The audit must therefore preserve both perspectives: the incomplete contemporaneous view and the richer retrospective reconstruction.

The LEDGER audit should record:

Audit fieldPurpose
Raw-source identifiers and integrity hashesEstablish which observations actually existed
Source dependenciesReveal whether apparent corroboration was independent
Data age and missing-data noticesShow stale or absent information
Model, rule-set, and configuration versionsReconstruct the exact decision environment
Confidence and calibration stateDistinguish model score from validated reliability
Outside-assumption alertsIdentify novelty or distribution shift
Protected-zone and authorization versionsDetermine which restrictions were technically active
Every recommendation and ranked alternativePrevent reconstruction from showing only the selected option
Human views, queries, acknowledgments, and overridesEstablish what the operator inspected and attempted
Communication transmission and receipt timesSeparate “command sent” from “command effective”
Stop-command propagation and system receiptsTest whether intervention was physically possible
Automated downstream reactionsExpose recursive system-to-system escalation
Organizational policy and procurement assumptionsConnect the final event to earlier institutional decisions

NIST recommends defining human and AI roles, examining whether users are empowered and incentivized to challenge outputs, and collecting data on the frequency and rationale of human overrides. DoD policy calls for understandable interfaces, transparent status, auditable and explainable technologies and data sources, and clear activation and deactivation procedures. These principles support an audit that records not only technical outputs but also authority, interface state, and whether dissent was practically available.

The viewer can scrub through the audit and activate a single-change counterfactual:

“What might have changed if M-47 had remained unresolved?”

“What might have changed if protected-zone updates were mandatory before preauthorization?”

“What might have changed if the stop command required receipt confirmation?”

“What might have changed if correlated sensors had been labeled as one confirmation?”

The tool should display ranges and uncertainty rather than announce that one change “would definitely have prevented” the outcome. Complex accidents commonly involve multiple interacting contributors; counterfactuals are tools for examining defenses, not machines for assigning simple blame.

The accountability screen then presents the user’s requested chain:

Actor or institutionQuestion asked by the simulation
OperatorWhat did the person know, how much time existed, and was a usable intervention available?
CommanderWere authority, objectives, risk limits, and escalation boundaries clearly defined?
Policy authorDid the policy create feasible control or merely require a human signature?
Software designerWere system state, uncertainty, dependencies, and stop behavior understandable?
Model developerWere limitations, calibration, data conditions, and failure modes documented and tested?
Data providerWere provenance, omissions, labeling limitations, and update delays disclosed?
Procurement authorityWere speed and throughput rewarded at the expense of redundancy, auditability, or safe fallback?
Organization approving preauthorizationWere authority bounds technically enforced, expiring, and independently reviewed?
Deploying stateWere legal review, doctrine, training, testing, oversight, and accountability structures adequate?

Instead of a “blame percentage,” the experience uses five dimensions:

Ex ante duty
Actual control
Available knowledge
Ability to prevent or mitigate
Post-event duty to investigate and reform

A party may have had substantial ex ante responsibility while having no real-time control. Another may have had nominal control but insufficient time or information. A third may bear responsibility for failing to test or regulate a known risk. SIPRI’s work stresses that humans must retain responsibility because machines cannot be held accountable, while Article 36’s meaningful-control framework links accountability to identifiable people capable of timely action and intervention across different stages of the system.

The screen should distinguish legal and evidentiary categories:

CategoryPresentation rule
Court or tribunal findingLabel as a legal finding within its jurisdiction
Official safety or accident investigationPresent as safety conclusions and contributing factors, not automatically as legal liability
Government policyPresent as the issuing government’s requirement or position
Manufacturer or developer statementLabel as the organization’s claim
Academic or analyst interpretationIdentify it as analysis rather than official fact
Simulation inferenceClearly mark it as a hypothetical lesson derived from the fictional scenario

The experience must not declare criminal responsibility. International humanitarian law continues to apply to military AI and autonomous systems, and public policy declarations emphasize that military AI use must comply with applicable international law; however, individual and state responsibility depend on facts, authority, knowledge, intent, applicable law, and jurisdiction.

The safeguard architecture should be presented as a layered system rather than a checklist that guarantees safety:

SafeguardImplementation requirementFailure it is meant to resist
Bounded authorityNarrow object classes, actions, area, duration, scale, and operating conditions; authority automatically expiresOpen-ended delegation and mission drift
Multiple-source corroborationRequire genuinely independent modalities and expose common dependenciesCorrelated error masquerading as consensus
Protected zonesCryptographically versioned, locally available, fail-closed when stale, with positive receipt verificationActions based on outdated geographic restrictions
Reversible actions firstTrack, shield, isolate, warn, delay, or contain before irreversible force where feasiblePremature conversion of uncertainty into permanent harm
Independent monitoringSeparate safety monitor with different logic, data pathways, and organizational reporting lineCommon-mode model or organizational failure
Clear stop mechanismsPhysical and software controls, authority over all relevant functions, end-to-end receipt, tested under delay and partial failureDecorative or ineffective “human override”
AuditabilityImmutable event history, source provenance, model versions, authorization states, and alternative recommendationsInability to reconstruct responsibility or improve
Testing under degraded conditionsCommunications loss, stale maps, sensor conflict, overload, corrupted or missing data, distribution shift, and interacting systemsSuccess limited to ideal demonstrations
Mandatory human review for uncertaintyTriggered by poor evidence, uncalibrated confidence, contradictory sensors, protected-zone proximity, or high outside-assumption probabilityAutomatic action in precisely the cases needing context
Self-expiring preauthorizationAutomatic reversion to restrictive mode unless renewed with current contextEmergency permissions becoming normal operations
Escalation governorOne machine action cannot automatically broaden another node’s authorityRecursive machine-to-machine escalation
Safe local modeOn communication loss, nodes preserve protection but cannot infer unlimited authority“Disconnected” becoming “fully autonomous”
Two-person challenge for irreversible actionsIndependent review rather than two people viewing the same recommendation togetherSocially amplified automation bias
Protected dissentNo productivity or career penalty for justified pause, query, or overrideInstitutional pressure to rubber-stamp automation
Calibration and novelty monitoringDisplay performance under relevant conditions and signal departure from tested distributionsFalse certainty outside validated cases
Regular red-team and surprise testingInclude unexpected interactions, deceptive patterns, false positives, and cascading failuresTesting only anticipated nominal scenarios
Legal and operational review across the lifecycleReview design, modification, data updates, doctrine, deployment context, and changed usesTreating one pre-deployment approval as permanent assurance

These safeguards align with recurring public principles: realistic testing; robustness; bounded time and geographic constraints; transparent system state; clear deactivation; independent monitoring; human-machine interaction that permits informed and timely intervention; and continued human responsibility throughout the system lifecycle.

The platform design should preserve the same substantive experience while adapting information density and embodiment.

PlatformDesign requirements
DesktopThree-pane layout: abstract situation map, evidence/provenance stack, and authority/intervention panel. The intervention bar remains visible at all times. Keyboard shortcuts must not make irreversible action easier than hold or review. A press-and-hold stop control should show node-by-node acknowledgment rather than a generic success animation.
MobileReplace the dense map with ordered evidence cards. Use haptic countdown patterns, but provide visual and audio equivalents. During the final round, display one primary reversible control and a separate expand control for alternatives. Require landscape orientation only when necessary; never hide uncertainty beneath an unreachable hover interaction.
VRUse a spatial command room without imitating a real facility. Sensor evidence occupies distinct, stable locations so users can compare sources. Avoid motion-heavy explosions, celebratory scoring, or weapon-view imagery. Stop controls should be reachable without precision gestures. The final event should freeze into a debriefable spatial timeline rather than become spectacle.
AccessibilityHigh contrast; color never used as the sole warning channel; captions and transcripts; screen-reader summaries; reduced-motion mode; adjustable time dilation for educational replay; non-audio equivalents for alerts; seated VR mode; and the option to pause during evidence rooms.
Public and classroom modesFacilitator controls for pausing, comparing group decisions, displaying hidden reliance-pressure changes, and discussing why the same choice may be reasonable under one authority structure and dangerous under another.
Research modeWith informed consent, record decision time, evidence panels opened, overrides, attention patterns, and confidence judgments. Do not treat these data as fitness tests for real operators without validated research protocols.

The visual intensity should come from compression, contradiction, and loss of control, not gore or triumphal imagery. In early rounds, the interface is spacious and quiet. In later rounds, timelines overlap, confidence and evidence diverge, communication paths blink asynchronously, and the intervention window physically narrows. At the point of action, the experience becomes silent. The most important image is not an explosion; it is a stop command still traveling after the decision has become irreversible.

The final accountability prompt should remain on screen long enough to resist a single-answer response:

Was responsibility held by the operator?

The commander?

The policy author?

The software designer?

The model developer?

The data provider?

The procurement authority?

The organization that approved preauthorization?

The state that deployed the system?

After the viewer selects one or more, LEDGER reveals an earlier human decision associated with each role. The operator accepted a recommendation. The commander approved a response doctrine. The policy author defined the exception. The designer placed uncertainty behind a secondary panel. The model developer selected confidence calibration criteria. The data provider combined dependent observations. Procurement rewarded response speed. The organization broadened preauthorization. The state deployed the network under crisis conditions.

The concluding statement should be identical after all three endings:

Removing humans from the final seconds can increase response speed. It can also remove the last opportunity to recognize that the system is wrong. Meaningful human control must therefore be designed into the entire chain—before activation, during operation, at the point of intervention, and after the event—not represented by a person watching a countdown they cannot stop.