Designing an AI Simulation for Real-Time Evasion and Goal-Directed Navigation

Full report

On this page

Executive assessment

The supplied interface establishes a strong mission-control visual language: a central three-dimensional scene, mission progress, vehicle telemetry, supervisory controls, and lost-link behavior. What it does not yet communicate clearly is the autonomous decision process. A viewer can see that the system is operating, but cannot readily determine what it has detected, what it predicts will happen, which actions it considered, why it selected its current maneuver, how much safety margin remains, or whether it can still reach the target.

A compelling autonomy simulation should therefore depict six continuously connected activities:

Perceive → track → predict → generate alternatives → select a safe action → verify the result.

The most important design principle is to separate three representations of the scene:

This distinction is essential because realistic sensors do not observe the environment perfectly. Gazebo, for example, supports configurable sensor and noise models precisely because idealized measurements can make perception appear unrealistically reliable; its sensor framework supports Gaussian or custom noise on simulated streams.

The central visualization should consequently be more than an animation of a vehicle avoiding a box. It should be an instrumented demonstration of closed-loop autonomy under uncertainty, showing both successful maneuvers and the reasoning-relevant evidence behind them. This aligns with NIST’s characterization of trustworthy AI as not only safe and reliable, but also accountable, transparent, explainable, and interpretable.

Information the simulation should expose

The mission geometry

At all times, the main view should show the autonomous vehicle, the target, the nominal route, the route currently being executed, and the surrounding operating boundaries. In the provided design, the target or detected object is visually prominent, but the relationship among the vehicle, the destination, and the intervening hazard is not yet obvious.

The display should include:

For a three-dimensional craft, the safety envelope should be a volume rather than merely a line. It should account for the vehicle’s physical footprint, velocity, maximum acceleration, turn rate, braking capability, and required clearance. A vehicle moving quickly toward an obstacle may be unsafe even when its current geometric separation looks large.

Detected-object state

Every relevant object should receive a persistent track identifier such as OBJ-017, with the display showing:

• Observed position and last observation time.
• Estimated velocity and acceleration.
• Object classification, when classification affects policy.
• Detection or track confidence.
• Predicted future positions.
• Prediction uncertainty.
• Whether the object is considered cooperative, uncooperative, static, dynamic, or unknown.
• Whether it currently intersects the vehicle’s predicted corridor.

“Unexpected” should not merely mean that an object suddenly appears in the rendered world. It should mean that the object creates a discrepancy between the prior plan and the agent’s current belief. Examples include an object emerging from occlusion, entering at an unpredicted velocity, being detected later than expected, changing direction, or being only partially observed.

The object’s predicted occupancy should be visualized as a time-indexed tube, cone, or sequence of translucent volumes. A deterministic line gives a misleading impression of certainty. Research on dynamic-obstacle planning increasingly represents future obstacle states probabilistically or uses multiple motion hypotheses because an agent’s future path cannot always be reduced to one forecast.

Imminent risk

A small set of safety quantities should remain visible without requiring the operator to inspect a secondary analytics screen:

Time-to-conflict should be presented as both a number and a spatial cue. For example, the projected collision point can be marked in the scene, while a countdown appears adjacent to it. The interface should distinguish between:

• Potential conflict: current predictions overlap, but substantial maneuvering margin remains.
• Avoidance required: the nominal route is no longer acceptably safe.
• Emergency action: ordinary replanning cannot maintain the required safety margin.
• Unavoidable under current assumptions: no valid maneuver has been found within the reachable control set.

A useful internal and displayed margin is:

\[ M_{\text{reaction}} = T_{\text{conflict}} - T_{\text{sense}} - T_{\text{fusion}} - T_{\text{planning}} - T_{\text{command}} - T_{\text{actuation}} - T_{\text{physical response}} \]

A positive but shrinking margin means the autonomy stack is still functioning but losing options. A negative margin indicates that, under its current estimates, timely avoidance is no longer physically achievable.

Candidate actions and selection

The viewer should be able to see that the AI made a choice rather than merely following a pre-authored animation. Display the best few dynamically feasible alternatives, not every sampled trajectory.

A candidate-action panel might show:

The display should explain rejection using verifiable factors:

Selected left-and-up deviation. OBJ-017 entered the nominal corridor. Predicted time to conflict is 2.3 seconds. The selected trajectory maintains 12.4 meters of separation and adds 1.8 seconds to arrival. Right deviation violates the boundary constraint; braking alone does not preserve the required clearance.

This is more useful than exposing raw neural activations, an unstructured model monologue, or an alleged “chain of thought.” The appropriate objective is an auditable decision explanation derived from logged inputs, constraints, scores, and state transitions. NIST’s AI Risk Management Framework emphasizes access to information needed for transparency, the definition of acceptable performance limits, and course-correction mechanisms when performance exceeds those limits.

Autonomy and decision-making architecture

A hybrid planning stack

For this type of simulation, the strongest architecture is usually not one monolithic “AI brain.” A layered system makes the behavior easier to validate, debug, and explain:

\[ \text{Sensors} \rightarrow \text{Perception} \rightarrow \text{Tracking} \rightarrow \text{Motion prediction} \rightarrow \text{Global planning} \rightarrow \text{Local trajectory optimization} \rightarrow \text{Safety filter} \rightarrow \text{Control} \]

A modular driving-policy study describes a central advantage of modular autonomy: components can be developed and analyzed separately, though designers must also manage errors that accumulate across component boundaries.

The recommended division of responsibility is:

Global replanning

When a new obstacle invalidates a route, the system needs an incremental or sampling-based global planner.

D* Lite was developed for goal-directed navigation in partially known terrain and reuses information from previous searches instead of planning every changed route entirely from scratch.

RRT and related sampling-based planners are useful in high-dimensional or geometrically complex spaces. RRT was introduced as an asymptotically optimal variant whose solution cost approaches the optimum as sampling continues, subject to its assumptions.

For the simulation display, global replanning should be visible as:

• The invalidated portion of the old route.
• The newly blocked region.
• The updated route or safe corridor.
• The increase in route cost or arrival time.
• A planning-status indicator such as SEARCHING, FEASIBLE, IMPROVING, or NO ROUTE.

Local avoidance

A local planner should react at a faster cadence than the global planner and respect the vehicle’s actual dynamics.

The Dynamic Window Approach searches directly over reachable velocity commands and evaluates short trajectories under velocity and acceleration limits. Its original formulation was explicitly derived from robot motion dynamics for reactive collision avoidance.

Optimal Reciprocal Collision Avoidance is particularly relevant when several autonomous agents are expected to share responsibility for avoiding one another. ORCA converts pairwise reciprocal avoidance into low-dimensional linear constraints; the original work demonstrated smooth collision-free actions in dense multi-agent simulations.

Model Predictive Control is preferable when the simulation needs to show a predicted horizon, explicit vehicle dynamics, competing objectives, and constraints. MPC repeatedly optimizes a finite-horizon trajectory and then replans as new observations arrive. Recent work combines MPC with dynamic-obstacle prediction and safety constraints to improve behavior in uncertain environments.

A pragmatic implementation could use:

• **D* Lite or RRT*** for route-level replanning.
• MPC for dynamically feasible short-horizon choices.
• ORCA constraints when other agents are assumed to cooperate.
• A reactive fallback such as braking, hovering, or holding when prediction quality becomes inadequate.

Independent safety enforcement

The system should maintain a safety layer that can override the nominal AI decision. Control Barrier Functions are one established technique for encoding safety conditions as constraints on control inputs. They can be combined with performance objectives through quadratic programming, but any resulting guarantees remain dependent on the correctness of the model, state estimates, constraint definitions, and feasibility assumptions.

The interface should explicitly show when the safety layer intervenes:

Safety filter active: nominal turn command reduced because lateral acceleration would exceed the safe reachable set.

This distinction lets evaluators determine whether avoidance was produced by the primary planner or rescued by the last-line safety controller. It also prevents a system with a poor nominal policy from appearing successful merely because emergency overrides repeatedly save it.

Degraded autonomy and human supervision

The existing CONTINUE, HOLD, RETURN TO SHIP, and ABORT MISSION controls are appropriate high-level supervisory actions. They should be augmented with a visible autonomy state:

NOMINAL → AVOIDING → RECOVERING → DEGRADED → MINIMUM-RISK MANEUVER → HUMAN CONTROL

The operator should not be expected to manually fly the vehicle immediately after takeover unless the interface, communications link, and operating concept truly support that action. Instead, takeover should normally select a bounded supervisory mode: approve a route, command a hold, choose among alternatives, or authorize a return.

FAA human-factors studies of detect-and-avoid interfaces have specifically examined unexpected traffic encounters and the effect of alert location within or outside the operator’s primary field of view. The implication for this interface is that urgent collision information must appear directly in the main scene, not exclusively in the right-hand control rail.

Scenarios that demonstrate unexpected-object handling

A high-quality demo should not consist of one predetermined obstacle crossing. It should contain a graduated scenario library in which the disturbance affects perception, prediction, planning, control, or communications in different ways.

ISO 34502 provides a scenario-based safety-evaluation framework for automated driving, while ISO 21448 addresses risks arising from functional insufficiencies where situational awareness depends on complex sensors and processing. Although written for road vehicles, the underlying ideas—scenario coverage, foreseeable operating conditions, perception limitations, and evidence-based validation—transfer well to other autonomous mobile systems.

Core demonstration scenarios

Adversarial and rare-event generation

Hand-authored scenarios are necessary for acceptance testing, but they are insufficient for finding subtle controller weaknesses. Scenario variables should also be randomized across:

• Obstacle size, shape, reflectivity, and classification.
• Spawn position and initial velocity.
• Acceleration, turn rate, and intent changes.
• Occlusion timing.
• Lighting, weather, surface, or background conditions.
• Sensor noise, delay, dropout, bias, and calibration error.
• Vehicle mass, actuator lag, braking performance, and control disturbance.
• Map error and localization drift.
• Communications latency and loss.
• Number and behavior of interacting agents.

Domain randomization has been used to increase variation in simulated geometry and appearance so learned systems encounter a broader distribution than a single fixed simulation.

Rare safety failures are difficult to discover by naive random testing because the relevant event distribution is sparse. Adaptive stress-testing and learned scenario-generation methods instead search for trajectories and parameter combinations that cause or nearly cause failure. Research systems have demonstrated more efficient discovery of safety-critical cases than simple grid search or manually selected parameter sets.

The simulation should keep two test suites separate:

1. Realism suite: events sampled from credible operating distributions.
2. Stress suite: deliberately difficult, low-probability, boundary-seeking cases.

Passing the stress suite does not demonstrate real-world frequency, while passing only the realism suite does not establish adequate corner-case robustness.

Recommended interface redesign

Main three-dimensional viewport

The central viewport should become an “autonomy evidence view.” The proposed composition is:

┌──────────────────────────────────────────────────────────────────────┐
│ MODE: AVOIDING     CONFIDENCE: 0.87      REACTION MARGIN: 1.4 s     │
│                                                                      │
│     Predicted obstacle occupancy                                     │
│          ░░░░░░                                                      │
│        ░ OBJ-017 ░                TARGET                              │
│          ░░░░░░                     ◉                                 │
│             ↘ predicted motion       · · · recovered route            │
│                                                                      │
│ OWN VEHICLE ▶════ active trajectory ═══╮                              │
│      \                                 ╰══════►                       │
│       \ rejected candidate                                             │
│                                                                      │
│ Sensor boundary     Safety envelope     Projected closest approach    │
└──────────────────────────────────────────────────────────────────────┘

The viewport should support three display modes:

World truth should be hidden by default in the operator view because a real operator would not possess omniscient knowledge. In engineer and replay modes, truth can be shown with a distinct rendering so missed detections and estimation errors are visible.

Decision card

The current right rail should include a continuously updated decision card above the supervisory controls:

CURRENT DECISION
Maneuver: Left deviation + speed reduction
Cause: OBJ-017 entered planned corridor
Time to conflict: 2.3 s
Minimum predicted separation: 12.4 m
Goal impact: +1.8 s ETA
Planner confidence: 0.87
Safety filter: Active — acceleration limited
Next reassessment: 50 ms

The card should use stable categories rather than free-form text generated independently on every frame. Recommended cause codes include:

• NEW_OBSTACLE
• MOTION_CHANGED
• PREDICTION_UNCERTAIN
• ROUTE_BLOCKED
• SAFETY_MARGIN_LOW
• SENSOR_DEGRADED
• CONTROL_LIMIT
• LINK_LOSS
• NO_FEASIBLE_TRAJECTORY

This makes the explanation auditable and allows analytics to group decisions across thousands of runs.

Event timeline

The existing mission timeline should be complemented by a detailed event strip:

12.400  Object first visible
12.450  Detection created: OBJ-017, confidence 0.62
12.550  Track confirmed, confidence 0.84
12.600  Nominal path predicted unsafe
12.630  Four candidate trajectories generated
12.672  Candidate 3 selected
12.690  Safety filter modified acceleration
12.710  Control command applied
13.280  Minimum separation reached
14.100  Nominal route reacquired

Every row should be clickable, seeking the replay to the corresponding frame and displaying the world state, AI belief, planner input, candidate set, selected action, and applied command.

CARLA’s ScenarioRunner metrics tooling illustrates the value of recording simulation state and calculating metrics after execution without rerunning the scenario; its recorder can expose transforms, velocities, accelerations, controls, and other state needed for post-run analysis.

Supervisory-control behavior

The supervisory buttons should reflect when an action is safe or available:

A command should display its expected consequence before execution whenever time allows. During imminent collision risk, the autonomous safety layer should not wait for human confirmation.

Alert hierarchy

Alerts should combine location, symbol, text, and timing rather than color alone:

A “confidence” value should not be displayed without naming what it describes. Detection confidence, track confidence, motion-prediction confidence, route-feasibility confidence, and overall mission confidence are different quantities and should not be collapsed into one unsupported score. Research on uncertainty visualization in robot-assisted decision-making shows that the form and context of confidence displays can affect how people use automation advice; therefore, uncertainty should be actionable and clearly scoped.

Evaluation metrics and test methodology

Safety metrics

The primary safety measures should be computed from simulator ground truth, with parallel measures computed from the AI’s estimated state.

A collision-free result alone is not sufficient. A system that narrowly misses obstacles, oscillates, repeatedly applies emergency braking, or succeeds only because another agent compensates should not receive the same score as a stable, early, high-clearance avoidance maneuver.

NIST’s robotics measurement program emphasizes repeatable metrics, datasets, protocols, and test methods for attributes such as navigation and obstacle avoidance. NIST work on autonomous guided vehicles also distinguishes merely stopping for an obstacle from recognizing it, estimating its characteristics, and regenerating a route around it.

Mission-performance metrics

Safety and target attainment must be reported together. A controller that always stops indefinitely may score well on collision avoidance but fails the mission objective.

Perception and prediction metrics

A valuable visualization is a side-by-side comparison of the object’s actual trajectory, predicted mean trajectory, and uncertainty envelope. This reveals whether avoidance succeeded because the predictor was accurate, because the planner was conservative, or by coincidence.

Real-time computation metrics

Average latency can conceal dangerous outliers, so percentile and maximum values should be reported. A plan delivered after the physical decision deadline is a failure even if its computed trajectory is excellent.

Control-quality metrics

ORCA’s original formulation emphasizes smooth reciprocal motion, while differential-drive extensions explicitly address smoothness under vehicle constraints. These sources reinforce the need to assess not only collision avoidance but also whether the resulting motion is executable and non-oscillatory.

Robustness and human-oversight metrics

The same scenario should be repeated across random seeds and parameter variations. Report distributions over obstacle behavior, sensor quality, dynamics, and environmental conditions rather than a single curated run.

Human-supervision measures should include:

The DECISIVE test-method program for small unmanned aircraft includes obstacle avoidance, navigation, autonomy, communications, interface, trust, and situation awareness as distinct test concerns, illustrating why a complete evaluation must cover both vehicle performance and the human-autonomy team.

Implementation blueprint and acceptance criteria

Instrumentation-first implementation

The simulation should be built around a common timestamped event model before visual polish is added. Each cycle should record:

{
  "simulation_time": 12.672,
  "autonomy_mode": "AVOIDING",
  "vehicle_state": {
    "position": [42.1, 18.4, 74.8],
    "velocity": [6.4, -0.8, 0.3],
    "heading": 98.9
  },
  "active_tracks": ["OBJ-017", "OBJ-021"],
  "primary_hazard": "OBJ-017",
  "time_to_conflict_seconds": 2.3,
  "candidate_trajectories": [
    {
      "id": "TRAJ-001",
      "feasible": true,
      "risk": 0.04,
      "goal_delay_seconds": 1.8,
      "minimum_clearance_meters": 12.4
    }
  ],
  "selected_trajectory": "TRAJ-001",
  "selection_reason_codes": [
    "LOWEST_ACCEPTABLE_RISK",
    "GOAL_REACHABLE",
    "CONTROL_LIMITS_SATISFIED"
  ],
  "safety_filter_intervened": true,
  "applied_command": {
    "throttle": 0.31,
    "yaw_rate": -0.24,
    "vertical_rate": 0.12
  }
}

Ground truth, sensor observations, estimated state, planner input, planner output, safety-filter output, and applied control should be logged separately. Otherwise, a later replay cannot identify where a bad outcome originated.

Development sequence

The first increment should implement world truth, the vehicle, target, nominal route, one unexpected obstacle, and deterministic replay. The second should add noisy sensing, track estimation, prediction uncertainty, and visualization of the AI’s belief. The third should add multiple candidate trajectories, explicit scoring, and a safety override. The fourth should add scenario randomization, batch evaluation, and percentile metrics. The final increment should add human supervisory controls and formal usability testing.

Gazebo provides actors, sensor models, noise, and plugin-based environment control, making it appropriate for general robotic prototypes. CARLA provides detailed vehicle actors, collision events, recording, and scenario metrics for road-vehicle use cases. The choice should be made from the required vehicle dynamics and operating environment rather than visual quality alone.

Minimum acceptance criteria

A credible initial demonstration should satisfy all of the following:

The resulting experience should make one story unmistakable: an unforeseen object changed the AI’s understanding of the environment; the system predicted a conflict, evaluated physically possible alternatives, selected the safest acceptable maneuver, executed it within the available reaction margin, and then resumed progress toward the target. That story—not merely the visual spectacle of a moving vehicle—is the evidence that the simulation demonstrates real-time autonomous decision-making.