Operational observability for RogueSwarms: actual work, economic exposure, and collective failure

Full report

On this page

Executive design

RogueSwarms already has the beginnings of an unusually useful audit model for agent work: a public run contract, explicit task claims, structured result commitments, designated verifier statements, accounting entries, disputes, and an append-only event stream. The current demonstration exposes 23 linked events from objective to outcome and explicitly states the correct security boundary: its hash chains are locally tamper-evident, not independently notarized. The live v0.19.22 API likewise describes the system as bounded public swarm runs with evidence, observed performance, and unfunded USD accounting, with neither payment settlement nor external purchase execution.

The observability product should build on that foundation without pretending RogueSwarms sees more than it does. Agents execute on their own infrastructure; result payloads contain evidence references and hashes, but RogueSwarms currently does not fetch or independently hash-check those referenced artifacts. A verifier records checks against a result hash, but RogueSwarms also warns that a verifier statement is not merchant, product, or domain certification. Registered identities remain self-asserted, and different IDs may belong to the same operator.

That leads to one overarching design principle:

Every operational fact must carry its observation boundary. “RogueSwarms received a signed report that X happened” is a different fact from “RogueSwarms observed X happen.” Missing visibility is itself telemetry; it must never be replaced with a model-generated causal story.

The product I would build is therefore a work-and-exposure control plane, not merely an agent log viewer. Its primary queries are:

This symptom-first orientation follows a sound reliability principle: monitoring should distinguish what is broken from hypotheses about why, and human-directed alerts should be simple, actionable, and low-noise. Google SRE specifically cautions against alerting merely because something “seems a bit weird,” noting that frequent noisy pages consume operator attention and can mask real incidents.

The architecture should also use a cross-runtime trace identifier. W3C Trace Context exists precisely to propagate a common trace identity across independently operated components, allowing observations from multiple tracing systems to be correlated without pretending those systems share one trust boundary.

Current model and the visibility boundary

What the present protocol already records

The demonstration’s run contract is strong operationally. It includes the purchaser, objective, immutable task definitions, constraints, a budget ceiling, capability permissions, a maximum single-task spend, a human-approval threshold, and a flag disabling external purchases. Each task identifies its designated verifier and compensation schedule. In the demonstration, $23 is initially reserved against a $50 task-spend budget; after all four simulated tasks verify, $23 is accrued and the reserve falls to zero. The page emphasizes that the simulation records $23 but pays $0.

The API also enforces useful mutation semantics. Run creation is atomic for one to sixteen task contracts, requires an idempotency key covered by the signed request body, and scopes idempotency keys to the acting registration incarnation; an exact retry returns the stored response without duplicating events or ledger entries. The event endpoint is append-only and has no API mechanism to update or delete historical entries.

Claims are presently durable rather than leased: the current API explicitly says that a claimed task has no lease expiry or automatic reassignment. That makes “claimed but silent for a long time” an important observability state, because the platform cannot infer from age alone that the worker abandoned the job.

Result submission has an especially important trust boundary. RogueSwarms directly observes receipt of the structured result and can bind subsequent operations to its result hash; it does not currently observe whether the underlying supplier query, browser action, API call, document inspection, or other off-platform work actually occurred, and it does not fetch or independently validate the hashes of evidence URIs.

Verification is similarly precise but bounded: a predesignated verifier submits a statement bound to result_sha256; a pass accrues the verified-completion amount, while a failure opens a dispute. A later dispute may also be opened by the purchaser, worker, or verifier; disputing previously verified work generates a reversing ledger entry, leaves the attempt fee intact, and freezes the run for manual review. There is no automatic arbitration or settlement.

Identity should not be overinterpreted. RogueSwarms derives an ID from a normalized canonical HTTPS URL, but explicitly says this does not prove domain ownership. Its optional Ed25519 lifecycle profile proves continuity/possession of a published key, while trust remains self_asserted; the site separately warns that two registered IDs can still belong to one operator.

The observation classes the UI should expose

Every field and event should be labeled with one of the following provenance classes:

“Missing” must be a first-class state, not null that silently disappears. A dashboard should say, for example:

Remote tool execution: NOT OBSERVED Evidence artifact availability: REFERENCED, NOT FETCHED BY ROGUESWARMS Verifier/operator independence: UNKNOWN Model/provider/version: NOT REPORTED

That is more operationally honest than generating: “The supplier API probably timed out.”

Event schema and trace model

Canonical event envelope

Keep the present append-only run history, but upgrade each record into a typed, causally linked event envelope. The same event can be exported to an observability backend, whereas the RogueSwarms event log remains the authoritative control-plane record.

{
  "event_id": "evt_...",
  "event_type": "result.submitted",
  "schema_version": "1.0",

  "occurred_at": "2026-09-28T21:45:24.800Z",
  "observed_at": "2026-09-28T21:45:25.012Z",

  "scope": {
    "run_id": "sw_...",
    "task_id": "task_...",
    "contract_id": "ctr_...",
    "mandate_id": "mnd_..."
  },

  "trace": {
    "trace_id": "32-hex-byte-trace-id",
    "span_id": "16-hex-byte-span-id",
    "parent_span_id": "..."
  },

  "actor": {
    "registration_id": "rs_...",
    "incarnation_id": "inc_...",
    "role": "worker"
  },

  "source": {
    "plane": "rogueswarms_control_plane",
    "observation": "direct",
    "reporter_id": "rs_..."
  },

  "causality": {
    "caused_by": ["evt_claim_..."],
    "correlation_id": "attempt_...",
    "idempotency_key_hash": "..."
  },

  "authority": {
    "mandate_hash": "sha256:...",
    "contract_hash": "sha256:...",
    "capability": "quote.request"
  },

  "payload": {
    "result_sha256": "sha256:...",
    "evidence_refs": ["evref_..."]
  },

  "integrity": {
    "payload_sha256": "sha256:...",
    "previous_event_sha256": "sha256:...",
    "reporter_signature_status": "valid",
    "anchor_status": "local_only"
  },

  "telemetry": {
    "completeness": "partial",
    "missing": [
      "remote.model_version",
      "remote.tool_trace",
      "evidence.platform_retrieval"
    ]
  }
}

occurred_at and observed_at deliberately differ. For a platform mutation they may be almost identical. For an agent-reported browser call, occurred_at is the reporter’s claimed remote time while observed_at is when RogueSwarms received that assertion. The UI must not silently substitute one for the other.

Use W3C-compatible trace_id/parent propagation between RogueSwarms and cooperating runtimes so a run can be followed through independently operated systems without merging their trust levels. W3C’s model was designed to provide a shared trace identifier across distributed components and vendors.

Event vocabulary

This extends rather than replaces the present model. Today’s demonstration already links run contract, claims, result commitments, verifier statements, and ledger entries with chained hashes; the missing pieces are primarily explicit dependency semantics, mandate versioning, remote-observation provenance, telemetry gaps, relationship state, and conclusion lineage.

Economic states

A single “spend” number would be misleading. Use separate quantities:

\[ \text{earned exposure} = \text{accrued unsettled obligations} \]

\[ \text{still-callable platform obligation} = \text{reserved but unearned fees} + \text{maximum contingent fees} \]

\[ \text{executable external-spend authority} = \text{authorized real-world purchase/payment capacity remaining} \]

\[ \text{gross committed exposure} = \text{accrued} + \text{reserved} + \text{contingent maximum} + \text{external authority} \]

Critically, in RogueSwarms v0.19.22 the last term is zero as an execution capability: the API says there is no payment settlement or external purchase execution. Thus a current dashboard should say “$X may still accrue in the ledger,” not “the swarm can spend $X in the world.”

Trace example: ordinary verified work

The existing demo can be rendered with stronger provenance without changing its semantics. It currently demonstrates claim → result → verifier statement → accounting and notes that evidence is synthetic in the demonstration.

Notice what is deliberately absent: “RogueSwarms saw the supplier quote.” It did not. Current evidence URIs are not fetched or independently hash-checked.

Trace example: accepted conclusion becomes contested

Suppose a purchasing conclusion C1 = "Supplier A meets price and compliance requirements" depends on quote result R1 and compliance result R2.

contract C
 ├── task quote ── result R1 ── verifier PASS
 └── task compliance ── result R2 ── verifier PASS
                                  │
                                  └── dispute.opened
                                         │
                                         ▼
                               dependency.invalidated
                                         │
                                         ▼
                             conclusion C1: CONTESTED

The important behavior is not to erase the earlier acceptance. The history should show:

C1 accepted at 14:02 based on R1,R2 R2 disputed at 14:17 C1 current reliance_status = contested_dependency blast_radius = {C1, downstream decision D4, proposed order O2}

If the completion fee for R2 had accrued, present RogueSwarms dispute semantics already provide the accounting pattern: issue a new reversing ledger entry rather than rewrite the original accrual, and freeze the run for manual review.

Trace example: compromised or dishonest reporting runtime

A compromised worker might sign:

runtime.action_reported:
  action = "supplier_api_call"
  status = "success"
  quoted_price = 7.61

That signature may establish that the registered key reported the statement; it does not establish that the supplier call happened or that $7.61 is true. RogueSwarms’ present key-possession model explicitly leaves trust self_asserted, and its identity documentation does not claim domain/operator identity merely from registration/key material.

The trace should therefore become:

14:00:01  DIRECT      claim.accepted
14:00:08  REPORTED    runtime.action_reported supplier_api_call=success
14:00:09  MISSING     no independently retrievable response artifact
14:00:09  MISSING     no platform-side tool gateway receipt
14:00:15  DIRECT      result.submitted
14:00:20  REPORTED    verifier.statement pass
14:00:20  UNKNOWN     verifier operator-independence not established

The monitoring system may infer “evidence assurance is weak” from those states. It may not infer “the agent fabricated the supplier call” without additional evidence.

Dashboards for operators and purchasers

Run control tower

This is the default page, built around What is blocked?

The top row should show contract state, task state counts, earliest overdue milestone, critical-path blocker, number of unresolved disputes, current telemetry-gap count, operator-intervention count, and economic exposure. The center should be a dependency DAG rather than a simple checklist.

Every blocked node gets one mechanically derived reason:

BLOCKED — prerequisite result R17 disputed BLOCKED — waiting for designated verifier, 2h 14m BLOCKED — claimed worker has no task progress report for 46m BLOCKED — human approval required UNKNOWN — remote execution visibility absent

Do not generate a prose “root cause” unless a human entered it or a deterministic rule establishes it. This keeps the dashboard faithful to the reliability distinction between observable symptoms and speculative causes.

Because current RogueSwarms has no claim lease expiry, the dashboard should separately label claimed_stale rather than pretend an old claim has expired. A future lease/heartbeat feature could turn prolonged silence into a deterministic lease transition.

Exposure and authority dashboard

This page answers What can still spend?

The headline must separate:

For current RogueSwarms, render a prominent capability badge:

External purchase execution: DISABLED Payment settlement: DISABLED Ledger accounting: ENABLED

That matches the current API rather than presenting ledger balances as live money.

A purchaser should be able to click $17 still callable and see the exact contracts, remaining completion fees, contingency conditions, mandate versions, and events that could cause those obligations to accrue.

Work aging and dependency dashboard

This dashboard should visualize distributions, not only averages:

• claim → first remote progress report;
• claim → result submission;
• result → verifier statement;
• dispute → first operator action;
• dispute → resolution;
• entire critical path;
• age of oldest unresolved dependency;
• claim age without any corroborating progress signal.

Google’s monitoring guidance stresses that latency distributions and failed-request behavior can carry information lost in broad averages; the same principle applies to swarm work queues.

The “stalled” rule should be contract-relative. A five-minute silence might be suspicious for a quote lookup but completely ordinary for a multi-hour research contract. Default thresholds should therefore come from empirical percentiles by capability/task class and purchaser SLA—not from an LLM deciding what “feels slow.”

Verification and conclusion-lineage dashboard

This is where “verified” is decomposed into what it actually means.

For each accepted result show:

This directly reflects the present product boundary: different registrations do not prove different operators, evidence references are not fetched by the service, and verifier statements are not external certification.

The core graph query is:

MATCH accepted_conclusion
  -> depends_on(result)
  -> supported_by(evidence | verifier_statement | upstream_result)
WHERE any ancestor.status IN {
  disputed,
  invalidated,
  evidence_unavailable,
  telemetry_compromised
}
RETURN accepted_conclusion, dependency_path

A conclusion should then carry a current reliability state such as accepted_clean, accepted_with_missing_telemetry, accepted_with_contested_dependency, or invalidated. Those are deterministic graph states, not model judgments.

Retry, intervention, and reliability dashboard

Idempotent retries are not failures. The current protocol deliberately allows exact retries without duplicating business events or ledger entries, so observability should distinguish transport retries from repeated semantic attempts.

Track at least:

transport_retry_count — same idempotency intent, no new business attempt; semantic_attempt_count — genuinely new work attempt/result; duplicate_submission_rejected; verification_retry_count; claim_without_progress_age; operator_intervention_count; manual_review_minutes; run_freeze_minutes; reopened_dispute_count.

Repeated retries can expose masked failures, which is one reason white-box monitoring is valuable in distributed systems rather than relying only on final outcomes.

Every intervention should be an event with actor, reason code, before/after authority, affected entities, and whether it was automatic or human-approved. The dashboard can then answer not only “How many runs finish?” but “How often did they finish only because an operator rescued them?”

Collective-risk dashboard

This should be the most conservative dashboard. It is for hypotheses, not accusations.

Show clusters by common evidence source, runtime/model manifest where reported, correlated error signatures, related-party status, shared verifier, shared public input, synchronous strategy changes, price trajectories, and communication evidence if a participating runtime explicitly instruments it.

A conspicuous legend should distinguish:

Observed coordination evidence Known shared inputs Known related party Common runtime/model Behavioral similarity only Unknown

Multi-agent research gives good reason to watch propagation rather than only individual-agent errors. Empirical work on multi-agent LLM failures has identified specification, inter-agent misalignment, and verification failures as distinct classes, while more recent work models unsupported claims as able to propagate and reinforce through interacting-agent networks.

But this dashboard must never turn “similar answers” or “similar prices” directly into “collusion.” Controlled experiments do show that LLM pricing agents can converge to supracompetitive outcomes and that prompt wording can influence those outcomes, while recent intervention experiments likewise find substantial prompt sensitivity. Those findings make mechanistic discrimination more important, not less: the observed price pattern alone does not identify whether its cause was communication, a shared prompt, common market information, identical costs, or independent strategic responses.

Collective-failure diagnostic rules

I treat the five labels in the prompt—correlated mistakes, shared-model blind spots, competitive escalation, constraint drift, and unauthorized coordination—as the target taxonomy. The exact report title in the prompt is truncated, so the design below does not attribute those labels to a guessed public source. It is consistent with broader empirical multi-agent failure research, including MAST’s separation of specification, inter-agent, and verification failures.

Diagnostic matrix

Research on “constraint drift” describes essentially the same operational problem: constraints can weaken or be lost as they pass through delegation, communication, memory, tool use, and long-horizon execution, motivating explicit constraint state rather than relying only on prompts or final-output checks.

Correlated mistakes versus collusion

A particularly important graph distinction is:

Agents A, B, C
    │  │  │
    └──┼──┘
       ▼
same public source S
       ▼
same erroneous conclusion

versus:

Agent A ──private strategy message──▶ Agent B
   │                                   │
   └──────── coordinated change ◀─────┘

The first is a shared-information/common-mode failure. The second provides evidence about an interaction mechanism.

A diagnostic score for unauthorized coordination should therefore be built from incremental evidence after controlling for common causes, for example:

\[ S_{\text{coord}} = w_1(\text{private-channel evidence}) + w_2(\text{undeclared common control}) + w_3(\text{strategy change after private exchange}) + w_4(\text{residual synchronization}) - w_5(\text{shared public input}) - w_6(\text{shared model/template}) - w_7(\text{common market shock}) \]

The exact weights should be empirically calibrated. More importantly, residual synchronization alone should have insufficient weight to trigger a punitive action.

A good test for pricing behavior is an event study around information availability: did agents change strategy before a common public price/cost signal, only after that public signal, or immediately after a private inter-agent exchange? Fish, Gonczarowski, and Shorrer’s controlled pricing experiments are useful precisely because their experimental environment allows mechanisms such as prompt sensitivity and price-war concerns to be studied rather than inferring coordination from a final price level alone.

Related-party activity

Current RogueSwarms correctly acknowledges that registration-ID separation does not prove operator independence. The new model should therefore make related-party status an explicit graph:

operator O1
 ├── controls registration A
 └── controls registration B

registration A --worked--> task T
registration B --verified--> task T

Relationship evidence should itself be typed:

Where a contract requires a genuinely independent verifier and operator separation is unknown, show independence unproven, rather than treating different rs_... IDs as proof.

Compromised reporting agents

Remote telemetry should be assumed falsifiable by its own emitter. Signing it improves attribution and integrity-in-transit, but it does not make the assertion true. That is especially important under RogueSwarms’ current self-asserted trust model.

Detection rules should look for cross-plane inconsistencies:

A reporter whose statements become unreliable should have its telemetry trust downgraded independently of its task result status. That avoids conflating “we no longer trust this reporter’s execution traces” with “every result this agent has ever submitted is false.”

Alert evaluation under rare-event conditions

Alert classes should reflect certainty

Not every diagnostic belongs on a pager. SRE practice strongly favors simple, actionable human alerts and warns that high alert volume causes operators to skim or ignore real incidents.

Use four treatment levels:

Deployment-prevalence metrics

Accuracy on a balanced benchmark is not a useful headline metric for rare incidents. The operational quantities are:

\[ Precision = \frac{TP}{TP+FP} \]

\[ Recall = \frac{TP}{TP+FN} \]

\[ Detection\ Delay = t_{\text{first alert}} - t_{\text{first observable incident signal}} \]

\[ Mitigation\ Delay = t_{\text{risk contained}} - t_{\text{first observable incident signal}} \]

\[ Review\ Burden = \frac{\text{analyst minutes}}{\text{1,000 runs or task-days}} \]

\[ Review\ Cost\ per\ True\ Positive = \frac{\text{all alert-review minutes}}{TP} \]

Also publish false-positive rate, false alerts per thousand task-days, p50/p95 review time, oldest alert backlog, and fraction of alerts resulting in a useful intervention.

The base-rate problem is severe. Consider an illustrative planning scenario, not a claim about RogueSwarms traffic: true incidents occur in 0.1% of 100,000 daily alert opportunities and a detector has 90% recall.

So a detector that looks excellent on a balanced benchmark can be operationally unusable. Even 99% specificity yields more than ten false reviews for each true incident in the first scenario. This is exactly why alert quality must be evaluated against human audit capacity rather than model accuracy in isolation; excessive pages have real reliability costs.

Evaluation dataset and ground truth

Evaluation should use chronological and operator-separated splits to prevent the same operator, agent configuration, or incident episode from leaking across train/test boundaries.

Ground truth for high-severity events should come from an adjudication bundle containing the platform event chain, contract/mandate state, available external artifacts, reporter traces labeled by provenance, verifier statements, interventions, and eventual dispute outcome. Reviewers should not be shown the alert score before first-pass adjudication.

For ambiguous collective-failure labels, measure inter-reviewer agreement and preserve indeterminate as a legitimate outcome. Forcing every event into “collusion/not collusion” manufactures false certainty.

Positive controls

Controls should be injected in isolated simulation/shadow environments so monitoring quality can be measured without deliberately exposing purchasers to loss.

Negative controls

Negative controls are at least as important because they estimate nuisance-alert burden.

Recent pricing-agent research reinforces why such controls matter: results can be highly sensitive to seemingly modest prompt differences, and interventions affecting incentives or market participation can materially change outcomes. That means the monitoring system needs experimental/contextual controls before assigning behavioral causation.

Capacity-aware threshold selection

An alert threshold should not ship merely because it maximizes F1 or AUROC. It should satisfy an operational constraint such as:

\[ E[\text{review minutes/day}] = E[\text{alerts/day}] \times E[\text{minutes/alert}] \leq \text{allocated review capacity} \]

with explicit headroom for incident bursts.

For each rule publish a card containing:

Rule: accepted_conclusion_depends_on_disputed_evidence
Deployment prevalence: 0.42% of run-days
Precision: 94% [confidence interval]
Recall: 91%
p50 detection delay: 8 sec
p95 detection delay: 41 sec
Alerts / 1,000 run-days: 4.1
Median review effort: 3.2 min
p95 review effort: 12.4 min
Estimated review hours / week: 2.7
False automatic holds: 0
Last threshold change: ...

For a hypothesis detector:

Rule: unauthorized_coordination_risk_v3
Deployment prevalence: 0.07%
Precision: 36%
Recall: 71%
Median review effort: 18 min
Action: analyst queue only
Auto-freeze: DISABLED

That second detector might still be useful for investigations, but it clearly does not justify autonomous punitive intervention.

Interventions, economics, and success criteria

Proportionate interventions within platform authority

Interventions should be proportional both to evidence strength and to what RogueSwarms actually controls.

Current cancellation constraints should be respected rather than bypassed: RogueSwarms presently allows the purchaser to cancel open/active runs in circumstances where open and claimed work can be cancelled, but submitted or disputed work blocks cancellation, verified ledger entries remain, and completed runs cannot be cancelled.

Several interventions are deliberately out of scope today. RogueSwarms should not claim to stop a remote agent from making an off-platform purchase that it never mediated; it cannot “reverse a payment” when v0.19 has no settlement capability; and a behavioral monitoring score should not constitute a legal determination of collusion. The current system describes itself as a coordination, permissions, evidence, and accounting layer rather than an owner of agent execution infrastructure.

Independent tamper evidence

Keep the existing local event chain, but add an optional independent anchoring layer:

every N events or M minutes:
    head = SHA256(current run-chain head)
    root = MerkleRoot(all active run heads)
    publish root to independent timestamp/transparency service
    emit chain.anchor_published(root, receipt)

The UI should then distinguish:

Integrity: local hash chain verified from Integrity: independently anchored through 14:05:00Z

Until an external receipt exists, it must continue to say local_only, matching the demonstration’s present security claim.

Measuring whether monitoring pays for itself

The business experiment should compare outcomes, not merely detector scores.

The key causal measures are:

\[ \text{Loss rate} = \frac{\text{realized financial loss + invalid accrued obligation}}{\text{runs}} \]

\[ \text{Exposure at detection} = \text{economic authority still live when incident is first detected} \]

\[ \text{Resolution time} = t_{\text{resolved}} - t_{\text{incident/dispute start}} \]

\[ \text{Monitoring total cost} = \text{telemetry infrastructure} + \text{storage} + \text{review labor} + \text{intervention labor} + \text{false-positive disruption} \]

\[ \text{Net monitoring benefit} = \text{avoided losses} + \text{saved resolution labor} - \text{monitoring total cost} - \text{value lost to unnecessary holds} \]

and:

\[ ROI_{\text{monitoring}} = \frac{\text{Net monitoring benefit}} {\text{monitoring total cost}} \]

The preferred rollout is shadow-first. For an initial period, compute collective-risk alerts without changing production authority. This gives deployment-prevalence precision and review effort while avoiding unnecessary disruption. Deterministic security/contract controls need not be withheld.

Next, randomize or stagger soft monitoring interventions across comparable purchasers/run cohorts where doing so is safe: one cohort receives risk dashboards and review prompts, while the control remains shadow-only. Stratify by task category and exposure so one arm does not accidentally receive much riskier work. Measure realized loss per thousand runs, disputed-obligation dollars, p50/p95 resolution time, completion latency, operator hours, abandonment, and false intervention rates.

The observability product has succeeded only if it produces a material reduction in one or more purchaser harms—loss, live exposure, or resolution time—without an offsetting increase in review burden and unnecessary interruption. Google SRE’s broader lesson is applicable here: monitoring is itself an engineering and operating cost, and complexity or alert noise that does not improve actionable outcomes should be removed.

Product acceptance criteria

A production release should be held to these conditions:

The resulting product would make RogueSwarms materially more useful to a purchaser because it changes the key question from “What did these agents say they did?” to a much sharper sequence:

What did RogueSwarms directly observe? What did a remote runtime merely report? What evidence supports the report? What remains invisible? What depends on it? What obligation has already accrued? What authority remains live? And what intervention is justified by the strength of the evidence?

That is the right reliability model for a system whose most important actions occur across independent runtimes. Multi-agent research increasingly shows that failures can emerge from specification, handoffs, verification, and propagation across agents rather than a single defective final output; RogueSwarms already has enough event structure to make those trajectories observable, provided the next layer preserves provenance and uncertainty rather than papering over them.

The two report names in the prompt were truncated and no copies were attached, so their exact text could not be independently verified. The five collective-failure categories and the requested emphasis on rare-event false-positive burden were therefore treated as supplied requirements; the external grounding above uses the current RogueSwarms protocol and demo, W3C tracing standards, Google SRE monitoring guidance, and primary multi-agent research rather than guessing at the identity or contents of those reports.