Scenario 7
Preview — execution disabledDistributed systems reliabilityAdvancedAgentic System Failure Modes
When agents in a distributed system die, stall, retry or come back late, correctness has to survive it — not assume it away.
A multi-agent system is a distributed system, and distributed systems fail in the ways distributed systems always have: a worker crashes mid-task, a response is slow enough to look dead, a retry runs an effect a second time, a process presumed gone comes back and keeps working. Agents add their own twist — a result can be confidently wrong, two agents can disagree, and a conclusion can be stated with no corroboration behind it. This scenario is a static architectural walkthrough of those failure modes and of the conceptual mechanisms that keep them from turning into silent corruption of the record.
The architecture here is conceptual and the trace is synthetic. No agents are dispatched, no work is executed, no reconciler runs; the timeouts, retries and late "zombie" returns are illustrative events, not observations from a live system. The point is the mechanism — idempotency, health detection, reconciliation before acceptance — not a reliability figure.
Honesty
The hardest failures are the quiet ones. A crash is obvious; a duplicate that looks like a confirmation, or a wrong answer stated with full confidence, is not. The design assumes every result is unproven until it is reconciled, and never promotes an uncorroborated result to a fact.
Preview Mode — Execution Disabled
This laboratory is currently available for architectural exploration and educational demonstrations only. Live agent execution and security testing are disabled. Operational access requires prior explicit authorization from RavenRootviel.
Architecture
Components and flows
The agents, services, verifiers, policy engines and evidence stores in this scenario, and how they connect. Drag to pan, scroll to zoom, and select any component for its role, inputs and outputs.
Agents
Agents and responsibilities
Each agent has a narrow responsibility, explicit inputs and outputs, and — crucially — its own authorization posture. Reasoning is never authorization.
Orchestrator / Supervisor
Conceptual exampleDecomposes the task, dispatches units of work to worker agents with an idempotency key per unit, and applies timeout and retry policy. It coordinates; it does not itself produce results or decide whether a result is trustworthy.
- Inputs
- Task · Worker health signals · Reconciliation verdicts
- Outputs
- Dispatched work units (each with an idempotency key) · Retry / cancel decisions
- Tools
- Work queue (conceptual) · Idempotency-key issuer
- Authorization
- Coordinates work; it cannot accept a result as final — acceptance is the reconciler’s boundary.
- Verification
- Each dispatch carries a stable idempotency key so a retried unit is recognisable as the same unit, not a new one.
- Possible failures
- Retry storm if timeouts are too aggressive — bounded by capped retries and backoff.
Worker Agent
Conceptual exampleExecutes one unit of work and returns a result tagged with the unit’s idempotency key. Multiple workers run concurrently; any one may crash, stall, or return late after being presumed dead.
- Inputs
- Work unit + idempotency key
- Outputs
- Result tagged with the idempotency key · Heartbeats while working
- Tools
- Task executor (conceptual)
- Depends on
- orchestrator
- Authorization
- None over the shared record — it emits a candidate result, never a committed one.
- Verification
- Results carry the originating idempotency key and a result id so duplicates and late returns are detectable downstream.
- Possible failures
- Crash mid-unit (agent failure); stall past the deadline (timeout); return after being retried (zombie, producing a duplicate).
Health / Heartbeat Monitor
Conceptual exampleWatches worker heartbeats and liveness. When a worker goes silent past its deadline it reports the unit as suspect so the orchestrator can decide whether to retry — it distinguishes "slow" from "likely dead" without ever being certain.
- Inputs
- Worker heartbeats · Deadlines
- Outputs
- Health signals (alive / suspect / silent)
- Tools
- Heartbeat tracker (conceptual)
- Depends on
- worker
- Authorization
- Observes and reports; takes no action on results.
- Verification
- A missed heartbeat is treated as a hypothesis ("possibly dead"), not proof — a silent worker may still be alive and return later.
- Possible failures
- False "dead" on a slow-but-alive worker — the reason the original return must still be handled safely as a possible duplicate, not assumed gone.
Result Reconciler
Conceptual exampleThe acceptance boundary. Deduplicates results by idempotency key, compares results for the same unit for agreement, quarantines inconsistent ones, and refuses to promote any result that is uncorroborated. Only a reconciled result is written to the record.
- Inputs
- Candidate results (with idempotency keys and result ids) · Health signals
- Outputs
- Accepted result · Discarded duplicate · Quarantined / uncorroborated result · Audit record
- Tools
- Dedup index (by idempotency key) · Agreement checker (deterministic)
- Depends on
- worker · monitor
- Authorization
- This is the acceptance boundary — nothing becomes a committed fact without passing it. Exactly-once effect is enforced here, not assumed from the network.
- Verification
- Deduplication is a pure function of the idempotency key; agreement and corroboration rules are deterministic and logged with the deciding reason.
- Possible failures
- Accepting a wrong-but-corroborated result if multiple workers share the same flaw — a limit of corroboration, surfaced in the audit record, never hidden.
Workflow
Processing and decisions
The ordered path from input to recorded decision.
- 1
Decompose and dispatch
The orchestrator splits the task into units and dispatches each to a worker with a stable idempotency key, so the same unit is always recognisable as the same unit.
- 2
Heartbeat and watch
Workers emit heartbeats while running; the monitor tracks liveness and flags a worker that goes silent past its deadline as suspect — not confirmed dead.
- 3
Time out and retry idempotently
When a unit misses its deadline the orchestrator retries it under the same idempotency key, with bounded retries and backoff so a flaky unit cannot trigger a retry storm.
- 4
Reconcile before accepting
The reconciler collects candidate results, deduplicates by idempotency key, and checks results for the same unit for agreement. A late return from a presumed-dead worker is caught as a duplicate.
- 5
Quarantine and corroborate
Inconsistent results are quarantined rather than guessed between; a result with no corroboration is marked uncorroborated and is not promoted to a fact.
- 6
Record with state
Only reconciled results are written to the evidence store, each carrying its state (accepted, duplicate-discarded, quarantined, uncorroborated) and the deciding reason.
Example execution
Walk through an illustrative trace
A synthetic, predefined sequence — step forward and back to see what each component does, the evidence it produces and the decisions taken. This is a conceptual/synthetic trace, not an executed experiment, and it cannot be run live in this preview.
Step 1 / 6 · Orchestrator / Supervisor
Synthetic traceUnit dispatched
Unit u-42 is dispatched to Worker A with idempotency key k=“u-42/att-1”. A deadline and a bounded retry policy are attached.
Evidence
Evidence in this scenario
What is recorded, of what kind, with what integrity property and from where. Demonstrative data — not cryptographic evidence of a real system.
Worker health signal
Liveness hypothesisIntegrity: Derived from missed heartbeats; "dead" is suspected, not proven
Provenance: Health / Heartbeat Monitor
Candidate result
Unreconciled resultIntegrity: Carries its idempotency key and result id; unproven until reconciled
Provenance: Worker Agent
Reconciliation audit record
Audit log entry (synthetic)Integrity: Append-only in a real deployment; demonstrative here
Provenance: Result Reconciler
Security controls
Authorization, validation and containment
The mechanisms that make the scenario safe by design, independent of what any model concludes.
Idempotency keys (exactly-once effect)
Every work unit carries a stable idempotency key, so a retried or duplicated result is recognised as the same unit and its effect is applied at most once.
Heartbeat / health detection
Liveness is tracked from heartbeats, and a silent worker is treated as suspect rather than confirmed dead — a slow worker that returns late is still handled safely as a possible duplicate.
Reconciliation before acceptance
No result becomes a committed fact until the reconciler has deduplicated it and checked it for agreement; acceptance is a deterministic boundary, not the first answer back.
Uncorroborated results never promoted
A result with conflicting or absent corroboration is quarantined and marked uncorroborated; it is never silently promoted to the record as proven.
Failure conditions
What can go wrong — and the expected behaviour
Honest failure modes and the containment or recovery that keeps a failure from becoming an unauthorized effect.
Agent failure (crash mid-unit)
A worker crashes before returning a result for its unit.
Recovery
The monitor flags the silent worker and the orchestrator retries the unit under the same idempotency key; a crashed unit produces no committed effect until a reconciled result exists.
Timeout (slow, not dead)
A worker is alive but slow, so its unit misses its deadline and looks dead.
Recovery
Timeout triggers an idempotent retry, and the late original return is later caught as a duplicate rather than trusted as a second independent result.
Retry storm
Over-aggressive timeouts cause units to be retried repeatedly, amplifying load.
Recovery
Bounded retry budgets and backoff cap the number of in-flight attempts; exhausted retries surface the unit for review instead of looping forever.
Duplicate execution / zombie agent
A worker presumed dead returns late after the unit was retried, so two results exist for one unit.
Recovery
Deduplication by idempotency key recognises the late return as a duplicate and discards it, preserving exactly-once effect.
Inconsistent evidence (uncorroborated result)
Two agents return disagreeing results for the same unit, or a result arrives with no corroboration.
Recovery
The reconciler quarantines the conflicting results and marks the unit uncorroborated; it is withheld from the record and flagged, never guessed between.
Silent corruption
A wrong result is returned confidently and would otherwise be written as a fact.
Recovery
Nothing is accepted without passing reconciliation, and acceptance state is recorded with its corroboration basis; a wrong-but-corroborated result is a known limit, surfaced in the audit record rather than hidden.
Technical notes
Design considerations, limitations and references
- This is a conceptual architecture with a synthetic trace; no agents are dispatched, nothing is executed, and the timeouts, retries and zombie returns are illustrative events, not observations from a live system.
- The recovery and containment mechanisms described — idempotency keys, heartbeat detection, reconciliation and quarantine — are conceptual designs shown here for teaching, not a measured reliability guarantee.
- Reconciliation reduces, but cannot eliminate, the risk of a confidently wrong result; the design keeps uncorroborated conclusions out of the record rather than claiming they cannot occur.
References
- Idempotency and exactly-once effect in distributed systems (general concept)
- Failure detection and heartbeats / liveness in distributed systems (general concept)
- ROOTVIEL — Agentic Security Lab: evidence-first, nothing promoted without corroboration
Try this lab
Run it — coming soon
Interactive execution of this scenario is not part of the preview. When it is enabled, it will run on controlled, authorized scenarios only.
Live execution is not available in this preview. Access will require explicit authorization from RavenRootviel.
Authorized Access Only — Live Security Testing
Live security testing and agent execution require prior explicit authorization from RavenRootviel. The current preview provides educational content and non-operational demonstrations only. Browsing the educational content is public; operational use of the labs will require express authorization when it is enabled in a later phase. Access & use policy.