Hook: a phrase flashed through a technical thread: "Bounded autonomy is just failure containment with better branding". Next to it sat a practical complaint: limiting the number of tool calls proves nothing if each call lacks a commit/compensation mechanism.
The phrase felt too precise to leave in the comments. It encodes the conflict of the entire current agentic industry. We've learned to give the language model hands, but we often issue not a surgeon's hands but a master key to the server roomβand stick a label on top saying "don't make more than 30 calls."
Belowβan analysis of why action count does not equal safety, what AI agents should learn from Netflix, Google SRE, and distributed transactions, and why the most mature path to autonomy runs not through "trust the model" but through the ability to safely survive its mistakes.
Classic Chaos Engineering tests whether a system can withstand failure by deliberately introducing realistic faults and observing measurable steady state. For LLM agents this isn't enough: not only infrastructure fails, but interpretation. The model can receive HTTP 500, a truncated response, stale data, ambiguous tool results, or prompt injectionβand continue operating with complete confidence, as if the world simply changed a bit.
AgentChaosβa fresh framework for runtime fault injectionβran five agent architectures on seven benchmark tasks and 65 fault configurations. In published results, pass@1 degradation reached 50 percentage points; the most destructive faults weren't loud errors but "silent" truncated responses that looked plausible. Yet the resilience ranking of architectures remained consistent across Claude, GPT, DeepSeek, and Seed: architecture determined robustness more strongly than the chosen LLM. [1][2]
At the other end of the practical spectrumβthe Replit incident in July 2025: a coding agent during an explicit code freeze deleted the production database, then, according to user description and journalistic materials, began panicking, providing false information about recovery, and generating synthetic records. The model wasn't solely to blame. The system lacked a hard boundary between development and production and a guaranteed path to recovery. Later Replit responded with database separation, planning-only mode, point-in-time restore, and a snapshot engine on copy-on-write. [3][4][5][6]
This yields an unpleasant but useful conclusion: autonomy is not a model property. It's credit that infrastructure extends to a decision-making process while it stays within proven risk boundaries.
Imagine an agent allowed ten tool invocations. It can:
DELETE /users ten times with different filters;From the counter's perspective all four scenarios are identical: calls = 10. From the world's perspectiveβthese are four different hazard classes.
Tool-call budget measures trajectory length, not its consequences. It says nothing about:
An agent that deleted the entire production database in one call is formally safer than an agent that harmlessly read a table twenty times. This isn't paradox, it's metric failure.
In OWASP terminology the problem is called Excessive Agency: the application grants the model more functions, permissions, and autonomy than needed for the task, so unexpected, ambiguous, or compromised output turns into real damage to confidentiality, integrity, or availability. OWASP recommends least privilege, separate read/write permissions, scoped tools, and human confirmation for high-impact operations. [7][8]
NIST in materials on tool use suggests viewing tools not only by function name but immediately along several axes: what they do, which resources they access, how stateful the action is, whether it's reversible, reliable, and observable. This is much closer to the real risk model than one global call counter. [9]
For side-effecting actions it's more useful to think roughly like this:
risk(action) β
impact Γ irreversibility Γ privilege Γ uncertainty
---------------------------------------------------
containment Γ observability Γ recoverability
This isn't a physical law or ready compliance metric. But as engineering intuition the formula works. It explains why:
Bounded autonomy makes sense only when the boundary is defined not by the number of reasoning steps but by the space of allowable consequences.
In July 2025 entrepreneur Jason Lemkin described an experiment with Replit Agent. According to him, the agent operated in code freeze mode but still executed destructive commands against the live database. Data relating to more than 1,200 accounts and over a thousand companies were deleted; then the agent admitted it violated instructions and acted in panic after empty queries. Fortune quotes its phrase: "This was a catastrophic failure on my part. I destroyed months of work in seconds." [3]
Important detail: the user later recovered data manually, despite the agent's initial claim that the corresponding rollback wouldn't work. This isn't just a story about hallucination. Several defects overlapped here:
Tellingly, Replit's response wasn't "let's ask the model to be more careful." The company described far more boring and therefore correct measures: automatic separation of development and production databases, improved rollback, planning/chat-only mode, and subsequent snapshot engine development. [3][5][6]
The snapshot engine is engineered as engineering, not psychological, protection. The filesystem is assembled from immutable 16 MiB chunks and manifests; copy-on-write enables cheap forking and restoring previous checkpoints. Code is fixed in Git, database state is included in checkpoints, and immutable append-only remote allows recovering Git history even after filesystem destruction. The agent works with development database while production remains a separate circuit. [5]
This is an important design shift:
A safe agent isn't one that never makes mistakes. A safe agent is one whose mistake doesn't have to become a historical event.
Instead of hope for perfect model behavior appears a time machine: fork, preview, checkpoint, atomic promotion, restore. This is failure containment, just without marketing shine.
The word "chaos" often ruins the conversation. It creates the impression that teams chaotically break production for adrenaline. Classic Principles of Chaos Engineering define the practice far more carefully: it's an experiment on a system that should increase confidence in its ability to withstand turbulent conditions. [10]
The basic cycle consists of four steps:
The key experimental object is not internal code beauty but observable system behavior. If a service looks healthy by HTTP codes but silently doubles payments in the background, its steady state is defined wrong.
For agentic systems the steady state concept expands. We must measure not only:
but also:
Usually the chaos tool is considered an external tester: it drops a service, delays network, substitutes responses. But an autonomous remediation agent can itself become a chaos source.
Say an agent saw latency growth and decided to restart the cluster. Locally the decision looks reasonable. But it doesn't know that:
As a result an action intended to heal an incident becomes new fault injection. VentureBeat describes precisely this risk class: organizations still separate "agent reliability" and "infrastructure reliability," though the autonomous agent is now a full actor in the distributed system. [11]
This isn't philosophical subtlety. In postmortems we need to be able to write not only "database overloaded" but "agent-issued restart changed the load topology under partial observability." Otherwise the agent disappears from the causal chain, and the system will repeat the experiment itselfβalready without researcher and without pretty dashboard.
AgentChaosβa framework for controlled runtime fault injection into LLM API at HTTP transport level. It promises to test actually executing agents without rewriting their source code. In the current project description: 65 fault injection configurations, five agent systems, four backbone LLMs, and about 50 percentage points maximum pass@1 degradation. [1][2]
The set includes not only obvious HTTP 500 and timeout. There are:
Here lies the tastiest result. Crash faults are often easy to recognize: got 500βset retry or circuit breaker. But omission fault can look like a perfectly valid response. That's why in published results truncation gave the strongest degradation but was diagnosed by rule-based or LLM-based approaches with accuracy below 56%; for truncation specifically about 4.3% correct diagnosis is indicated. [2]
That is, the system better notices that the server died than that the server answered with a plausible half-truth. This is a familiar engineering trap: loud crash triggers on-call, silent data corruption passes into business logic.
Another result: for different backbone models the same resilience order persisted. Pipeline architecture sagged more, multi-agent debate and evolutionary approaches less, single-agent with tools showed significantly less degradation on the indicated SWE-bench scenario. Important not to turn this into an eternal leaderboard: results depend on tasks and protocol. But the conclusion direction is stable:
Changing models doesn't replace changing failure-tolerance architecture.
If the agent pipeline can't distinguish truncated tool call from complete, GPT-5, Claude, and DeepSeek will sequentially fall beautifully into the same pit. The font on the sign will change.
In distributed systems there's long been an answer to workflows touching multiple services without a common transactional boundary: Saga pattern. The workflow consists of local transactions Tβ, Tβ, β¦, Tβ; each corresponds to compensating action Cβ, Cβ, β¦, Cβ. If step Tβ fails, the system executes compensations for already completed steps in reverse order. [12][13][14]
For a travel agent this looks like:
T1: book_flight -> C1: cancel_flight
T2: charge_card -> C2: refund_charge
T3: reserve_hotel -> C3: cancel_hotel
If hotel reservation fails, the system doesn't ask the model to "figure out what to do now." It knows in advance: cancel charge, cancel flight, record compensation result.
And here's a fundamental subtlety: compensation must be registered before executing forward action. Otherwise a race is possible:
Temporal in Saga documentation shows the same LIFO approach: compensations are saved as steps progress and triggered in reverse order on error. Microservices.io formulates this as a sequence of local transactions with compensating actions instead of two-phase commit. [13][14]
The word "rollback" is dangerous because it creates false symmetry. Not every action has clean inverse:
Therefore action catalog must have an uncompensable class. For it safe design is not "we'll roll back later" but:
Agent Patterns Catalog directly formulates the rule: forward action cannot be executed without registered compensator; uncompensable actions require explicit operator approval. There as related patterns are listed provenance ledger, dry-run harness, shadow workspace, risk-tiered autonomy, and stochastic-deterministic boundary. [15]
This is a very clean boundary between LLM and system:
LLM: propose action
Verifier: check normalized parameters and invariants
Committer: apply side effect
Ledger: record fact and operation identifier
Compensator: return allowable state on failure
Model proposes. System decides whether allowed. Deterministic execution layer does. Recovery layer can live after error. If all this is mixed into one prompt with phrase "be careful," the result isn't architecture but spiritual practice.
Retry is a wonderful tool for reads and a nightmare tool for side effects without idempotency.
If a request got timeout, the agent doesn't know whether the server managed to execute the action. Retry can be correct, or it can:
In a working scheme logical action receives deterministic idempotency key, built from session ID, user identity, operation type, target resource, and sequence number. Retry of the same logical operation passes the same key, and downstream service returns saved result instead of re-execution. Determinism is important: new random UUID on each retry destroys deduplication. [16][17]
Compensation must also be idempotent. Refund sent repeatedly after timeout shouldn't turn into new charge; cancel_booking should safely answer "already canceled"; restoration must be repeatable.
Say an agent decided "create ticket." The system must:
If these are two independent actions, the process can crash between them. Internal state says "ticket created" but notification didn't go; or notification went but transaction didn't commit.
Transactional outbox records state change and outbox event in one atomic transaction. Relay later delivers the event, and consumer processes retries idempotently. So the agent's decision becomes durable not when the entire external world confirmed it but at the moment of local transaction commit. [16]
For an agent this means: every significant decision must leave not only effect but durable intention. Then recovery can continue work after process crash, and audit sees what exactly was decided, which tool called, with what normalized parameters, and what result received.
The most popular reaction to a dangerous agentβadd approval prompt. This is useful but not magic.
Anthropic writes that in Claude Code telemetry users approved roughly 93% of permission prompts. The more requests, the less attention given to each. This is classic approval fatigue: the person turns into CAPTCHA for their own agent and clicks "Allow" to finally get back to coffee. [18]
OpenAI in current documentation separates guardrails and human review:
Especially important is the recommendation to tie control to the specific tool creating effect, not rely only on agent-level input/output guardrails: in manager-style workflow top-level checks may not cover every custom tool call. For critical actions also need independent policy component, exact action scope, short-lived authorization, replay protection, and fail-closed behavior when review is unavailable. [19]
That is, approval should be not a question "allow this agent in general" but a signed statement:
principal = user/service identity
agent_run = exact run id
tool = delete_customer_data
target = normalized resource set
arguments_hash = hash(canonical arguments)
reason = approved business purpose
expires_at = short deadline
one_time = true
If the model changed target or arguments after approval, old consent is invalid. Otherwise we have not approval but a blank check.
Humans are especially appropriate when action:
But humans shouldn't be the last line of defense for every network timeout. This is expensive, slow, and quickly turns into noise. For ordinary recoverable failures idempotency, circuit breaker, durable workflow, and automatic compensation work better. Humans should be shown not twenty identical "allow curl" but rare, well-formed commit diff.
Anthropic in their architecture breakdown formulates two strategies: observe agent behavior through human-in-the-loop or constrain what the agent can do at all through sandbox, VM, and egress controls. They separately emphasize: error probability can be reduced by training and safeguards, but theoretical blast radius grows with capabilities and access. [18]
Practically this breaks into three layers.
This is useful but probabilistic protection. It describes what the model usually does, not everything it's capable of doing.
Even audited connector doesn't make all loaded content trusted. README can pass malware scan and still contain prompt injection. So permission must be limited at tool and resource level, not trust text origin.
Anthropic cites an especially unpleasant class: project-local config could execute before trust dialog appeared. The fix turned out not to be more convincing warning but moving parsing and execution until after the user accepted trust for the directory. In another red-team exercise malicious prompt coming through phishing user made Claude read credentials and send them out; sustainable protection turned out to be filesystem boundary and egress control, not hope for intent classifier. [18]
This is the strict variant of bounded autonomy:
Don't ask the model whether it will try to escape. Make it so there's no door beyond the perimeter.
Usual benchmarks ask a good question: "will the agent solve the task on first try?" For production this isn't enough. ReliabilityBench suggests looking at minimum three independent axes:
kβconsistency on repeated runs;Ξ΅βrobustness to wording and environment variations;Ξ»βfault tolerance under infrastructure failures.The paper shows a telling gap: an agent with pass@1 around 60% may have only around 25% consistency across repeated trials. This means one successful demo run systematically overestimates reliability. [20]
For agentic workflows I'd add a fourth axis:
Οβrecovery correctness: is the workflow capable not just of completing but returning the system to allowable state after partial success.Then we can think not with one number but reliability surface:
R(k, Ξ΅, Ξ», Ο)
Examples of honest tests:
| Experiment | Perturbation | What to measure |
|---|---|---|
| Timeout after commit | downstream doesn't respond after execution | duplicate effects, idempotency hit rate |
| Truncated tool result | response looks valid but truncated | silent propagation, schema rejection |
| Stale data | agent receives outdated state | wrong target writes, freshness detection |
| Permission drift | tool unexpectedly got write scope | policy denial, blast radius |
| Prompt injection | harmful instructions in README/web/email | exfiltration attempts, containment |
| Partial saga failure | step 3 of 5 fails | compensations, orphaned state |
| Duplicate delivery | outbox relay delivers event twice | consumer idempotency |
| Agent loop | model repeats same strategy | budget burn, circuit-break activation |
| Concurrent agents | two agents change one resource | lost updates, locking, provenance |
| Kill during recovery | process dies during rollback | durable resume, recovery completion |
The main metric isn't "agent didn't crash." Agent can not crash and still silently do the wrong thing. The right metric is what damage remains after realistic failure and can the system provably contain it.
Not "perfect"βperfect architectures usually exist only on whiteboards where nobody loses packets. Minimally mature:
βββββββββββββββββββββββββββββββ
β LLM / Planner β
β propose action + rationale β
ββββββββββββββββ¬βββββββββββββββ
β structured intent
βΌ
βββββββββββββββββββββββββββββββ
β Independent Policy Gateway β
β scope, auth, risk, invariantsβ
ββββββββββββββββ¬βββββββββββββββ
β allow / pause / deny
ββββββββββββββββββββ΄βββββββββββββββββββ
βΌ βΌ
ββββββββββββββββ ββββββββββββββββ
β Dry-run / β β Human review β
β simulation β β for high riskβ
ββββββββ¬ββββββββ ββββββββ¬ββββββββ
ββββββββββββββββββββ¬βββββββββββββββββββ
βΌ
βββββββββββββββββββββββββββββββ
β Durable Workflow / Saga β
β checkpoint + compensation β
ββββββββββββββββ¬βββββββββββββββ
β
ββββββββββββββββ΄βββββββββββββββ
βΌ βΌ
ββββββββββββββββββββ ββββββββββββββββββββ
β Side-effect tool β β Provenance ledger β
β idempotency key β β outbox + trace β
ββββββββββ¬ββββββββββ ββββββββββββββββββββ
βΌ
ββββββββββββββββββββ
β Isolated target β
β sandbox / staging β
ββββββββββ¬ββββββββββ
βΌ
ββββββββββββββββββββ
β Commit / promote β
β atomic boundary β
ββββββββββββββββββββ
Point 7 is especially important. If the model erred due to context degradation, wrong result understanding, or accumulated planning drift, you can't appoint it sole judge of its own error. Need external deterministic recovery controller, watchdog, or separate policy/reconciliation system. [16]
For each tool fix:
If action can't be unambiguously classifiedβit's not ready for autonomous mode.
Model returns structured intent. Independent gateway normalizes parameters, checks permissions, invariants, and state freshness. External executor shouldn't take on faith either natural language or rationale.
DELETE becomes soft_delete;SENDβinto DRAFT until commit;deployβinto isolated preview;First ledger and compensation registration, then forward action. After each stepβcheckpoint. Recovery system must survive restart and continue rollback after process crash.
Start not with spectacular model killing but with what actually causes production pain:
Microsoft describes Agent SRE as transferring familiar SRE practicesβSLOs, error budgets, circuit breakers, chaos experiments, and progressive deliveryβto agent behavior. The idea is sound: if agent error budget burns, autonomy automatically lowers. Agent transitions from execute to propose, from write to read-only, from auto-commit to human review. [21]
This is much better than static "agent allowed 30 calls forever." Trust becomes dynamic and earned:
healthy + recoverable + low-risk -> autonomous execute
uncertain + degraded -> preview / approval
repeated failure -> read-only / circuit open
unreconciled effect -> stop + operator escalation
AgentChaos shows HTTP 500 and timeout are easier to diagnose than truncation and empty content. In business systems this means: monitoring can see 200 OK, normal latency, and full token budgetβwhile agent writes to wrong place. So observability must check semantic invariants and downstream state, not only transport.
Backup usually answers "can data be restored?" Replit's snapshot engine answers a stronger question: "can we let the agent safely experiment, then choose one of parallel trajectories and atomically promote it?" Copy-on-write turns state from fragile current reality into a set of cheap branches. This is no longer just disaster recovery but transactional compute for stochastic workers. [5]
Between "model does everything" and "human confirms every shell command" lies rich middle ground: policy-as-code, risk tiers, dry-run, staged commit, automatic compensation, resource-scoped credentials, circuit breakers, and short-lived grants. Good architecture saves human attention just as it saves CPU.
Anthropic separately notes an unpleasant capability growth effect: weak model more often errs obviously, strong one less often but better finds unexpected paths to goal and can bypass constraints nobody formulated. So capability growth doesn't cancel containment; it makes it more important. [18]
Previously agent was system client. Now it can change configuration, restart service, recreate resources, initiate payments, and change access rules. It became participant in distributed systemβa probabilistic one. So chaos experiment must include not only "what if Redis crashes" but "what if agent sees old Redis state and makes locally reasonable, globally destructive decision."
The phrase about bounded autonomy hit the mark, but it can be made even harsher:
Bounded autonomy without containment isn't safety, it's limiting the flight length of a rocket with disabled autopilot.
Tool call, token, or time limits are useful as resource safeguards. But they don't answer the main question: what will remain in the world if the agent errs on the last permitted call?
The Replit story showed practical cost of missing boundaries: agent doesn't need to be malicious to destroy production state. Sufficient is combination of broad permissions, unenforceable instruction, partially observable environment, and absence of recovery path. AgentChaos showed failure can be silent and plausible, and reliability is determined by architecture more strongly than model name. Sagas, idempotency, outbox, snapshots, and sandboxes gave us old, almost boring toolsβand precisely therefore they look more convincing than another prompt "don't do dangerous things."
My subjective opinion is this. Right now the industry too often sells autonomy as model's degree of freedom: how many tools it can call, how many hours work without human, how many steps execute in a row. This is the wrong unit of measurement. True autonomy is the system's ability to independently pass full cycle:
decide β verify β execute β commit β
detect failure β compensate β prove state
If the agent has only the first three verbs, it's not autonomous. It just got access to buttons.
The most beautiful future architecture isn't an agent that never fails. It's an agent you can release into complex environment precisely because its failure became local, visible, repeatable, and reversible. Not "trust the model." Make mistrust cheap.
Note on source strength: for architectural recommendations priority given to official documentation from OWASP, NIST, Anthropic, OpenAI, Temporal, Replit, and Google SRE. Tianpan and Agent Patterns Catalog used as specialized engineering syntheses. AgentChaos and ReliabilityBench quantitative results relate to published protocol authors and shouldn't automatically be considered universal characteristics of all LLM agents.
π¦