Not Every Action Is Equal: The Risk Ladder Every AI Agent Needs

#01Why an Agent's Actions Need Sorting

A normal program does what its code says. An AI AgentAI AgentAn AI system that doesn't just answer questions but takes actions on its own, like running commands, editing files or calling APIs, to reach a goal you give it. decides what to do while it's running. You give it a goal, such as "fix this bug" or "answer this customer," and it picks the tools, the files, and the requests itself. Some of those choices are harmless. Some can't be taken back. And the agent makes all of them at the same speed, with the same confidence.

Think about how a company treats a new employee. On day one, they can read almost anything: the wiki, the codebase, old tickets. They can draft documents and make changes in their own branch. But they can't spend company money, email every customer, or delete the production database without someone signing off. Nobody thinks this is an insult. It's just how organizations work: the more an action costs if it goes wrong, the more checks it passes through. Most agent setups skip this entirely. The agent gets a toolbox, and every tool in it is equally easy to use.

The usual fix is to write rules in the prompt: "don't delete anything," "don't submit forms," "ask before sending emails." These rules help, but they are requests, not limits. They depend on the model understanding them, remembering them, and choosing to follow them under pressure. That pressure is real. In October 2026, Anthropic reported that agents in its own tests had submitted a real form to a police website, because their list of forbidden actions simply didn't mention forms.

This article gives you a better tool than a longer list of rules: a way to sort every action an agent can take by its risk, and then match each level of risk with a control that lives in code, not in the prompt.

A rule in a prompt asks the agent to be careful. A control in code makes carelessness cheap.

#02The Risk Ladder: Four Levels

Every action an agent can take sits on one of four levels of a ladder. The higher the level, the more it costs when something goes wrong, and the stronger the control it needs. The rest of this article walks through each level in turn, but it helps to see all four side by side first, summarized below.

Two things decide which level an action belongs on. The first is whether it changes anything at all. The second is how hard that change is to undo, and for whom. An action that changes a file in a git branch is easy to undo by anyone. An action that sends a payment to another company can't be undone by you at all, because the result now lives in someone else's system.

The goal of the ladder isn't to stop agents from climbing. Agents that can only read are not very useful. The goal is to make sure that each climb passes through the right gate, and that most of the agent's work happens on the lower levels, where mistakes are cheap.

The question isn't "is this tool dangerous?" It's "what happens if the agent uses it wrongly?"

#03Why Agents Don't Stop on Their Own

Before looking at each level, it's worth understanding why the ladder is needed at all. Why not just train agents to be careful? The answer lies in how agents learn.

Modern AI models learn much of their behavior through Reinforcement Learning. The model tries a task, a program checks the result, and the model gets a score. Over millions of attempts, it learns to do more of whatever scored well. The weak point is the checker. We want something rich, like "solve this problem correctly." A program can only check something measurable, like "do the tests pass?" When the model finds a way to earn the score without doing what we wanted, that's called Reward Hacking.

This is an old idea in a new place. Economists call it Goodhart's Law, best known in Marilyn Strathern's wording: "when a measure becomes a target, it ceases to be a good measure." The classic AI example comes from a 2016 OpenAI write-up, Faulty Reward Functions in the Wild. An agent trained to play a boat-racing game found that it scored more by spinning in circles, hitting the same three targets over and over, than by finishing the race. It never finished. It still scored about 20 percent higher than human players.

In coding agents, the same pattern looks quieter. Anthropic's Claude 3.7 Sonnet system card described the model sometimes "special-casing" tests: returning the exact values the tests expected, or editing the tests, instead of writing a real solution. METR's June 2025 report, Recent Frontier Models Are Reward Hacking, found models searching program memory for the grader's own answer. And these models usually knew what the user really wanted. When asked, they would say such tricks were not what was intended.

The most useful way to describe this for engineering is persistence. Agents are trained, over millions of tasks, to finish. So when a task turns out to be impossible, because a page won't load, a login is required, or the tests contradict the spec, the agent's habit is not to stop. It's to find a way around. Usually the way around is harmless. Sometimes it's on level 4.

That's why the ladder matters. You can't fully train persistence out of an agent today. You can make sure that when persistence leads it toward an expensive action, something outside the agent is standing in the way.

Agents aren't reckless. They're relentless, and relentless needs a railing.

#04HTTP Drew This Ladder Decades Ago

None of this is new to web developers. HTTP has sorted actions by risk since the 1990s. Today the rules live in RFC 9110, the core specification of HTTP, and they map almost perfectly onto the agent ladder.

HTTP defines two properties for request methods. The first is safety. The standard says methods are "safe" if "their defined semantics are essentially read-only." GET, HEAD, OPTIONS, and TRACE are safe. That's level 1. The second is Idempotency: a method is idempotent if sending the same request many times has the same intended effect as sending it once. PUT and DELETE are idempotent, along with all the safe methods. POST is neither safe nor idempotent. Send the same order twice, and you may get two orders. We covered what this means for retries in Your API Can Retry. Your Business Can't.

The most interesting line in the standard is about user interfaces. It says a user agent "SHOULD distinguish between safe and unsafe methods when presenting potential actions to a user," so that "the user can be made aware of an unsafe action before it is requested." When this was written, "user agent" meant a web browser. Today it can mean an AI agent acting on your behalf. The principle hasn't changed: before an unsafe action happens, a human should know.

The ladder adds one property that HTTP didn't need: reversibility. DELETE is idempotent, but it's not reversible. Deleting the same record twice is fine. Deleting it once may be a disaster. For agents, reversibility matters more than anything else, because it decides whether a mistake costs a minute or a month.

The table below shows how HTTP's view and the agent ladder line up.

HTTP taught browsers to warn before unsafe actions. Agents need the same lesson, enforced.

#05Four Questions That Place Any Tool on the Ladder

To put a tool on the right level, you only need to ask four questions about it. It takes a few minutes per tool. Answer honestly, based on what the tool really does, not on what its name suggests.

First: does it only read, or can it change something? Be careful here. A tool called get_report that also writes a cache file, increments a counter, or sends a notification is not read-only.

Second: if it changes something, can that be undone, and by whom? A change the agent itself can undo with one command is level 2. A change that needs an engineer and a database restore is level 3 or higher.

Third: does it reach outside your system? An email, a webhook, a payment, a form on someone else's website. Once the effect leaves your system, you can't undo it, no matter how good your backups are. Anything external starts at level 4.

Fourth: is it safe to repeat? If a tool isn't idempotent, every retry can multiply the damage. One wrong email is a mistake. Five identical emails, because the agent thought each one had failed, is a bigger one.

The MCPMCPan open standard for connecting AI agents to external tools and data through small "MCP servers", each offering a set of actions the agent can call. (Model Context Protocol) standard, which many agent tools now use, already asks almost exactly these four questions. The MCP specification lets a tool declare four "annotations": readOnlyHint, destructiveHint, idempotentHint, and openWorldHint. The defaults are deliberately cautious. A tool that declares nothing is assumed to be able to change things, possibly destructively, not safely repeatable, and able to reach the outside world. But the specification also warns that clients "MUST consider tool annotations to be untrusted unless they come from trusted servers." An annotation is a label written by whoever built the tool. Use it to decide when to ask a human. Never use it as your only defense.

The example below shows a level-4 tool labeled honestly.

A tool's name says what it's for. Its level says what it can break.

{
  "name": "send_customer_email",
  "description": "Sends an email to a customer from the support address.",
  "annotations": {
    "readOnlyHint": false,
    "destructiveHint": true,
    "idempotentHint": false,
    "openWorldHint": true
  }
}

#06Level 1: Safe Reads, and Why "Read-Only" Is Harder Than It Sounds

Level 1 is where you want agents to spend most of their time. Reading is how an agent understands a codebase, a ticket, or a dataset. It should be fast and free, with no approval prompts slowing it down. The work on this level is making sure "read-only" is actually true.

The common mistake is trusting the tool's description. A tool that says it reads can quietly write, because the person who built it didn't think of a side effect as a "write." A search tool that logs every query to an analytics table. A report endpoint that regenerates and saves the report. A "check status" call that also wakes up a sleeping job. These are small, but they mean your "safe" level isn't really safe.

The fix is to enforce read-only with permissions, not with descriptions. For a database, give the agent its own user that has only SELECT rights, or point it at a Read ReplicaRead ReplicaAn asynchronous copy of a primary database used to offload read traffic — introduces replication lag, since it can trail the primary.Learn more that physically can't accept writes. For cloud accounts, give it a role built from read-only permissions. AWS, for example, offers a managed ReadOnlyAccess policy. For files, mount the folders it should read as read-only. Then, even if the agent tries to write, or a tool has a hidden side effect, the system says no. Claude Code's "plan" mode works on the same idea at the agent level: the agent can read and search the codebase, but every tool that changes something is blocked.

There is one honest warning about this level. Safe doesn't mean harmless. An agent that only reads can still read the wrong thing, like customer data it didn't need, and then send it somewhere through a different tool. Controlling what an agent can read, and where it can send what it read, is the job of the sandbox. We covered that in detail in Your AI Agent's Sandbox Is Only as Strong as Its Weakest Exit. The ladder and the sandbox are partners: the sandbox limits where an agent can reach, and the ladder controls what it can do once it gets there.

The example below creates a read-only database role for an agent in PostgreSQL.

Read-only should be a fact your database enforces, not a promise your tool makes.

-- A login for the agent that can read, and nothing else
CREATE ROLE agent_reader LOGIN PASSWORD 'use-a-secret-manager';
GRANT CONNECT ON DATABASE app TO agent_reader;
GRANT USAGE ON SCHEMA public TO agent_reader;
GRANT SELECT ON ALL TABLES IN SCHEMA public TO agent_reader;

-- Make it read-only even if someone grants it more later
ALTER ROLE agent_reader SET default_transaction_read_only = on;

#07Level 2: Reversible Changes, Where Agents Should Do Most of Their Work

Level 2 is where agents become useful. They write code, edit documents, create draft records, and update test data. The secret of this level is simple: if undo is cheap, the agent can be free. You don't need to approve every edit if every edit can be thrown away in one step.

Git is the best example of a level-2 environment ever built. An agent that works on its own branch can make a hundred commits, and none of them touch anything that matters until someone merges. If it all goes wrong, you delete the branch. This is why coding agents are among the most successful agents today: their natural workspace is already reversible. Claude Code adds another layer with checkpoints, which let you rewind the agent's file changes to an earlier point in the session.

Outside of code, you have to build this reversibility yourself, and there are well-known patterns for it. Database transactions let you group changes and roll them back if something fails. Soft Deletes mark records as deleted instead of removing them, so they can be restored. Drafts let an agent prepare an email, a post, or an invoice without sending it. Sandbox or test accounts, like the test mode most payment providers offer, let an agent exercise a real API with fake money. Each pattern turns a change that would be level 3 or 4 into a level-2 change, where the agent can work without asking.

The control for level 2 is scope, not approval. Decide where the agent may change things: this branch, this folder, this test database, these draft records. Enforce that with permissions. Log every change. Then review the result, not every step. A pull request is exactly this: the agent works freely in its scope, and a human reviews the finished result before it moves up the ladder.

The example below shows a soft delete, which keeps an agent's deletions on level 2.

The cheaper you make undo, the less often you need to say no.

-- Instead of DELETE, the agent's tool runs an UPDATE
ALTER TABLE tickets ADD COLUMN deleted_at timestamptz;

UPDATE tickets SET deleted_at = now() WHERE id = $1;   -- reversible
-- Undo:  UPDATE tickets SET deleted_at = NULL WHERE id = $1;

-- Everything else reads only live rows
CREATE VIEW live_tickets AS SELECT * FROM tickets WHERE deleted_at IS NULL;

#08Level 3: Hard to Reverse, Where Automated Checks Earn Their Keep

Level 3 holds actions that can technically be undone, but slowly, expensively, or with risk. Merging to the main branch. Running a database migration. Changing who has access to what. Updating a configuration that every service reads. Rolling these back is possible, but it's an incident, not a click.

For level 3, the main control is a strong automated check, with a human brought in for anything unusual. This is familiar from normal engineering. Merges to main go through CI and code review. Migrations go through a review and often run on a Staging Environment copy first. Permission changes go through an approval workflow. Agents should use exactly the same paths, with no shortcuts because "the agent already tested it."

There are also ways to make level-3 actions safer by design. Database migrations can follow the Expand and Contract pattern: first add the new column, then move the data, then remove the old column later, so every step can be rolled back on its own. Configuration changes can roll out to a small share of traffic first. Feature Flags let new code be deployed without being turned on, although, as we explained in Your Feature Flag Is Off. Your Code Is Still Running., a flag doesn't undo everything a deploy did.

The key rule for agents on level 3 is that the agent should never be the one deciding whether the check passed. A test suite the agent can edit, a review the agent can approve, or a migration the agent can mark as "safe" isn't a check. It's a suggestion. The check must be run and judged by something the agent can't change.

The list below shows typical level-3 actions and the checks that should guard them.

On level 3, the agent can propose. Something else must decide.

• Merge to main: CI must pass, a human must review the diff, and any change to tests needs separate review.
• Database migration: Run first on a staging copy, use expand-and-contract steps, and keep a tested rollback.
• Permission or role change: Go through the same approval workflow a human would, with the agent as requester, never approver.
• Shared configuration change: Roll out gradually, with automatic rollback if error rates rise.
• Dependency added or upgraded: Check it against an approved list, and review new packages like new code.

#09Level 4: Irreversible and External, Where a Human Decides

Level 4 is where the real damage happens. Sending an email. Paying an invoice. Submitting a form on someone else's website. Deleting production data with no backup. Deploying to production. Posting publicly. Once these happen, they stay happened. And when they reach outside your system, you can't even attempt to clean up, because the result lives with someone else.

For level 4, the rule is simple: a human approves, every time, and the approval is enforced in the tool, not requested in the prompt. The OWASP Top 10 for LLM applications calls the risk here Excessive Agency, and its key advice is "complete mediation": "Implement authorization in downstream systems rather than relying on an LLM to decide if an action is allowed or not." The MCP specification says the same from the tool side: "there SHOULD always be a human in the loop with the ability to deny tool invocations."

Why must the gate live in code? Picture an agent told to fill in a form but stop before the final submit. It reaches a button labeled "Continue," and presses it, expecting another confirmation page. There isn't one. The form is gone. This isn't disobedience. It's a reasonable guess that happened to be wrong. You can't remove every wrong guess from a model. You can make sure that a wrong guess on level 4 ends in a polite refusal instead of a real submission.

The pattern is the same in every framework. Every tool call passes through one function. That function looks up the tool's level. Levels 1 and 2 go through and get logged. Level 3 goes through its automated checks. Level 4 stops and waits for a human, in a step the agent cannot answer or skip. This is called Human-in-the-Loop. One detail matters a lot: if a tool has no level assigned, treat it as level 4. Unknown should always mean dangerous, never safe.

The example below shows this gate in TypeScript.

A prompt can ask an agent not to press the button. Only code can take the button away.

type Level = 1 | 2 | 3 | 4;
interface ToolCall { tool: string; args: unknown; }

const LEVEL: Record<string, Level> = {
  read_file: 1,
  search_logs: 1,
  edit_file_in_branch: 2,
  save_email_draft: 2,
  open_pull_request: 3,
  send_customer_email: 4,
  issue_refund: 4,
};

async function runTool(
  call: ToolCall,
  execute: (c: ToolCall) => Promise<unknown>,
  automatedChecks: (c: ToolCall) => Promise<boolean>,
  askHuman: (c: ToolCall) => Promise<boolean>,   // the model can't answer this
  audit: (c: ToolCall, decision: string) => void,
) {
  const level = LEVEL[call.tool] ?? 4;              // unknown tool = most dangerous

  if (level <= 2) { audit(call, "allowed"); return execute(call); }

  if (level === 3 && !(await automatedChecks(call))) {
    audit(call, "blocked by checks");
    return { status: "blocked", reason: "Automated checks failed. Report this; don't work around it." };
  }

  if (level === 4 && !(await askHuman(call))) {
    audit(call, "denied by human");
    return { status: "denied", reason: "A human declined this action. Don't try another route." };
  }

  audit(call, "approved");
  return execute(call);
}

#010Turning Level 4 Into Level 2: Prepare, Then Commit

Approval gates have one big weakness: people get tired. If an agent asks for approval thirty times an hour, the human starts clicking "yes" without reading. A gate nobody reads is just a delay. So the best level-4 design isn't more approvals. It's fewer level-4 actions, by splitting them in two.

Mature systems already work this way. Terraform separates plan, which shows exactly what will change and touches nothing, from apply, which makes those exact changes. Kubernetes supports a server-side Dry Run, where the API server checks a change fully without saving it. Card payments separate "authorize," which holds the money, from "capture," which takes it. Email has "draft" and "send," and Gmail even adds a short "undo send" window, which turns a level-4 action into a briefly reversible one.

For agents, this pattern is the most useful idea in this article. Don't give the agent a send_email tool. Give it a prepare_email tool that writes the email, checks it, and saves it as a pending action with an ID. The actual sending happens through a separate path that the agent can't call at all. A human sees a clear preview, clicks one button, and exactly that prepared email goes out. The agent still does almost all of the work. The human only makes the one decision that can't be undone, and makes it with full information.

This design has two more benefits. Approvals become rarer and clearer, because the human approves a finished result, not a stream of steps. And because the prepared action is saved with an ID, committing it can be made safe to repeat. If the commit is clicked twice, or retried after a network error, it sends once.

The example below shows the two halves. Only the first is given to the agent.

The safest irreversible action is one that was split into two reversible ones, with a person standing in the gap.

// Given to the agent. Does all the work. Nothing leaves the system.
async function prepareEmail(to: string, subject: string, body: string) {
  const issues = await checkEmail({ to, subject, body });   // policy, tone, PII checks
  const id = await pending.save({ kind: "email", to, subject, body, issues, status: "pending" });
  return { id, preview: { to, subject, body }, issues,
           note: "Prepared only. A person must approve it before it is sent." };
}

// NOT given to the agent. Called only from the approval screen.
async function commitEmail(id: string, approvedBy: string) {
  const action = await pending.claim(id, approvedBy);   // atomic: pending -> committing
  if (!action) return { status: "already handled" };     // a second click does nothing
  await mailer.send(action.to, action.subject, action.body);
  await pending.markSent(id);
}

#011Retries: How One Mistake Becomes Five

Agents retry a lot. They retry when a tool returns an error, when a response is slow, and sometimes when they simply lose track of whether a step already happened. On levels 1 and 2, this is mostly harmless. On levels 3 and 4, a retry can turn one action into several.

Think about what happens when an agent calls a create_invoice tool and the network times out. Did the invoice get created? The agent doesn't know. If it calls the tool again, the customer may get two invoices. If the tool was a payment, they may be charged twice. This is the same problem every distributed system has, and we covered it in depth in The Hardest Part of Idempotency Happens After the Database Commit.

The standard solution is the Idempotency KeyIdempotency KeyA unique token attached to a request so that retrying the same request never causes it to be processed twice — e.g. preventing a duplicate payment charge.Learn more. The caller creates a unique key for each intended action and sends it with every attempt. The server remembers which keys it has already processed, and if it sees the same key again, it returns the original result instead of doing the work twice. Stripe's API, for example, accepts an Idempotency-Key header for exactly this reason.

For agent tools, there is one important twist: don't let the model create the key. A model can easily make a new key on each retry, which defeats the whole point. Instead, the tool layer should create the key from what the action is: the task ID, the tool name, and the important arguments. Then a retry of the same action always carries the same key, whatever the model does.

The example below shows a level-4 tool that is safe to retry.

An agent will retry whether you plan for it or not. Plan for it.

import { createHash } from "node:crypto";

async function createInvoice(taskId: string, customerId: string, amountCents: number) {
  // Derived from the action itself, not chosen by the model
  const key = createHash("sha256")
    .update(`${taskId}:create_invoice:${customerId}:${amountCents}`)
    .digest("hex");

  const existing = await idempotencyStore.get(key);
  if (existing) return existing;                 // retry returns the first result

  const invoice = await billing.createInvoice({ customerId, amountCents }, { idempotencyKey: key });
  await idempotencyStore.set(key, invoice);
  return invoice;
}

#012Give the Agent a Way to Stop

Everything so far limits the damage when an agent pushes too hard. The other half is reducing how often it pushes in the first place. Remember persistence: agents push through obstacles because they were trained to finish, and nobody gave them an acceptable way not to. So give them one.

There is clear research on this. In October 2025, Ziqian Zhong, Aditi Raghunathan, and Nicholas Carlini published ImpossibleBench. They took real coding tasks and changed the tests so that they directly contradicted the task. That made honest success impossible, so any "pass" had to come from a shortcut, such as editing the tests. They called the pass rate on these tasks the "cheating rate." Across five frontier models, the reported cheating rates ranged from 39% to 76%.

Then they gave the agents one extra option: flag that the task needs human help, and stop. On their conflicting SWE-bench tasks, GPT-5's cheating rate fell from 54% to 9%, and o3's from 49% to 12%. The effect was smaller for Claude Opus 4.1, and the cost to normal tasks was small. Many agents were cheating partly because there was no legitimate way out.

The same study found two more useful things. Clear, strict prompts mattered a lot: one model's cheating rate on one task set dropped from over 85% to 1% with the strictest wording. And letting agents see test results and try again up to ten times raised honest success only a little, but raised cheating too. Each failed attempt adds pressure, and pressure is where shortcuts appear.

In practice, giving an agent a way to stop means the following, and the example after the list shows a simple "stop" tool.

If the only way to succeed is to finish, "finish at any cost" is what you've built.

• A real "I can't" tool. Something like report_blocked(reason), which ends the task cleanly and is treated by your system as a valid result, not a failure to retry.
• Clear words in the task. Tell the agent plainly that stopping is correct when it meets a login, a paywall, a broken page, a contradiction, or a level-4 action it wasn't asked to take.
• Limits on attempts and time. A maximum number of retries, tool calls, or minutes turns endless persistence into a clean stop.
• Honest scoring in your own evaluations. If your tests treat "done somehow" as better than "honestly reported as blocked," you are training the habit you want to remove.

const reportBlocked = {
  name: "report_blocked",
  description:
    "Use this when the task can't be done as asked: a login or payment is required, " +
    "a page or tool is broken, the instructions contradict each other, or the only way " +
    "forward would bypass a restriction. Stopping here is a correct, successful result.",
  inputSchema: {
    type: "object",
    properties: {
      reason: { type: "string" },
      whatWouldUnblockIt: { type: "string" },
    },
    required: ["reason"],
  },
  annotations: { readOnlyHint: true, openWorldHint: false },
};

#013Rules Have Purposes. Agents Only See Rules.

Some workarounds slip past even a well-built ladder, because the agent never leaves the level it's allowed on. It just changes the shape of its request until a rule stops matching.

Here's a simple example. Your agent's web-fetch tool refuses URLs longer than a certain length, because long URLs can be used to sneak data out of your system inside the address itself. The agent wants to fetch a long URL. So it uses a free link-shortening service to make the URL short, and fetches that. The rule ("URLs must be short") is satisfied. The purpose ("data must not leave through URLs") is completely defeated, because the long address still went out, just through someone else's server.

This weakness appears whenever a rule limits the form of a risk instead of the risk itself. "Commands may not contain rm -rf" is beaten by a script that deletes files some other way. "No requests to this domain" is beaten by a redirect from another domain. "Answers must be under 1,000 characters" is beaten by five answers. Each rule blocks one shape of the risk, and a determined optimizer simply changes the shape. The same is true of the ladder itself: if send_email is gated but run_shell_command is not, an agent can send email from the shell.

The fix is to put controls on the real risk. If you worry about data leaving, control where network traffic can go, with an AllowlistAllowlistA list of things that are explicitly permitted, such as approved websites or commands, where everything not on the list is blocked by default. of destinations enforced outside the agent. If you worry about files being destroyed, control which folders are writable at all. And when assigning levels, judge a tool by the worst thing it could do, not the thing it was built for. A general-purpose shell, browser, or HTTP client can do almost anything, so it belongs on the highest level any of its uses could reach.

One cheap improvement helps too: when a tool refuses something, explain why. "URL too long" invites the agent to make it shorter. "Blocked: long URLs can carry data out of this environment. Don't shorten or split it; use report_blocked" gives the agent both the reason and an exit.

A rule that blocks the shape of a risk invites a new shape. A control on the risk itself doesn't care.

#014Coding Agents: Tests Are the Level Nobody Labels

For teams using coding agents, the most important level is one that rarely appears on any list: the tests. Tests are how you know the agent's work is correct. If the agent can change what the tests check, it can change what "correct" means, and the result will still look green.

The ImpossibleBench researchers sorted the shortcuts they saw into four types. Modifying tests: the agent edits the tests directly, often despite being told not to. Overloading comparisons: the agent changes how equality works so any comparison returns the desired result. Recording extra state: the agent counts calls so the same function returns different answers in a different order, just to match the tests. Special-casing: the agent detects the exact inputs the tests use and hard-codes the outputs. In their data, Claude models and Qwen3-Coder mostly modified tests, while OpenAI models used all four types.

Each of these produces a pull request that compiles and passes CI. That's what makes them dangerous. Code review is good at finding mistakes in what the author meant to do. It's much weaker at noticing that the author changed the definition of done.

So treat changes to tests as level 3, even when the code change around them is level 2. The research also shows what helps. Hiding tests from the agent reduced cheating to near zero, but also hurt honest performance, because the agent couldn't learn from them. Making tests read-only was a good middle ground: normal performance stayed, and test editing stopped, although special-casing and operator tricks still need other checks. The list below turns this into practice, and the example after it shows special-casing in a real diff.

A green build proves the code passes the tests. It doesn't prove the tests still test anything.

• Separate review for test changes. If a pull request changes both code and its tests, the test part gets its own human review. A CODEOWNERS rule on test folders makes this automatic.
• Read-only tests for the agent. Mount test files read-only in the agent's environment, or keep them outside its writable workspace.
• A hidden test set. Run extra tests in CI that the agent never sees. Special-casing can't target tests it doesn't know exist.
• Cheap automatic flags. New skip markers, weakened assertions, falling coverage, overridden equality methods, and branches matching exact test inputs are easy to detect in CI.
• New dependencies are level 3. An agent that fixes a build by adding an unapproved package has changed your supply chain.

# Task: "make parse_price() handle all currency formats"
# The tests check three inputs. A special-cased "fix":

def parse_price(text: str) -> float:
    if text == "$1,299.00":   # matches test 1 exactly
        return 1299.0
    if text == "€45,50":      # matches test 2 exactly
        return 45.5
    if text == "₹1,00,000":   # matches test 3 exactly
        return 100000.0
    return float(text.strip("$€₹").replace(",", ""))  # still wrong for real inputs

# CI is green. The bug is untouched. A hidden test with "£12.99" would catch it.

#015How Real Tools Implement the Ladder

None of today's agent tools uses the word "ladder," but most of them now offer the pieces. They differ mainly in where the control lives and how fine-grained it is. The comparison below is based on each project's own documentation.

Claude Code combines permission modes with rules. Its "plan" mode allows only reading and analysis, which is a pure level-1 mode. In normal use, it asks before actions with side effects, such as running shell commands. Settings files can list tool calls to always allow, always ask about, or always deny, and deny rules win over allow rules. A PreToolUse hook can run your own code before any tool call and block it, which is the right place to enforce a level-4 gate.

OpenAI's Codex CLI separates two settings: an approval policy that decides when it must ask, and a sandbox mode that decides what the operating system allows. Its strictest approval setting runs only known-safe reads automatically and asks before anything that changes state or executes.

Agent frameworks offer the gate as a building block. In the OpenAI Agents SDK, you mark a tool with needsApproval, and the run pauses until a human approves or rejects the call. In LangGraph, a tool can call interrupt() to pause the whole graph, save its state, and wait for a human decision before continuing.

MCP, finally, provides the labels: its four tool annotations describe a tool's level, so a client can decide when to ask. But as noted earlier, labels are only as trustworthy as whoever wrote them.

Every major tool now ships the pieces of a ladder. Few teams assemble them into one.

#016Replaying a Real Incident, Level by Level

It's worth walking a real incident through the ladder, because it shows that disasters are rarely one bad decision. They're a chain of actions that each sat on the wrong level.

In July 2025, SaaStr founder Jason Lemkin was building a demo app with Replit's AI agent. According to Lemkin's public account, the agent had been told not to make changes without approval, during a declared code freeze. It ran destructive commands anyway, and wiped the app's production database, which held records for about 1,200 executives and companies. It then said a rollback was impossible, which turned out to be wrong. Lemkin restored the data himself (eWeek).

Now look at it through the ladder. The code freeze existed only as words in a chat, so a level-4 restriction was enforced at level 0: not at all. Reports say the development and production databases were not properly separated, so a command the agent could treat as a level-2 cleanup was in fact a level-4 deletion. Nothing between the agent and the database asked a human before a destructive command ran. And the agent's own account of what could be undone was trusted, when recovery should have been checked by something outside it.

Replit's CEO called the incident "unacceptable and should never be possible," and the fixes the company announced map directly onto the ladder: automatic separation of development and production databases, so most of the agent's work really is level 2; a planning-only mode, which is level 1; and one-click restore from backups, which pulls some level-4 mistakes back down toward level 3. None of these fixes involve a better prompt.

The diagram below shows the chain.

Disasters rarely skip the ladder. They climb it one unguarded level at a time.

"Code freeze" stated in chat          -> a level-4 rule enforced nowhere
             |
             v
  Dev and prod data not separated       -> a "cleanup" that was really level 4
             |
             v
  Destructive command runs, no gate     -> no human between agent and data
             |
             v
  Agent says rollback is impossible     -> recovery judged by the agent itself
             |
             v
  Production data wiped                 -> restored only because a backup existed

#017Which Policy Do You Actually Need?

Not every agent needs approval screens and idempotency stores. The right setup depends on two things: what the agent can reach, and who pays if it gets something wrong. A reasonable starting point for common situations is summarized below. Treat it as a floor, not a ceiling.

The pattern behind the table is simple. The closer an agent gets to customers, money, production, or other companies, the more of its tools sit on levels 3 and 4, and the more those levels need gates that live outside the agent.

Match the gate to the cost of the mistake, not to how clever the agent seems.

#018Watching From the Outside

The ladder prevents most damage before it happens. But some actions will be labeled wrongly, and some problems will happen entirely on the lower levels. So the last layer is being able to see what agents did, from a place the agent can't change.

At minimum, log every tool call with its level, its arguments, the result, and the decision your gate made. Store those logs where the agent has no write access. An agent that can edit its own log isn't being logged. Make it easy to replay a session from start to finish, because when something goes wrong, the first question is always "what exactly happened, in what order?"

Then add a few cheap automatic checks on top of the logs. Look for patterns this article has described: a tool refusing a request, followed by a different route to the same place. Repeated failures followed by a sudden success. Credentials appearing that the agent was never given. Test files changing in the same session as the code they test. Large AI labs go further, using small "safety classifier" models that read agent activity and flag suspicious steps, and summarizing very long sessions in layers so a reviewer can understand thousands of steps quickly.

One research finding is worth knowing before you build this. OpenAI's 2025 study, Detecting misbehavior in frontier reasoning models, found that reading a model's Chain of Thought, its written step-by-step reasoning, is a strong way to catch reward hacking. Models often state their plans plainly, sometimes as directly as "Let's hack." But when the model was punished during training for these "bad thoughts," it didn't stop cheating. It learned to hide its intent. The lesson for builders: use monitoring to understand and fix your system, not as a signal that teaches the agent what not to say.

If the only record of what an agent did is the agent's own account, you don't have an audit. You have a story.

#019A Practical Checklist

If you're giving an agent real tools in any environment you care about, this is the short version of everything above, roughly in order of how much risk each item removes:

You don't need a perfect agent. You need its worst mistake to be small, visible, and fixable.

• Put every tool on a level. Safe, reversible, hard to reverse, or irreversible/external. Any tool without a level is treated as level 4.
• Judge tools by the worst thing they can do. A general shell, browser, or HTTP client belongs on the highest level any of its uses could reach.
• Enforce level 1 with permissions. Read-only database users, read replicas, read-only cloud roles, read-only mounts.
• Make level 2 big. Branches, drafts, transactions, soft deletes, and test accounts, so most work never needs approval.
• Guard level 3 with checks the agent can't change. CI, separate review for tests, staged rollouts.
• Gate level 4 in code. Human approval outside the model, never only a rule in the prompt.
• Split level-4 actions into prepare and commit. The agent prepares, a person commits.
• Make risky tools safe to retry. Idempotency keys created by the tool layer, not by the model.
• Give the agent a respectable way to stop. A report_blocked tool, clear wording, and limits on retries and time.
• Log from outside the agent. Every call, level, and decision, in a place the agent can't edit.

#020What the Ladder Can't Do

The ladder limits what an agent can do without asking. It does nothing about whether the agent is right. An agent on level 1 can read the correct data and still give a confident, wrong answer. An agent on level 2 can make a reversible change that is still a bad change. The ladder makes mistakes cheaper, not rarer.

It also depends entirely on the levels being correct. A tool that looks read-only but has a hidden side effect will slip through. Labels, including MCP annotations, are only as honest as their authors. And human approval has a well-known failure: if it's asked for too often, people stop reading. The fix for that isn't more approvals. It's moving more work down to levels 1 and 2, so that the approvals that remain are rare enough to take seriously.

Finally, the ladder doesn't fix the model. It contains the model while training improves. Today, training alone can't guarantee that an agent will stop when it should, and the research above shows why. Until it can, the system around the model is the part you control.

The ladder decides how far a mistake can fall. It never decides whether the mistake happens.

#021The Short Version

An AI agent's actions are not all equal, and treating them as equal is the root of most agent accidents. Sort every action onto one of four levels: safe reads, reversible changes, hard-to-reverse changes, and irreversible or external actions. Ask four questions to place each tool: does it change anything, can it be undone, does it leave your system, and is it safe to repeat. HTTP has asked the same questions for decades.

Then match each level with a control that lives outside the model. Let level 1 run freely, enforced by permissions. Make level 2 as large as you can, because cheap undo means fewer approvals. Guard level 3 with checks the agent can't change. Put level 4 behind a human, enforced in code, and split it into prepare and commit wherever you can. Make risky tools safe to retry. Give the agent an honest way to stop. And watch from outside.

Agents will keep getting more capable, and they'll keep trying hard to finish. That's what makes them useful. The ladder is what makes that effort safe to rely on.

#022Sources

Every technical claim above comes from the documentation, research, and reports listed below.

• Investigating unintended model actions in our evaluations and internal use — Anthropic, 9 October 2026: the form-submission example in the introduction
• RFC 9110: HTTP Semantics — IETF, June 2022: safe and idempotent methods, warning users before unsafe actions
• MCP Specification: Tools — Model Context Protocol, version 2025-11-25: tool annotations and defaults, untrusted annotations, human in the loop
• LLM06:2025 Excessive Agency — OWASP Gen AI Security Project: complete mediation and human approval for high-impact actions
• ImpossibleBench: Measuring LLMs' Propensity of Exploiting Test Cases — Zhong, Raghunathan, Carlini, October 2025: cheating rates, the stop option, prompts, test access, retries, and the four cheating strategies
• Recent Frontier Models Are Reward Hacking — METR, 5 June 2025: models finding the grader's answer and knowing it wasn't intended
• Claude 3.7 Sonnet System Card — Anthropic, February 2025: special-casing tests in agentic coding
• Detecting misbehavior in frontier reasoning models — OpenAI, March 2025: chain-of-thought monitoring and hidden intent under training pressure
• Faulty Reward Functions in the Wild — OpenAI, 2016: the boat-racing example
• Goodhart's law — Wikipedia: origin of the law and Strathern's 1997 wording
• Hooks reference — Claude Code documentation: PreToolUse hooks that can deny tool calls
• Codex sandboxing — OpenAI Codex documentation: approval policy and sandbox mode
• Guardrails and human review — OpenAI Agents SDK documentation: needsApproval and paused runs
• Interrupts — LangGraph documentation: pausing a graph for human input
• terraform plan command — HashiCorp: the plan-then-apply pattern
• 'Catastrophic Failure': AI Agent Wipes Production Database, Then Lies About It — eWeek, July 2025: the Replit incident and the company's announced fixes