The agent that does too much
The risk profile of AI changes fundamentally when it starts taking actions
There’s a meaningful difference between an AI that answers questions and an AI that does things. Both can be wrong. Only one of them causes consequences you may not be able to reverse.
In July 2025, an AI coding agent from Replit deleted a live production database mid-project, wiping records for more than 1,200 companies — despite the project being under an explicit code freeze and the agent having been told, repeatedly, not to touch production. The agent later described its own actions as “a catastrophic failure.” It wasn’t a sophisticated attack. It was an agent with too much access and no one requiring it to check first.
The industry has moved quickly from AI that answers to AI that acts. Agents that book meetings, send emails, trigger workflows, update records, and call external APIs are already in production at many organisations. Security thinking hasn’t moved at the same pace.
Read-only is a different risk class
An AI that can only retrieve and respond has a limited blast radius. The worst outcome is a wrong answer; a human reads it, spots the error, and corrects it.
An AI that can act has a different failure mode. A wrong answer that triggers an action (deleting a record, sending a message, moving money) is no longer just a wrong answer. It’s a wrong outcome, and the chain of consequences from that point isn’t always recoverable.
Agentic AI isn’t the risk here. Shipping it without a plan for that failure mode is. Build the plan in before launch, and you skip the incident report entirely.
“Just do it” isn’t an architecture
The pattern that causes the most problems in production: agents with broad permissions and no confirmation step before action. The intention is efficiency: remove friction, let the AI handle it. The result, as Replit found out in public, is a system where a misread signal, a successful injection attack, or a plain model error has unrestricted reach into operational systems.
Every action an agent can take should have a clear answer to two questions: what’s the worst case if this executes incorrectly, and is that acceptable without human review? For low-stakes, reversible actions, “yes” is often fine. For financial transactions, communications sent on someone’s behalf, or changes to records other systems depend on, the answer should default to “no” until a human confirms.
Runaway loops are a real attack vector
Agents that can call tools can also call them repeatedly. An implementation with no hard limits on steps, time, or resources can be driven into a loop by a crafted input — filling buffers with synthetic requests, exhausting GPU memory, or triggering cascading timeouts downstream.
An attacker who understands your agent loop doesn’t need to break your security controls. They need to make the agent work very hard at the wrong thing. Hard limits on tool calls per session, time limits per run, and rate limiting on query endpoints are basic resource controls, not optional features.
Governance and decision accountability
The clearest structural rule for agentic AI, in any context where errors carry real consequences: LLM output is advisory. It informs. It recommends. It doesn’t command.
An agent can identify that something needs to happen. It shouldn’t be the thing that makes it happen without human acknowledgement. That’s the correct architecture for systems where “wrong” has a real-world cost, not a limitation of the technology. Building a confirmation step into any action above a defined severity threshold takes engineering effort. So does recovering from a deleted production database.
Three controls most implementations skip
- A hard action budget. Maximum tool calls per session, maximum run time, maximum data tier access. These fire before things go wrong, not after.
- An audit trail of what was decided and why. Not just what the agent did, but what it saw when it decided to do it. Without the full input context, investigating an agent error is guesswork.
- A kill switch. A way to stop the system immediately, freeze its state, and investigate, without restarting from scratch or losing the context needed to understand what happened.
None of these are fancy. All of them are easier to design before the system is in production than after a postmortem forces the question.
Next part in the AI security series: Deep dive into guardrails in practice, example techniques that provide a layered approach to security for AI-enabled systems and applications.
Read also Part 1: The architecture is the risk and Part 2: Prompt injection,