What Actually Breaks: Field Notes on AI Agents

Between April and October 2026 I moved most of my day-to-day engineering work onto AI coding agents. Output went up a lot; the numbers and how I measured them are in the case study. This post is about the other column: the things that broke.

None of them broke because the model was not smart enough.

Three failures from my own rollout

1. A scheduled agent posted my private brief to a team channel. I had a weekday-morning routine that pulled together open work, the PR queue and decisions waiting on me. One step posted to a team chat channel. It posted four times, with internal work items in it. I edited the prompt, but a run that had already started kept the old instructions and posted again almost a day later. The fix was to remove the posting step entirely, check the next run's transcript to confirm it was gone, move the brief to a local file, and set a standing rule: agents post to chat only when I ask for a specific message to a specific channel.

2. An agent invented a ticket ID, and it reached published release notes. With no real ticket to cite, the agent made one up. The fake ID spread into commit messages, PR descriptions, and the changelogs of two published package versions, which cannot be edited. Agents now have to look an ID up in the tracker before citing it.

3. Parallel sessions overwrote each other's work. I often ran two or more agent sessions at once. When they shared one git checkout, they overwrote each other more than once. Each session now gets its own git worktree, and commits name files explicitly.

There was a quieter fourth. My "router" session, built to classify requests and hand them off without doing heavy work itself, had drifted into running 19 of the last 30 sessions, with one conversation past 800 turns. I only found out because I reviewed the logs.

Look at what those have in common. The posting incident is an agent with write access to a shared channel and nobody gating it. The fake ticket is an agent filling a gap confidently, with nothing checking the output before it left the system. The worktree problem is two workers sharing mutable state. The router is drift that nobody was measuring. Every one is a distributed-systems or operations problem with a language model in the middle.

The arithmetic that makes this worse

Take an agent that gets each individual step right 95% of the time. Give it a twenty-step workflow: pull the ticket, look up the customer, check entitlement, query three systems, reconcile the answers, draft a response, post it.

0.95 to the twentieth power is 0.358. Your good agent finishes the whole job correctly about 36% of the time.

At 99% per step, a 20-step chain lands at 82%. At 99.9%, it is 98%. You do not get to production by making the agent smarter. You get there by adding nines to each step, and by making the failures that remain cheap.

What the industry numbers say

Gartner's widely quoted forecast is that more than 40% of agentic AI projects will be cancelled by the end of 2027, citing escalating costs, unclear business value, or inadequate risk controls. None of those causes is "the model wasn't good enough."

The cynical read is also wrong. Agents do run at scale for companies that treat them as an engineering problem. Salesforce reported 2.4 billion Agentic Work Units across Agentforce and Slack in its Q4 FY26 results, up 57% quarter on quarter. On its April 2026 earnings call, Microsoft said tens of thousands of companies were already managing tens of millions of agents in Agent 365, its agent control plane. ServiceNow said its AI products crossed $1 billion in annual contract value in Q2 2026.

So the technology ships. What separates the projects that work from the ones that get cancelled is mostly unglamorous discipline.

Four ways agents break

There is a useful academic version of this. "Beyond the Leaderboard", a July 2026 synthesis by Albayaydh, Zhao and Flechais, reviewed 27 papers covering 19 benchmarks and grouped agent failures into six clusters: tool invocation errors, planning failures, long-horizon degradation, multi-agent coordination failures, safety and security failures, and measurement validity problems. Its sharpest finding is that strong performance on individual sub-tasks does not reliably translate into end-to-end success.

My working version collapses to four.

1. Tool-call errors and silent failures

The model calls your API with a malformed parameter. Or the right parameter and a stale ID. Or correctly, and the API returns a 200 with an empty payload because an upstream cache expired.

The model does not know any of this went wrong. It treats whatever came back as ground truth and carries on. A silent failure is worse than a crash: a crash gets logged and paged, while a silent failure gets summarised into a plausible answer. The invented ticket ID is the same shape from the other direction: the agent had nothing to cite, and instead of saying so it produced something that looked right.

2. Context and memory drift

On a long task, the agent's working context fills with its own intermediate output. Twenty tool calls in, the original instruction is buried, and it starts optimising for the wrong objective. The same thing happens one level up. My router did not decide to become a mega-session; it drifted there one reasonable request at a time.

3. Compounding error rates

This is the 36% above. Toby Ord's paper "Is there a half-life for the success rates of AI agents?" models agents as having a roughly constant failure rate per minute of equivalent human task time, which produces exponentially declining success as tasks get longer.

METR's time-horizon research makes the same point with data, and the detail people miss is the gap between reliability thresholds. In METR's original 2025 paper, Claude 3.7 Sonnet managed about 59 minutes of human-equivalent work at 50% success, but only about 15 minutes at 80%. Frontier 50% horizons have since climbed into the hours, but the shape holds. The distance between "works half the time" and "works reliably" is most of the problem.

4. Over-broad permissions and nobody owning them

Gravitee's State of AI Agent Security 2026, based on a survey of 750 executives and technical leaders in April 2026, found that 54% of organisations had experienced or suspected an AI agent security or data privacy incident in the previous twelve months. Excessive permissions and over-privileged access were one of the recurring incident patterns.

The number that stopped me: only 7.2% had a named individual accountable for agent behaviour, and only 19.7% said all their agents were fully secured and governed before going live.

My channel-posting agent fits here. It had a write permission it did not need for its real job, and nobody had decided who signs off on what it publishes. That is the lesson I wrote down at the time: a scheduled agent with write access to a shared channel is a publishing system, and should be treated like one.

The reliability stack

None of the fixes are exotic. They are the patterns we already apply to any unreliable distributed dependency, which is what a model is.

Retries and idempotent tools. Every tool should be safe to call twice. Give writes an idempotency key, so "create ticket" returns the existing ticket rather than opening a duplicate. Then retry transient failures, because a retried timeout is the cheapest nine you will buy. And make failures loud: a tool returning an empty result must say so in a form the model cannot mistake for success.

Verify anything that leaves the system. IDs, numbers, URLs and root causes get checked against their source before they reach a commit, a customer or a changelog. After two root causes I had filed turned out wrong when re-measured, I added a second rule: re-measure the cause before implementing the fix.

Checkpoints and approval gates. Break long chains into segments with durable state between them, so a failure at step 14 resumes from step 13. Then gate anything irreversible or public: refunds, external messages, production writes, posts to shared channels. The gate turns the worst outcome from "the agent did it" into "a human said no."

Isolation for parallel work. Agents running at the same time need separate workspaces, the same way two developers need separate branches. Two agents editing the same files do not get a merge conflict. They get a silent disagreement that neither of them notices.

Evals as regression tests. Build a fixed scenario set with known-good outcomes, run it on every prompt, model and tool change, and gate deploys on it. A model version bump is a dependency upgrade. Treat it like one.

Observability. Log every tool call: inputs, outputs, latency, retries, token spend. When an agent does something strange at 3am, you need the trace, not the final answer. Then actually read the logs on a schedule. My router problem was in the logs for weeks before I looked.

Protocols and plumbing

A large share of agent fragility is integration fragility, which is why the standards matter.

MCP has become the default for the tool-access layer. When Anthropic donated it to the Linux Foundation's new Agentic AI Foundation in December 2025, it cited more than 10,000 active public MCP servers and over 97 million monthly SDK downloads, with support in ChatGPT, Gemini, Microsoft Copilot, Cursor and VS Code. If you have not built one, the MCP server tutorial takes an afternoon.

On the agent-to-agent side, the Linux Foundation reported in April 2026 that A2A had passed its first year with more than 150 supporting organisations and integrations in the major cloud platforms, following a v1.0 stable specification in March.

The reliability argument for both is simple: a standard interface is a testable interface. Bespoke glue code between every agent and every system is where undocumented behaviour lives.

The fair caveat: neither protocol solves meaning or authority. Both move requests reliably; neither guarantees the agent on the other end understood your intent, or tells it whose authority it is acting under. Plumbing is necessary, not sufficient. I go further into the identity problem in why your AI agents don't talk to each other.

Governance before scale

Four things, and none of them need a committee.

Scoped credentials. Each agent gets its own identity with least-privilege access to only the systems its job requires. Not a shared service account. Not the developer's token.

Audit trails. "Which agent changed this record, on whose behalf, and why" must be answerable in minutes.

Kill switches. One control that halts a class of agents immediately, tested before you need it. My posting incident showed why: editing the prompt did not stop a run that was already in flight.

A named owner. Per agent. See the 7.2% figure above.

On regulation, the EU's Digital Omnibus on AI entered into force on 27 July 2026. It moved high-risk obligations for Annex III systems to 2 December 2027, and to 2 August 2028 for AI embedded in regulated products. Article 50 transparency obligations still applied from 2 August 2026, so if your agent talks to EU users or generates synthetic media, nothing moved for you (details here).

The counterargument: "just wait for smarter models"

This pushback is not stupid. METR's original work found frontier time horizons doubling roughly every seven months. If that continues, some of today's per-step error rate will go away on its own, and some scaffolding we build now will be thrown away. Capability growth will fix part of categories one and two.

But look at my four incidents. A smarter model would not have removed a posting permission it should never have had, stopped a run already in flight, given two sessions separate checkouts, or noticed that my router had drifted. Gartner's cited causes are cost, unclear value and risk controls. Gravitee's most damning finding is that almost nobody has a named owner.

There is also a trap in waiting. Even at 99.9% per step you are only at 98% across twenty steps, and the tasks we hand agents grow at least as fast as the models improve.

Waiting is a strategy for the capability half of the problem. It is no strategy at all for the ownership half.

The production-readiness checklist

Ten questions. If you cannot answer all of them with a yes, you have a demo.

Named owner. One person accountable for this agent's production behaviour?
Scoped identity. Its own credential with least-privilege access, not a shared or human account?
Idempotent tools. Every tool safe to invoke twice without duplicating an effect?
Loud failures. Every tool returns an unambiguous error the model cannot mistake for a valid empty result?
Verified outputs. IDs, numbers and facts checked against the source before they leave your system?
Eval suite. A fixed scenario set with known-good outcomes gating every prompt, model and tool change?
Approval gates. Every irreversible, public or customer-facing action passes a human or a hard policy check?
Isolation. Parallel agents work in separate workspaces, never on shared mutable state?
Full-trace observability. Every tool call logged, any run reconstructable, and someone reads the logs on a schedule?
Tested kill switch. You can stop this agent, including runs already in flight, and you have actually tried?

Items 1, 9 and 10 are usually the last ones anybody builds, and they decide whether your first serious incident is a bad afternoon or a board conversation.

Agent reliability is not a frontier research problem. It is the operational discipline we already apply to databases, payment gateways and deploy pipelines, pointed at a component that happens to be probabilistic.

What Actually Breaks: Field Notes on AI Agents

Three failures from my own rollout

The arithmetic that makes this worse

What the industry numbers say