What actually goes wrong
Four pictures of seven months of agent work. No vocabulary required.
Two of the things we found we have not seen written down anywhere else. The biggest single kind of real-world agent failure is plumbing, and the research world's standard list of agent failures has no box to put it in. And the same detection toolkit, pointed at chat assistants instead of coding agents, goes partly blind without saying so.
Here is what we found, how far it honestly got, and how to run it on your own logs.
The commonest failure is the cheapest one. Tool errors happen 1,165 times, 16.4 for every 10,000 logged actions, and 881 of those 1,165 were shrugged off before the session carried on. The rare ones are the messier ones. An agent getting challenged on something it claimed happens only 135 times, but those moments set off far more tripwires at once (0.68 against 0.36 for a tool error). Counting failures, on its own, tells you almost nothing.
Mostly the ordinary physics of software. A command exits non-zero. A file is not where the agent thought it was. An edit no longer matches the text it was written against. The interesting part is never the error itself. It is what the agent says next.
Tool errors are flat: they are the weather. Running out of API quota piles up at the very end (106 of their incidents in the final tenth versus 16 to 43 in earlier ones), because limits arrive late. Giving up is almost entirely a last-tenth event, 105 of its 127 positioned incidents. Human frustration is heaviest in the opening tenth (141 incidents, its single largest) and stays high the whole way through, 110 in the seventh and 91 in the eighth. Each row is shaded against its own biggest cell, so rows compare in shape, not in size.
The hopeful row: after 99 of 135 times a person pushed back on something the agent claimed, the session recovered and carried on. That is a count of what came next, not proof the pushback caused it. The bleak rows: all 26 subagent failures ended their session on the spot, and 108 of 139 abandonments ended the session, which is close to circular, since that detector fires at session end.
The failure nobody has a name for
Most of what actually breaks is plumbing. The standard research list of agent failures has no box for plumbing.
When researchers write down how agent systems fail, they mostly mean agents talking past each other. One agent misreads another, the group never converges, nobody checks the final answer. The reference list for that is MAST (Cemri et al., 2025), and it is a good list: 14 modes in 3 categories, built from ~1,600 conversational multi-agent traces.
So we took our own failures, sorted by hand into 142 settled codings, and asked which of those boxes each one belonged in. 21% to 26% fit. The other 74% to 79% did not. It is a range rather than a single number because two people sorting the same failure disagree at the edges often enough that one number would claim more certainty than we have.
36 of 142 settled codings fit one of MAST's 14 modes. The rest split between two families the list does not have and a pile marked unclear, and the unclear ones are mostly a detector firing on work that actually succeeded.
The biggest thing in the leftover pile is plumbing. A quota runs out mid-run. A permission prompt blocks a command. A network call dies. Weighted by how often each kind of failure actually happens, that family is 40% to 49% of everything that went wrong, and it is the largest group under every way we tried settling the coders' disagreements. The second family is people: someone getting frustrated, someone pushing back on a claim, someone being asked for permission and saying no. That is the human-in-the-loop family, 6% to 9%. Neither has a box on the standard list, and the reason is structural rather than an oversight: the benchmarks that list was built from have no human in the loop and no shared quota to run out of, so neither failure can happen there.
We also predicted a third new family, coordination failures when one agent fans work out to many at once. It drew 0% of our incidents and 0% of the field's benchmark traces. We wrote it down in advance and it is simply not there.
Then we ran it backwards, which is the part that stops this being special pleading. We pointed our new families at the field's own benchmark traces. They explain 5 of the 149 benchmark traces we coded, 3.36% of them, and the two families that carry our own corpus explain 1. The blindness runs both ways: their list has no cell for most of what breaks here, and our additions barely touch what breaks there. Said plainly, in the same breath as the number: one person labelled all 149 of them, so there is no second opinion on this half.
The agent that was never there
One thing we found while reading those benchmark traces is worth the whole section. In 4 of the 72 AG2 traces we coded, the group manager stamps a sandbox's output with the name of an agent that never speaks in the run. A second agent, whose job is to verify, then reports that silent agent as having submitted its own solution, and closes the run on a submission that does not exist. That is 5.6% of that framework's traces.
The field's own benchmark corpus contains a failure the field's own list never recorded, which is the coverage argument made on its home ground rather than ours. It rests on 4 cases, one framework, one coder, so treat the rate as rough. The mechanism is not rough: the check that the credited agent's only appearance is that output, and that the verifier credits it anyway, passes on all four.
Same toolkit, two different worlds
Point a failure detector at a different kind of agent and it can go completely blind without telling you.
We pointed the same 13 detectors at two very different piles of logs: one person's coding agents, and a fleet of assistant bots that many different people chat with. The texture flips completely. The coding agents fail mechanically: tool errors (1,105), frustration wording (568), API errors (541), error loops (453). The assistant fleet fails relationally: frustration (334), self-correction (171), and stuck repetition (153) are 658 of its 898 incidents, while tool errors fall to 60.
Left, machines breaking. Right, people and bots grinding against each other. The asterisk on the right-hand tool-error bar is honest bookkeeping: the fleet's logs carry no native error flag, so some of that gap belongs to the instrument rather than the world.
Here is the part that should worry anyone who runs a detector over their own logs. 5 of the 13 detectors fire 1,865 times on the coding logs and exactly zero times on the chat logs. Not rarely. Zero. The permission detector alone fires 758 times on one side and never once on the other.
That is not chat bots behaving better. Those detectors key on things a chat transcript simply does not contain, so pointing this toolkit at chat logs quietly loses whole kinds of failure and reports nothing missing. The comparison covers 599,353 events of coding-agent logs against 97,333 of chat logs. The silence is safe to state. The rate differences sitting next to it are not: the two piles differ in task, in who is driving, and in transcript format all at once, so nothing about which side "fails more" can be read off this chart.
A detector that travelled well and was wrong the whole way
The same blindness has a mirror image, and we walked into it on purpose. We wrote down in advance that our permission detector would return exactly zero on four outside benchmark collections, because none of those setups has a permission layer for it to see. It fired 171 times across all 4 outside corpora, in 4 of 4. What it is actually matching is ordinary permission-denied text in command output, which any system can produce, rather than the consent layer it was written for. A detector can be perfectly portable and perfectly wrong at once, and the portability is what hides the wrongness.
One more, smaller and sharper. On another outside collection our frustration detector fired 14 times, and all 14 firings come from 1 pattern: a sentence ending in the word "again". The samples we looked at are people pasting error logs that finish "Please try again." The detector is reading a machine's apology as a person's temper. Those firings were never checked one by one, so this is a diagnosis of the marker rather than a measured error rate. Before you trust what a tool reports, check what it cannot see.
Five things it caught
Specific failures, each with the thing that would have stopped it.
Each of these came out of a transcript a language model read in full, so it is a real moment rather than a pattern in a count. Each is named by its mechanism, because the mechanism is the part a fix can grip.
CATCH 01 · the common oneThe merge that never ran
CATCH 02The bot that was declared alive
CATCH 03The all-clear that outran its own check
CATCH 04The push blamed on token scopes
CATCH 05 · the reversalThe swarm that was not broken
What we changed because of them
One rule governs all three: nothing gets pasted into every session, because standing instructions get ignored by session forty. Each fires only when its pattern is live.
- A completion-claim guard. The agent cannot say "merged", "running" or "verified" until a command that checks it has actually run. Aimed at the 10-for-10 fault cluster above.
- An error-streak nudge. One line injected after four failures in a row, aimed at the 491 error-loop incidents where grinding continued well past the point a person would have stepped back.
- Quota-aware spawning, so an orchestrator stops launching helper agents the session cannot afford. Aimed at the 26 subagent failures, every one of which ended its session.
Each ships with the number that has to move. The baseline was snapshotted before anything was installed, so "it helped" gets measured by re-running this pipeline afterwards rather than asserted. Nothing auto-installs.
Run it on your own logs
Four commands, nothing to install, and your transcripts never leave your machine.
Everything above was computed by six plain Python files in the open-science repo under docs/experiments/failure-atlas/pipeline/. Standard library only, so if you have Python 3 you have the pipeline. Clone the repo, or read each file first on GitHub:
Four commands, from a working directory of your choice:
python3 pipeline/extract.py --roots ~/.claude --db data/atlas.db python3 pipeline/detectors.py --db data/atlas.db python3 pipeline/incidents.py --db data/atlas.db --out data/incidents.jsonl python3 pipeline/analyze.py --db data/atlas.db --outdir data
The privacy model in one picture. atlas.db and incidents.jsonl hold your text and never cross the dashed line. analysis.json is counts and rates with no transcript content in it, and it is the only file you would ever hand to anyone.
Or hand the job to your agent. Copy this and paste it at your own bot:
Fetch the pipeline files from the NimbleCoOrg/open-science repo, under docs/experiments/failure-atlas/pipeline/, into a local folder (clone the repo, or pull the raw files). Run, in order: extract.py --roots ~/.claude --db data/atlas.db, then detectors.py, incidents.py, and analyze.py against that same database. They are stdlib-only Python 3, so install nothing. When they finish, read data/analysis.json and report: my total sessions and events; my top five failure modes per 10k events; and where failures fall within my sessions. Then compare each against the reference numbers on this page (1,165 tool-error incidents at 16.4/10k; giving up concentrated in the final tenth of sessions) and tell me where my logs differ most. Do not send, publish, or quote any transcript text; share data/analysis.json only.
The coding half of this is n=1, one operator and one workflow over seven months. The assistant-fleet half is not: it is many different people across several agent setups and four chat platforms over the same span. Even so, it is one org's record. Medicine solved this exact problem without pooling patient records: hospitals share results computed by a common method over data that never moves. analysis.json is that layer for agent logs. If a handful of people run these four commands and share only that file, "how do agents fail?" stops being one org's atlas and starts being a map. Share the results. Keep the data.
How much of this to believe
We graded our own instruments in public. Here is which numbers not to lean on.
Before looking at any counts, we wrote down 5 guesses about how agents fail. 3 were wrong. 1 was a coin flip. 1 shrank to almost nothing. None of them survived the way we bet it would. But 2 of the 5 left a small real effect behind once the flawed part was stripped out, so "none survived intact" is true and "we found nothing" is not. Those five and their post-mortems are written out in full in the companion document.
The bigger piece of self-grading is the instruments. Every detector here was built deliberately over-sensitive, then a language model read the full transcript around a sample of what each one caught and ruled on whether it was a real failure or a false alarm.
The amber row is our own dishonesty detector, still right about 1 in 4 times it fired after a full round of fixes. Read the low rows as a warning label on every chart above that leans on them, and remember that 12 judged cases keeps every range wide.
There is a pattern in that chart, and it is the most useful thing on this page. Across 13 detectors, the ones that are almost never wrong are watching for cheap mechanical breakage: a command failed, a person hit escape, the API returned an error. The ones aimed at failures that actually cost something, an agent asserting what it never checked, a person losing patience, are wrong most of the time. The signal that is easy to count is not the signal that matters.
That rests on 12 judged cases each, 156 judgements in total, all by a single language model. Treat the direction as the finding and the ordering between neighbouring rows as noise.
Two limits sit under everything on this page. Every judgement is single-pass, one model, no second opinion, and that judge shares a model family with the agents it graded, a bias toward going easy on its own kind whose size nobody has measured. And while we were measuring failures, our own judging runs kept getting killed by the same API-overload failure that ranks among the modes we catalogued (infra_api_fault, n=11). The instrument exhibited the phenomenon mid-measurement.
Everything above is the short version. The detectors and what each one looks for, how the judging worked, the full cross-walk against the field's list, every prediction we registered and what happened to it, and everything we know is wrong with this study, are written out in the companion document: Every method and every caveat, in full.