Five distributed-system failures on a board of AI agents: Hermes Agent, part 2

In part 1 a team of AI agents built one level of a browser game on a Hermes Agent Kanban board. It took 38 hours, and for 22 of them nobody was working.

The level they built teaches a rule of distributed systems: a lost acknowledgement must not undo a write that did happen. The board that built it broke that rule twice. So this part reads the board as what it is, a small distributed system, and goes through five of its failures.

The setup

Cards wait on the board until the dispatcher, a Hermes process, starts an agent on one. The agent works in its own git checkout and then hands the card to a reviewer. A separate coordinating agent, outside Hermes, prepared the cards and restarted whatever had stopped.

Each of these pieces has a counterpart in a distributed system:

Next to MAST

A group at UC Berkeley went through more than 1,600 traces of seven multi-agent frameworks and sorted the failures into 14 modes. The result is called MAST, and its conclusion is the one this run reached too: "many MAS failures arise from the challenges in organizational design and agent coordination", not from weak models.

Each of the five failures below has a relative on their list:

Failure here	Closest mode in MAST	Its share of failures there
1. Lost update	Unaware of termination conditions: nothing noticed that the run had already finished its job	12.4%
2. A retry that cannot succeed	Step repetition, their most frequent mode	15.7%
3. No clean state to start from	Premature termination: the work stopped before it was handed over	6.2%
4. Accepted is not delivered	Ignored other agent's input, except that here the input never arrived	1.9%
5. The control plane cost more	No mode: MAST does not count what coordination costs	none

The main finding of part 1, cards that closed while the level did not work, is on the list as well, twice: "No or incomplete verification" and "Incorrect verification", 17% of failures between them.

1. Lost update

An author's run handed its card to review on step 64 of 64. Five seconds later the handler for the step limit marked the same card as failed.

09:09:02  review_requested  run_id=68    "GC-011R3 implemented checkout Lookup→miss→Commit as real routed request/reply stages ..."
09:09:07  gave_up           run_id=null  "Iteration budget exhausted (64/64) — task could not complete within the allowed iterations"

The handler wrote the card's state without checking which run it was writing about. A write based on a stale read replaced a newer one, and the card sat blocked for 3 hours 12 minutes with its reviewer already assigned. It was the second time; the first one was caught in 82 seconds.

Verdict: a defect in Hermes. We patched it locally: the failure write now checks the run and its claim in the same transaction, a compare-and-set. Upstream has an open issue for the same code path, and on November 12, 2026 the main branch was still unguarded.

2. A retry that cannot succeed

A card ran out of its review rounds, so the limiter refused to start it. The dispatcher asked again, and again.

18:06:32  spawn_failed  failures=1  Author/review cycle allowance exhausted: GC-011; author=8, review=2, limit=6; authorLimit=7
18:06:52  spawn_failed  failures=2  (same text)
18:07:12  gave_up       failures=3  (same text)

Nothing changes between such attempts, so the retry cannot succeed. Each refusal still counted as a failure, and after the third the card stopped. The next morning the same pattern stopped another card, and nothing ran for 7.3 hours. Of the 43 launches the board refused before an agent started, 26 were refusals for used-up rounds or budget.

Verdict: our configuration. The rounds, the budget and the limiter that enforced them were ours, and the limiter had two bugs of its own. One thing belongs to Hermes: a refused start is counted like a crashed one.

3. No clean state to start from

A run cut off by a limit leaves its edits in the checkout, uncommitted. Our rule said a card may start only in a clean checkout. So the next launch was refused before the agent started.

Child GC-014 worktree <path> is dirty or has untracked files; all workers must commit their code before preparation/review:
  ?? src/cart/ui/CartRequestTimeline.tsx
  ?? src/cart/ui/CartResultsPanel.test.tsx
  ?? src/cart/ui/CartResultsPanel.tsx

A retry assumes the job can start again from a known state. Here the state was half-written files, and committing them blindly would have been wrong. It happened 17 times on 8 cards.

Verdict: our configuration. The rule had no path for interrupted work. The coordinating agent later wrote a recovery step that saves such files as a checkpoint, and it grew case by case.

4. Accepted is not delivered

A watcher polled the board every 30 seconds and sent an alert about each blocked card to the coordinating agent. The queue accepted every message and returned a receipt. Twice no turn followed until I reported the stop myself.

The alerts went to a conversation id written in the configuration. The coordinating agent was working in another branch of that conversation. The notes of the run say it plainly:

Messages/probes were addressed to a different conversation; their queue receipts never
demonstrated delivery here.

Three things had been treated as one: the queue accepted the message, the message arrived, the agent acted on it. Two of the three long waits of the run were of this kind, 3.2 and 4.1 hours.

Verdict: our configuration. The coordinating agent lived outside Hermes, so nothing in Hermes showed where its alerts went or whether they arrived.

5. The control plane cost more than the data plane

One agent prepared the cards, restarted stopped work, received the alerts and wrote the watcher, the limiter and the recovery code. It was a single session of one model, so when it was silent the board stood still.

It was also most of the bill. At API list prices the agents on the board used about $40 of tokens, and the coordinating agent about $350. It made 1,775 model calls, and each one re-read about 164,000 tokens of its own context. In a distributed system the control plane is usually small next to the data plane. Here it cost more than eight times as much.

Verdict: our configuration, and its main flaw.

What I learned

Good limits come with experience and depend on the task. Where the budget allows, I would set them loose, because a tight limit can stop all progress, and here it did. I would also not leave these settings to an agent unchecked: I will read them before the start.

The coordinating agent was a stranger to Hermes. It sat outside, so its configuration was invisible from the board, and that is one reason the alerts went nowhere unnoticed. A second, cruder channel would have helped: waking the agent on a schedule every few minutes, whatever the alerts say.

The coordinator's role turned out huge, and it cost more than all the other work together. That is a mistake of this setup as a whole. A cheaper model could have done the job; at the start it was not clear at all how much coordination there would be. With this experience I would also write the coordinator a detailed brief on what to do in each typical situation.

References

• Hermes Agent, the agent harness with the Kanban board and the dispatcher that this series is about.
• Issue 146973 in its repository: the late write after the step limit, the code path of the first failure here.
• Mert Cemri and others, Why Do Multi-Agent LLM Systems Fail?, arXiv:2503.13657, version 3, October 2025: the paper that introduces MAST and its dataset of annotated traces.
• Part 1 of this series: the run, its numbers and its cost.