48 closed cards, no passing design: a multi-agent build with Hermes Agent, part 1

I wanted to find out whether a team of AI agents could build a feature mostly on their own. The feature was one new level for a system design browser game, described in a detailed document. I used Hermes Agent, an open-source agent from Nous Research that can run a team of agents from a Kanban board. The run took 38 hours, and four things went differently from what I expected:

1. Cards closed while the level did not work. With 48 cards closed, no design could pass it.
2. Most of the time nobody was working. For 22 of the 38 hours, no agent was running.
3. One card held up the rest. It took 17 runs, with twelve other cards waiting behind it.
4. Fifteen minutes of my own play found defects that 56 reviewed cards had missed.

In the end the level was accepted with two scenarios out of the six that were planned. At API list prices the agents on the board used about $40 of tokens. The coordinating agent that kept the board moving used about $350. That is the price of the tokens, not what I paid: most of the models ran on flat subscriptions.

The game is live: albedoweb.github.io/system-design-game. The source is public.

The game and the task

System Design Game is a browser game where you draw an architecture and then run load through it. In the Global Cart level, you build a shopping cart for customers in San Francisco, Frankfurt and Tokyo. You assemble the system from a set of components, and it has to pass several tests. The simulation runs two scenarios against your design: normal load and a rush.

Behind this is a deterministic discrete-event simulation with queues for CPU, disk and network links. It has to keep its books straight: a write that was never delivered must have no effect, and a lost acknowledgement must not undo a write that did happen.

The game already existed. The task was to add this one level.

Why Hermes

I wanted to experiment with a multi-agent harness. What I liked about Hermes is that the work is built around task cards on a Kanban board. A card has an assignee, dependencies and a reviewer, like a ticket in a real development team, and that way of working is familiar to me.

The setup

Every card is assigned to a profile: a role with its own model and instructions. I started with five roles.

Role	Model	What it does
Lead	GPT-6 Astra	Writes the engineering contract and splits the work into cards
Game master	GPT-6.1 Sol	Owns the player's side: the concept, the balance and the final verdict on the level
Programmer	DeepSeek V4 Flash	Writes the code, in two profiles: one for the simulation, one for the interface
Tester	DeepSeek V4 Flash	Writes independent checks and takes measurements
Reviewer	GPT-6 Luna	Reviews every card before it can close

During the run I added programmers on stronger models: one on Sol, then several on Claude Sonnet 5.5.

One more agent stood outside the board: a coordinating agent, a long Codex session on Astra. It prepared the cards and their git worktrees, restarted work that had stopped, and was the one to be alerted when a card got blocked.

Three units of work appear below:

• A run is one session of one agent on one card. It ends when the agent hands the card on, or when a limit stops it.
• A step is one model call with its tool call inside a run.
• A round is one author run followed by a review.

The limits at the start were 15 minutes and 64 steps per run, three rounds per card and at most three runs at once. I raised several of them during the run.

The lead split the level description into 24 cards with 44 dependency links.

Cards closed, level not working

The board grew from 24 cards to 68, and 56 of them were closed, each after an independent review. The other 44 cards were added by the coordinating agent along the way: repairs and diagnostics after each failed integration or acceptance, and at the end the cards for a reduced level.

For the first 27 hours, the count of closed cards kept rising while the level did not work. A design here is one architecture built in the game. The game master first ran the whole level at hour 20, with 24 of the 30 cards closed: seven test runs, no passing design. At hour 27, with 48 cards closed, three more complete designs were run against all six scenarios, and again none passed. Checkout latency at the 95th percentile was 953 to 974 ms against a limit of 750 ms.

At that point I cut the level from six scenarios to two. The four I dropped covered customers who move between regions, two devices editing one cart, a network partition and a race for the last items in stock. Ten hours later, 7.3 of them spent waiting, the two-scenario level passed acceptance.

Reviews did their job: the reviewer sent work back 30 times, on 20 different cards. But each review checked one card against its own description, and nothing ran the whole player path until hour 20.

Most of the time, nobody was working

The whole run took 38 hours by the clock, and for 22 of them no agent was running. At least one agent was working for 16 hours, and two or three at once for under 3 of those.

Most of the idle time was three long waits, 14.6 hours together. Each was caused by a limit that did not fit the work, or by an alert that never reached the coordinating agent.

The first was 3.2 hours. A run reached its 64-step limit just as it handed a finished card to review, and the limit handler in Hermes overwrote the handoff. The alert about the blocked card went to the wrong conversation of the coordinating agent, so nothing woke it.

The second was 4.1 hours. Five repair cards were done, and the next step was an integration that only the coordinating agent could do. Again the alert did not reach it.

The third was 7.3 hours. A card that had already passed its checks had used up its reviews, and only I could allow one more.

The board recorded 276 launches of an agent. In 43 of them the agent never started, 26 times because the card had used up its rounds or the shared budget. Seven more are bookkeeping entries. Of the 226 runs that did start, 49 ended without handing the work on, which is about one in five. Of those, 17 hit the step or time limit, 17 stopped to ask for a decision, 9 crashed and 6 failed in other ways.

One card held up the rest

The parts of the simulation were built in parallel, and the card that joined them turned out to be a redesign of how the simulation moves time forward. The DeepSeek programmer made six author runs on it without getting it through review, and it was then split into five smaller repair cards. In the end it took 17 runs, and at one point twelve other cards were waiting behind it.

About half of the cards took the short path: 32 of the 60 that were started closed in two or three runs.

Most of these problems came from planning, mine and the coordinating agent's, and I do not hold them against the tool. Hermes had two defects of its own, which I patched locally. The overwritten handoff was one of them, and part 2 goes into both.

The result and the cost

The level with two scenarios was accepted: a reference design passes, and designs that should fail do fail. The level as first specified was not built. Twelve cards stayed open, all of them work for the original six scenarios.

Only DeepSeek was billed by the token, about $4.50. The other models ran on subscriptions, so I estimated what their tokens would have cost at API list prices.

Role	Model	Tokens in / out	Cost*
Programmers and tester	DeepSeek V4 Flash	281 M / 2.6 M	$4.50, billed
Extra programmers	Claude Sonnet 5.5	63 M / 0.6 M	about $20
Reviewer	GPT-6 Luna	90 M / 0.4 M	about $2
Game master and one programmer	GPT-6.1 Sol	39 M / 0.2 M	about $9
Lead	GPT-6 Astra	3 M / 0.03 M	about $7
Coordinating agent	GPT-6 Astra	290 M / 0.6 M	about $350

* The cost of the tokens at API list prices, not the cost of a subscription. Only DeepSeek was really billed this way. The other models ran on flat subscriptions, so this is not what I paid.

The agents on the board come to about $40. The coordinating agent comes to about $350, more than eight times as much. It ran on the most expensive model, and in one long session it re-read a large context on every step: 99% of its input tokens were cache reads. I did not count my own time.

For the same money, one strong agent with a cheap reviewer could probably have done this work without the overhead of the board. I have not measured that yet.

Fifteen minutes of play

The version you can play took one more evening in a simpler setup, on top of the work the board had done: one coordinating agent, Claude Code, handing small tasks to cheap sub-agents while I played the level between rounds. Fifteen minutes of play found things that 56 reviewed cards had not:

• The design checklist always reported purchased circuits as missing.
• A connection dropped on a component, a few pixels away from its small port, was silently lost.
• A failed run opened a report 64,000 pixels tall, on the scenario that had passed.
• A rule survived from the scenarios I had cut: every application had to read storage in all three regions. No review noticed, because every review tested the reference design, which obeys the rule.

The first three are fixed in the version you can play. The rule is still in the level, and the game now explains it.

What I learned

• Split the project into smaller and more concrete tasks. The card that joined the simulation parts was a super-task: the DeepSeek programmer could not get it through in six runs, and twelve cards waited behind it.
• Check the dependencies between tasks before the start. I need a dependency viewer, or a checker that finds a card with twelve others behind it before the first run.
• Set the limits with care. Limits are needed, but mine did not fit the work: a run that hit its 64-step limit lost a finished handoff, and a card that had passed its checks waited seven hours for one more review. Badly set limits and a broken alert path to the coordinating agent caused the idle time.
• Cards that are each tested and reviewed do not add up to a working game. It is obvious, and still worth saying: 48 cards were closed while no design could pass the level.
• Full autonomy did not work out, and acceptance has to start early. Fifteen minutes of my own play found what the board had missed.

At the start the task looked very simple to me, and I did not expect this many problems. The preparation before the first run deserves far more work than I gave it.

Next

Two more parts will follow:

• Part 2, the cost of coordination: the board read as a distributed system, with its limits, lost handoffs and alerts that went to the wrong place.
• Part 3, Hermes next to a simpler harness: what I would use each for.