Skip to main content

What it took for five coding agents to beat one

One coding task, three setups, 18 runs, and five findings about what made the split pay off

Updated 14 min readMarkdown

For the past few weeks I have been testing multi-agent setups on real development work. In this round I gave one task from my browser game to three setups: one agent alone, five agents on a plan I wrote, and five agents on a plan that a lead model wrote for them. Each setup ran the task six times, and the same hidden tests judged them all.

The short version: if you split a coding task between several agents, what decides whether you gain time is what each agent is told to check. One agent alone took 36 minutes on average, five agents working side by side took 24. With my first instructions the five did more than three times the single agent's work, and a single line changed that.

Below are five findings. Two of them (1 and 5) come with measurements; the other three come from what I saw in particular runs.

The level screen of the System Design Game: an isometric factory floor with three plots, One server, Cache and Load balancer, and the first level's requirements on the right

The game the task came from: System Design Game, where you build a system and watch it handle traffic. It runs in the browser.

The setup

The task. Five independent changes to my browser game: levels that load on demand, a small kit of interface components, a screen of levels with saved progress, and two clean-ups. Then one more change that joins three of them. The five changes share no file.

Three ways to do it:

  1. One agent. A single session gets the task and plans the work itself.
  2. My split. I wrote a plan with five parts; five agents work on them side by side, each in its own copy of the repository, and one more step joins the parts. A script hands the parts out. (Agent frameworks call this a task graph.)
  3. Auto-split. The framework's lead model gets the same task with no plan and splits it by itself.

How a run goes. A run of the first two ways has two stages:

  • The build: the agents write the code (in my split, the five parts and then the step that joins them).
  • After the build: a script runs the six checks below and hands any failure to a repair agent; an independent reviewer model reads the whole change and can send it back for fixes, up to three rounds; then the script runs the checks once more.

The auto-split's lead model sets up its own checks and reviews. The time of a run covers both stages; the hidden tests are run afterwards, outside it.

The same judge for all three. Every result had to pass the same six checks (type check, lint, build, a check that levels load separately, the fast tests and a test of the whole app) and 30 hidden tests that no agent could see. The framework was Hermes Agent; the workers ran on a cheap, fast model (DeepSeek V4.1 Flash) and the reviewer on another (GPT-6 Luna).

Three ways to do the same work: one agent in one session that plans the work itself from the prompt, my split into five agents and one that joins the parts, and the auto-split, where a lead model decides the parts

My split won by twelve minutes, and the auto-split was a lottery

Six runs for each way.

One agent My split Auto-split
Time, minutes 36 (27 to 50) 24 (19 to 28) 33 (18 to 61)
Model calls 211 227 287
Cost $0.218 $0.232 $0.348
Passed the check 6 of 6 6 of 6* 5 of 6

* One run of my split first failed a hidden test; the fault was in my test, not in the agents' work (see the limits).

  • My split was a third faster than one agent, for 6% more money. Five of its six runs were faster than the single agent's fastest.
  • The auto-split had the best run of all and the worst. 18 minutes and 20 cents once, 61 minutes another time.
  • One run of 18 failed, and its own review had passed it. An auto-split run changed two files the task says to leave alone. Its reviewer ended on PASS and the framework reported the work as verified.
Minutes for every run: one agent 27 to 50 with a mean of 36, my split 19 to 28 with a mean of 24, the auto-split 18 to 61 with a mean of 33

1. Give each agent the exact checks for its part, and run the full check once at the end

In my first plan every agent's instructions ended like this (part T is the kit of interface components; the coordinator is the script that hands out the parts and runs the final check):

VERIFY: every rule of part T is met, and the commands of the formal acceptance that
concern your part will pass when the coordinator runs them. Which checks you run
yourself is yours to decide.

With this line, the trial run of my split took 31 minutes, against 23 for the single agent's trial run. Those 23 minutes were the single agent's fastest time of all (its six counted runs later took 27 to 50), so the gap in wall time says little. The work itself said more. I split every agent's session into the time the model spent answering and the time its commands took.

  • The five agents, with the joining step, had worked 62 minutes on what the single agent did in 17. Running five of them at once could not win that back.
  • They ran checks 92 times; the single agent, given the same sentence, ran them 25 times. The five agents spent 21 minutes in checks, the single agent 4.
  • Two of them wrote their own browser tests, 11 minutes of scripts nobody had asked for.

My reading of it: one agent runs the whole test suite a few times for the whole task, and five agents each run it a few times for their own part.

The auto-split had divided the work the same way, and its agents took 20 minutes. Its lead model had given each agent the exact checks for its own part. I rewrote my line to do the same:

VERIFY: every rule of part T is met, and these commands, run in your copy, exit 0:
`npx tsc --noEmit`; `npx eslint --max-warnings=0 src/ui/kit`; `npx vitest run src/ui/kit`.
Run no other check.
  • With the new line the five agents, with the joining step, worked 26 minutes instead of 62. Checks fell from 21 minutes to 6, and the browser scripts disappeared.
  • The split itself did not change. The same five parts and the same joining step lost with the first line and won with the second.
Minutes of agent work on the same five parts and the joining step: one agent 17, my split with the first instructions 62, my split with exact commands 26, the agents of the auto-split 20

The narrow line is safe only because something still runs everything at the end.

  • After the agents finish, a script runs all six checks, with no model involved. It takes about a minute.
  • If a command fails, one repair agent gets the output. That happened once in the six runs of my split and never with the single agent.
  • Then an independent reviewer reads the whole change, and the script runs the full check again.

2. Write the plan once and freeze it

The auto-split divided the work well every time: five parts and a joining step, exactly as I had. The instructions it wrote for its agents were different in every run.

  • In its 61-minute run the lead model wrote a test path one folder too wide: src/cart instead of src/cart/ui. That folder holds the game's slow simulation tests.
  • Two agents then spent 18 and 9 minutes in tests. The lead model waited for its agents for 57 of the 61 minutes.
  • In another run it sent one part to a more expensive model. It gave the part a category of work that the framework routes to Claude Sonnet 5.5, and that run cost 75 cents, three times the usual.
  • In its 18-minute run the paths were narrow.

A plan written once, checked and frozen gave me 19 to 28 minutes. A plan written anew in every run gave 18 to 61.

Two runs of the auto-split as timelines, one of 18 minutes and one of 61, with two agents that spent 9 and 18 minutes in tests marked

3. Have another model read the instructions before any agent does

Before any agent got the task, the reviewer model read each set of instructions exactly as an agent would see it. Its only job was to list contradictions, sentences with two readings and rules that cannot be met.

  • Four passes found 9, 6, 5 and 5 faults. The count stopped falling and the last ones were corner cases, so I stopped.
  • The faults were real ones. "The coordinator commits your changes" stood next to "do not commit". One agent was told to report test counts for commands that run no tests.
  • I learned this the expensive way. In an earlier batch one part said "no test file changes" and another part had to change four test files. The reviewer refused the work three times before I saw why.

4. Try a new setup on one task before you give it more

This is not about doing the work twice. Before a new multi-agent setup gets a lot of work, give it one task and look at where the time goes. In this experiment that was one trial run of each way before the counted series; in everyday work it is the first task you hand a new setup.

What the trial runs showed me:

  • All three trial runs failed my hidden tests, and in all three the agents were right. Behind the failures were six faults of my own checks, among them a rule that could not be met, two checks that matched words in comments, a file moved with git mv read as a changed test.
  • The trial is also where the 62 minutes showed up. Reading the plan had not found that line; neither had the model that checked the instructions.

5. Don't expect the gain from building in parallel

I expected the gain to come from five agents building at once. Most of it came from somewhere else.

  • Building side by side saved under 3 minutes: 16.0 against 18.7. The longest part and the joining step after it set the length of the build, whatever runs beside them.
  • Splitting made the work 1.5 times bigger. My five agents, with the joining step, worked 26 minutes on what one agent did in 18 (both are means of the six counted runs; the 17 of finding 1 is the trial run).
  • Nine of the twelve minutes were saved after the build. The full check, the review and the fixes took 17.6 minutes with one agent and 8.3 with my split.
  • The reviewer sent the single agent's work back 11 times in six runs, and my split's 3 times. One round of fixes costs about seven minutes.

My reading of it: an agent with one part reads the rules of that part more closely than an agent with all of them. I have not tested that reading.

Mean minutes of six runs, split into the build and what follows it: one agent 19 and 18, my split 16 and 8

One run of my split and one of the single agent, drawn on the same scale, show where those minutes go.

One run of my split and one run of the single agent on the same scale of minutes, each bar one agent

When I would not split at all

When the parts share code. On two earlier tasks where they did, the split cost 1.7 to 2.3 times as much as one agent, and on one of them it failed a hidden test that the single agent passed. Those runs still had the vague check line, so I cannot say how much of the loss came from the split.

Limits

What weakens finding 1, the main one:

  • With the first wording I have one run. 31 minutes, 368 model calls, 34 cents; it was compared with the single agent's trial run of 23 minutes, the fastest the single agent ever did.
  • The before and after were measured hours apart. The 62 minutes are from the afternoon trial, the 26 from the six counted runs of the evening and the night.
  • I rewrote the instructions after seeing the trial run, and I did not write down what I expected before the runs.

Other limits:

  • Six runs for each way; the auto-split's spread (18 to 61 minutes) is too wide for its mean to say much.
  • One repository, one stack (TypeScript, React), one pair of models.
  • My own checks were wrong seven times and the agents right. Six times in the trial, once in the counted runs; every failure was read against the task before it was counted.
  • The time I spent preparing the plan is in no number here. The single agent needs only the task.
  • In one auto-split run a different model did one of the parts.
  • The agents shared a scratch folder between runs. Twelve times in 21 runs an agent met a leftover of an earlier run: helper scripts, screenshots, build folders. None was part of a solution.

What I take from it

  • Split only work whose parts are really independent. It sounds obvious, and it is. Five changes that shared no file went from 36 minutes to 24. On earlier tasks where the parts touched the same code, the split was slower and cost about twice as much.
  • Prepare, and try a new setup on one task before the big ones. My trial round took an hour and a half. It found six bugs in my own checks and the one line that more than doubled the agents' work. I would not have found that line by reading the plan.
  • Letting the model split the work is a gamble. It cut the work the same way I did every time, and once it beat all my runs. It also took 61 minutes once, spent three times the usual money once, and once passed work that broke the task's rules. Writing the plan myself costs time before the start; letting the model write it costs predictability.
  • Hermes Agent did much better with a plan than with its kanban board. 11 of the 12 split runs passed, mine and the model's, each in about an hour or less. My earlier series with the board went very differently, though on other tasks.

Method in brief

  • Everything needed to check or repeat the runs is public: the task, the three plans, the grader with its hidden tests, the scripts, the ledger of all 21 runs and the code every run started from, in the experiment/b2 branch of the game's repository. SETUP.md there goes from a clean machine to one run.
  • Twenty-one runs, one at a time, on one laptop (Apple M2 Pro): one trial and six counted for each way.
  • Hermes Agent 0.23.0; the auto-split is the /dag-ulw mode of its oh-my-hermes plugin, version 4.0.7.
  • Workers on DeepSeek V4.1 Flash, the reviewer on GPT-6 Luna. Costs are at list prices.
  • Who wrote the plans. For my split I chose the parts and Claude, my coordinating agent, wrote each part's instructions; GPT-6 Luna read them for contradictions before any run (finding 3). The single agent planned its own work from the task. The auto-split's plan was written in every run by its lead model, DeepSeek V4 Flash.
  • Every run starts from a fresh copy of the repository at the same commit and is judged by the same six checks and 30 hidden tests.
  • The time of a run is its wall clock, from the start of the first agent to the end of the last check, without the cloning before and the grading after.

Running six trials per setup against 30 hidden tests makes the 36-minute versus 24-minute comparison actually credible instead of a one-off anecdote. Giving five agents disjoint file ownership in separate repository copies avoids merge-conflict thrashing while keeping the integration step explicit.

What stands out most is that a single instruction change cut redundant work by more than 3x. In multi-agent coding, telling each worker the exact local check it owns—and what NOT to re-verify across the whole repo—often saves more wall-clock time than picking a faster model.

Thanks! Agreed on the last point: in these runs the check line mattered more than the number of agents.

The finding that the same split won or lost depending on whether each agent got exact verify commands is a useful isolating result. With six runs per setup, though, the 18 to 61 minute auto-split spread makes me curious whether the win is the frozen plan or the narrower checks, since your own plan also carried the exact commands. A fourth arm with a frozen plan but the original loose VERIFY line, or the auto-split given your narrow-check template, would separate those two effects.

Fair point, the two effects are mixed. The frozen plan with the loose VERIFY line exists as one trial run (62 min of agent work vs 26 with narrow commands), which is why it's the first item in my limits. Running the auto-split with my narrow-check template is a good next arm, I'll add it.