Roundtable recap, Oct 30: an agent orchestration demo, and what separates models that score alike

The top coding models now solve most of the same tasks. If that holds, the difference between teams comes from how they run the models. This week an attendee gave an unscheduled demo of exactly that, and the discussion afterward tested the idea against benchmarks and harnesses.

Demo: one orchestrator per developer

One attendee's team built an orchestration layer where each developer has a single orchestrator. It starts full Claude Code sessions in the Warp terminal, each named after a spice, and keeps a queue of work with a cap of four sessions at a time. It also picks the model and effort level for each task: Haiku gathers information, Opus plans, and Sonnet does the work.

Each orchestrator posts status to a shared Slack channel about every 30 minutes and messages the others directly, so two of them can settle a question while both developers are away. Shared learnings land in an Obsidian vault through the team's own vault MCP server. A side panel built as a Claude Code mod shows what's running, and an away mode answers routine questions and holds the rest until the developer is back.

On context, sessions auto-compact at around 40% of the 1M-token window, and a separate session compacts the orchestrator during quiet moments. The idea of agents coordinating with each other came from OpenAI's report on its Hugging Face incident, where agents in an evaluation organized themselves through a message board they set up.

The team says a task now goes from assignment to dev in three or four hours. Local review rounds gate it, and a PR can't open until a verified sentinel shows the review passed. The orchestrator keeps working overnight, and everyone pulls updates from a shared repo so it can update itself.

The models are close, so what separates them?

The week's model news started the discussion. Google's Gemini 4 Carbon took the top spot on Artificial Analysis's new Intelligence Index, just ahead of Claude Opus 5.5, while Kimi K3.1 Pro fell from fourth to ninth under the new methodology. One attendee splits their dev work between Opus 5.5 and Codex and sees little difference between them.

That raises a question our own evals keep returning to. If every model can solve the task, is the race about capability or affordability? Early results from our internal evals show Haiku 5.5 handling most web work, such as Next.js apps. On some PRs Sonnet came out slightly better and cheaper, because it needed fewer turns and less feedback from the reviewing agent.

Benchmarks are getting harder in ways that aren't always useful. Harvey's legal agent benchmark scores a task at zero unless every rubric item passes, even when the model got 95% of a two-day task right. The group roughly agreed that the top models are equal now, and that the gains come from orchestration: knowing where a model tends to fail, building checks for it into the pipeline, and adding a learning loop.

Harnesses are the next variable

We're comparing harnesses across models, building on research one of our engineers started. Terminal-Bench ranks agent and model pairings, and Factory reports that its new Droid 1.0 harness beats Claude Code and Codex running the same models on Terminal-Bench 4.0.

We want more than a resolution rate. We want cost per task and whether it worked the first time. Picture a robot arm moving a glass globe, where two failures before a success means two broken globes. Someone wished for a benchmark that lets you weight the criteria that matter to you.

In our testing, Pi showed only small differences from the standard Claude Code harness. One attendee maintains a plugin that adds stricter, configurable permissions to an open-source harness they run on a VM and use from a phone, though they find the default harnesses mostly stay on the rails. On one team, developers preferred GitHub Copilot over Claude Code in an early comparison, and others run OpenCode on top of a Copilot subscription for a better experience.

AI inside tighter guardrails

An attendee at a large global company described strict controls. Frontier tools need security exceptions, many AI-powered apps are blocked outright (meeting-notes and dictation tools among them), and production AI runs on internal tools built on open-weight models. Their take is that those tools come close to the best frontier models without quite matching them.

The Econify.AI Roundtable meets Fridays at 11am ET / 4pm UK. Details and registration are at econify.com/ai-community.