Building Multi-Agent AI Systems That Actually Work

My first multi-agent system had five agents with job titles. A Researcher, an Analyst, a Writer, a Critic, and a Manager. I'd read the hype, and the idea was intoxicating: instead of one AI doing everything, a team of specialists collaborating. I built it in a weekend, ran it on a real task Monday morning, and then watched the logs with growing horror.

The Researcher asked the Analyst for clarification. The Analyst deferred to the Manager. The Manager asked the Researcher to research the clarification. Somewhere around iteration fourteen, they were politely thanking each other while the token counter spun like a taxi meter. The task never finished. The invoice did.

That failure taught me more about multi agent AI architecture than any article had, so here's both halves honestly: why the naive version collapses, and the shape that finally worked.

Why "a team of AI agents" fails by default

The mental model of hiring a team is exactly what's wrong. Human teams work because people have judgment about when to stop, escalate, or just do the thing. Agents given open-ended peer-to-peer conversation have none of that — they have politeness and probability. Three failure modes showed up immediately:

Circular delegation. With no hard structure, agents hand work sideways forever. Each handoff feels productive to the model. None of it converges.

Error compounding. Agent B trusts Agent A's output completely. If A hallucinated a detail, B builds on it, C polishes it, and the final answer is confidently wrong with three layers of varnish.

Cost multiplication. Every message between agents is a full LLM call carrying context. Five chatty agents don't cost five times one agent — they cost whatever their conversation decides, and their conversation decides a lot.

The question to ask before going multi-agent

Here's what I wish someone had told me: a single agent with good tools beats a mediocre multi-agent system almost every time. My Responses API agent handles a surprising range of work with one model and a handful of tools.

A single agent genuinely breaks down in only a few situations: when the instructions for different parts of the job conflict (a strict fact-checker persona and a creative writer persona pull one prompt in opposite directions), when the context each sub-task needs is large enough that stuffing it all into one conversation degrades quality, or when different steps deserve different models (a cheap fast model for triage, a strong one for the hard part). If none of those describe your problem, you don't have a multi-agent problem — you have a tools problem.

I hit a real one with the support agent: billing questions needed strict policy grounding and invoice tools, technical questions needed docs and debugging tools, and one prompt trying to be both got measurably worse at each. That's a genuine seam. That's where you split.

The architecture that works: orchestrator and workers

The version that succeeded looks less like a team chat and more like a kitchen. One orchestrator, several workers, and — this is the load-bearing rule — workers never talk to each other.

User → Orchestrator (plans, routes, assembles)
           ├→ Billing worker   (policy docs + invoice tools only)
           ├→ Technical worker (product docs + diagnostic tools only)
           └→ Research worker  (search tools only)

The orchestrator decomposes the request, dispatches tasks, receives results, and composes the final answer. Workers are narrow, stateless, and disposable. All the failure modes above trace back to peer-to-peer chatter, and this shape simply deletes the channel it happens on.

Three design decisions that made the difference

Handoffs are structured data, not conversation. The orchestrator doesn't "ask" a worker anything. It issues a task spec:

{
  "task": "check_refund_eligibility",
  "input": { "order_id": "ord_2291", "reason": "damaged item" },
  "constraints": "Answer only from policy. If not covered, return needs_human."
}

And workers return equally structured results — including an explicit confidence and a needs_human escape hatch. JSON in, JSON out. The moment I replaced conversational handoffs with specs, circular delegation became structurally impossible: there is no reply channel to circle in.

Shared state lives in the database, not in messages. My failed version passed growing context blobs between agents — expensive and lossy. Now there's a run row in Postgres: the plan, each task's status, each result. The orchestrator reads state from there; workers receive only their slice. Debugging changed from archaeology on transcripts to reading a table.

Budgets are hard limits, not vibes. Every run gets a maximum step count and a token budget, enforced in code. Hitting either doesn't fail silently — it returns partial results with an honest "ran out of budget at step 4." My taxi-meter morning made this non-negotiable; nothing in the model will ever volunteer to stop.

Validate at the seams

The subtlest fix: the orchestrator treats worker output as untrusted input, the same way you'd treat a form submission. Schema-validate it, sanity-check it, and when a result smells wrong, re-dispatch once with the failure noted — not into a discussion, into a retry. Error compounding dies at the boundary or not at all.

What it looks like in production

The support system now runs a cheap, fast triage worker first (classify + route, using a small model), then exactly one specialist. Median case: two model calls. Complex case: four or five. Compare that to my five-agent séance, which averaged dozens. Costs are boringly predictable, and quality went up because each worker's prompt got shorter, stricter, and testable in isolation — I have actual unit tests for the billing worker that would have been impossible against a group chat.

The honest summary

Multi agent AI architecture isn't about assembling a company of tiny employees. It's about splitting a job along genuine seams — conflicting instructions, oversized context, mismatched model needs — and then connecting the pieces with the least conversational plumbing you can get away with. Start with one agent. Add workers only where a seam is real. Make handoffs structured, state external, budgets hard, and validation mandatory.

And if you ever find your agents thanking each other in a loop at 2 a.m., know that you're in good company, and that the fix is not a sixth agent.