Notes on building an AI agent ecosystem

I work in online advertising, and over the past while I've been building a network of AI agents to handle the diagnostic and reporting work that used to eat up most of my day — checking why an account's spend dropped, producing health reports for clients, working out budget splits. This post is about the architecture that's emerged, not a pitch for it.

The starting problem

A single AI assistant that's supposed to do everything degrades fast. Give it instructions for ten different jobs and it forgets half of them, mixes up output formats, and gets harder to debug every time you add something new. The fix wasn't a smarter model — it was splitting the work into narrow, single-purpose agents and being strict about how they talk to each other.

Five layers

The ecosystem ended up with five layers, each with one job:

• Routing — a single "Agent Manager" that classifies an incoming request and hands it to the right specialist. It never does the work itself.
• Task agents — the specialists. Diagnostic agents (e.g. an AccountCheck agent that runs a fixed set of SQL checks when spend looks wrong), reporting agents, advisory agents, and a couple of automation agents that run on a schedule.
• Infrastructure — most importantly a DocsManager agent, which is the only agent allowed to write files. Everything else is read-only.
• Governance — an Auditor and an independent Quality Judge that score every output after the fact.
• Shared knowledge — a general/ folder with database schema docs and formatting standards that every task agent reads from, so they don't each develop their own (inconsistent) conventions.

The "one writer" rule

This is the decision I'd point to first if someone wanted to copy one thing. Every agent except DocsManager is read-only. If AccountCheck finishes a diagnostic and needs to log the result, it doesn't write anything itself — it hands the write to DocsManager.

The payoff is boring but real: no two agents can corrupt the same file at the same time, there's exactly one place to look when something's wrong with the stored data, and adding a new agent never means adding a new way for files to get written.

Quality scoring, and why it's silent

After a task agent answers, it gives itself a score out of 10. That score goes to the Auditor, which also asks the Quality Judge — a separate agent that scores the same output without seeing the self-score, to avoid anchoring. Both scores get logged.

None of this is visible to whoever asked the original question — they get their answer and move on. The scoring exists so that an agent's quality can be tracked over time without anyone having to manually review transcripts.

Agents move through a small lifecycle based on these scores:

DRAFT → TESTING → STABLE

An agent gets promoted to STABLE after at least 3 scored runs with an average score of 8/10 or higher and zero flagged issues. If a score drops below 6, that's a flagged issue; three flagged issues sends it back to TESTING and pings me.

What's still missing

Writing this down also made the gaps obvious:

• No error recovery. If an agent fails mid-task — an API timeout, a bad query — there's no automatic retry. I have to re-run it manually.
• No user feedback loop. The quality scores are all internal; there's no way for someone using the output to flag "this was wrong" and have that feed back into anything.
• No automated testing. When I change an agent's instructions, nothing checks that it still produces correct output — I just have to try it and watch the scores.

None of these are urgent, but they're the next things to build rather than adding more task agents.

Why files, not platform features

Everything — instructions, schema docs, the agent registry, the standards every agent follows — lives in plain Markdown files, not in platform-specific configuration. The agents themselves are just configurations that point at these files. If I ever move off the current platform, the files are what carry over; the platform is just the thing that executes them.