Right-sizing your agent swarm without chasing ghosts.

We had the kind of graph that makes infrastructure teams reach for a rollback: container CPU climbing in a perfect straight line after deploy. At the same time, a coder worker looked stuck above 1 GB of RAM while idle.

Both looked like production problems. Both turned out to be measurement artifacts. The CPU line was an average over a cumulative counter. The memory plateau was page cache plus a long-lived Bun/Node high-water mark. The swarm was fine; the dashboard was asking the wrong question.

The numbers we actually run

Agent containers do not have one resource profile. A lead agent mostly coordinates. A content worker may sit close to idle. A coding worker can read a repository, spawn a harness process, run TypeScript, execute tests, and stream tool calls for an hour.

That is the practical sizing split: implementation workers get headroom, leads get enough room to coordinate without pressure, and light specialists stay small until their actual workload says otherwise. Role names are a starting point, not a law. If a tester runs browser automation or repo-wide builds, size it like a heavy worker.

Kubernetes pod sizing

In Kubernetes, we set requests to the steady-state budget the scheduler should reserve and limits to the burst ceiling each pod can use during an active session. Heavy workers need the widest gap because build tools, language servers, test runners, and provider subprocesses can spike together.

Kubernetes is stricter about memory bursts than the generated Docker Compose setup I use for single-host deployments. Exceeding limits.memory is an immediate OOMKill, and pods running above requests.memory are the first eviction candidates under node memory pressure. Compose sets no hard memory ceiling by default, so Kubernetes requests should sit near the real working set and limits need real headroom. For a heavy coder, I budget the limit as MAX_CONCURRENT_TASKS * per-session peak (~2 GB) + page-cache headroom; at the default MAX_CONCURRENT_TASKS=1, that is one active task peaking around 2 GB, and raising it multiplies the peak. For the lead, request == limit gives Guaranteed QoS and keeps the orchestrator out of memory-pressure eviction; its observed peak is around 830 MB, so a modest equal request/limit is enough.

Docker Compose on a single host

On a single VPS or bare-metal host, we leave reserve capacity for the database, Docker, the kernel page cache, logs, and deploy-time overlap. The practical rule: allocate only 70-80% of host memory to steady-state containers and keep at least 1-2 vCPU uncommitted on busy boxes.

Concurrency is the real multiplier

The swarm scales most predictably when each local-runtime worker runs one active task at a time. The knob is MAX_CONCURRENT_TASKS; the generated Docker Compose default is MAX_CONCURRENT_TASKS=1, which means one active task per worker. Increasing per-worker concurrency can work for thin relay workers or low-tool tasks, but it multiplies memory peaks and makes local builds fight inside the same container.

Trap one: CPU that climbs because the query climbs

The CPU graph that triggered the investigation was suspicious because it was too clean. Real CPU usage jitters. This line rose with mechanical precision and reset on deploy.

The problem was the panel, not the container. It plotted container.cpu.utilization with average time aggregation, treating a cumulative-since-boot counter like an instantaneous usage gauge. Averaging a monotonic counter produces a fake climb by design.

# Wrong mental model:
avg(container.cpu.utilization)

# Better instantaneous CPU:
rate(container.cpu.usage.total)

We fixed the SigNoz Container CPU Percent panel by switching the time aggregation from average to rate. The straight-line climb disappeared. The actual instantaneous CPU was flat.

Trap two: memory that is high but reclaimable

The memory graph was more subtle. Our coder worker plateaued around 1,145 MB at idle, peaked at 1,979 MB during a heavy coding session, and reset to about 470 MB after the next redeploy. That shape can look like a leak if you only stare at container.memory.usage.total.

It was mostly two normal effects stacked together. First, cgroup page cache: coding sessions read and write a lot of files, and Linux keeps file-backed pages around until pressure forces eviction. Second, the high-water mark of a long-lived Bun/Node runner: heap can be reusable inside the runtime without RSS immediately returning to the OS.

# Better leak-triage panel:
working set = usage.total - inactive_file

Total memory still matters for OOM risk. It is not the same as a leak signal. For leak triage, graph working set and look for growth that survives restarts. If the plateau clears on every redeploy, start with page cache and runtime high-water marks before you go hunting through application code.

Harness choice is part of sizing

The harness provider changes the operational risk profile. claude is our most reliable general harness. codex is stable and good for deterministic implementation, review, and structured-output work. pi is fine for content and QA. opencode has produced intermittent session errors within roughly 10 seconds of task start, so we avoid it for determinism-critical workflow nodes.

That is not just reliability bookkeeping. It affects how much headroom you give a container and what work you route there. A Codex or Claude implementation worker gets heavy-worker resources. A content worker can stay light until the workflow adds builds, browser work, or large local processing.

Fix real leaks, then fix the dashboard

The same operational thread did uncover real cleanup work. PR #675, “Fix runner and MCP transport leaks,” bounded runner task-keyed bookkeeping and closed idle MCP owner/user transports that survived unclean disconnects. That was real.

The CPU climb after that was not. That distinction is the whole lesson: production agent infrastructure needs both code fixes and measurement discipline. If you mix them up, you either ignore a real leak or waste a morning tuning containers around a dashboard artifact.