The Orchestrator That Wrote 549 GB and Panicked the Kernel
My first autonomous-agent orchestrator wrote 549 GB to disk in one night, exhausted memory, and triggered a kernel panic. Then it kept dispatching into the dying machine. That night taught me where guardrails actually belong.
I built an AI agent that crashed my computer four times in one night. That failure designed everything I built afterward.
It was my first real attempt at an autonomous-agent orchestrator. One coding session, a buggy SQLite loop, and no guardrails worth the name. The agents had reach. Eventually they used it.
The cascade
It wrote 549 GB to disk. Memory filled, then kept filling: RAM at 100% compressed, 57 swap files spun up to buy room that did not exist. The window server starved for 188 seconds. Long enough that the OS gave up and the kernel panicked.
Here is the part I still think about. After the panic, the machine came back, and the orchestrator resumed dispatching. Into the dying host. Ten OOM-killed runs in a row. One run had turned into 17 runs, 14 failures, 31 escalations. The system was failing, and its response to failing was to do more of the thing that was failing.
Why it could happen
None of this needed a smarter model. It needed defenses that were not there.
- No process-group isolation. I never set
start_new_session=True, so the runaway child outlived every parent kill and orphaned itself. I could not stop it. - No memory monitoring. Nothing watched RSS. Nothing noticed the machine running out of room until the room was gone.
- No pre-dispatch health gate. The orchestrator never asked “is the host okay right now?” before launching the next task.
- No global concurrency cap. Nothing bounded how much it could try to do at once.
- No post-crash backoff. After the panic, it launched 5 tasks in 13 minutes while the host was still unstable.
Every one of those is boring. That is the point. The fix for an agent that ran wild was not cleverness. It was the unglamorous plumbing I had skipped.
The turn
The fix was structural, not prompt-level. None of these are clever:
- Process-group isolation, so a child can always be killed.
- A memory watchdog that halts dispatch before the host is in danger.
- A fail-closed pre-dispatch health gate: if it cannot confirm the host is healthy, it does not launch.
- A circuit breaker: when failures cluster, dispatch halts and backs off on an exponential cooldown instead of hammering a host that is already down.
- An E-STOP that only a human can clear.
Notice where all of that lives. Not in the prompt. In the tools, in the runner, in the machine itself. I had spent the night before trying to write a better prompt, a more careful instruction, a smarter set of rules for the agent to follow. The prompt is the wrong place for a guardrail. A guardrail you can argue your way past is not a guardrail. Pull it down into the tool, and run the agents under scoped service accounts, so the blast radius stays small even when everything else goes wrong.
That night the blast radius was one machine. In production it is a client’s account, their data, their bill. The mechanism changes; the principle does not. Decide up front how much damage is possible, and make that ceiling something the system enforces, not something you ask the agent to respect.
What it taught
The lesson was not “write better agents.” It was this: assume the agent will do the worst possible thing at the worst possible time, and make the machine, not the prompt, refuse to let it.
Almost everything I do around guardrails and governance traces back to that night.