The Most Important Table Is the One That Does Nothing
The smartest thing an anomaly detector can do is nothing, in the sense that it never touches a production account itself. It hands the finding to a table, and something else, under its own policy, decides whether to act.
The smartest thing an anomaly detector I built can do is nothing. Not nothing as in sit idle. Nothing as in it never reaches out and touches a real account itself.
A detector that’s wrong about a pattern costs you a false alert. Annoying, cheap to recover from. A detector that’s wrong and can also write to a client’s live account costs you an incident. The fix isn’t a smarter detector. It’s a table between the two.
Two planes, one seam
I split the system into two planes. A read and analysis plane does the actual work of noticing something: ingest, detect, recommend, serve. A separate action plane is the only thing allowed to touch a real account.
Detectors never write to a client’s ad account directly. The only thing a detector is allowed to do about a finding is enqueue it, into a queue table it can never drain itself.
The table itself
An action_queue row moves through a fixed lifecycle: new, claimed, in_progress, done, failed, or dismissed. A separate consumer polls that queue, claims a row, executes under its own policy, and updates the status when it’s finished.
That queue is the whole boundary. Nothing about it is clever. Each row is a plain typed record, the same kind of typed boundary I use everywhere else in this stack, not a builder object handed to me by some framework’s task API. It’s a table with a status column, and that’s exactly why it works. It’s durable. It’s inspectable. And it’s owned by something other than the thing that filled it.
The bookkeeping that makes it teachable
Every detection emits two linked rows. One for a person: a human-digest notification. One for the machine: the action_queue row. They’re tied together by a foreign key, so they can’t drift apart, duplicate, or go missing independently of each other.
The row is a request. The write happens somewhere else, by something else, later, under someone else’s policy. That sentence is the whole design. I keep coming back to it.
The lifecycle discipline that keeps it honest
A first sighting is NEW. A worsening issue updates the same row in place instead of opening a new one and paging me twice for one problem. A recovering metric only clears once it shows sustained improvement, not the instant it crosses back over a threshold. That’s hysteresis, not a coin flip at the boundary. Dedup keys mean a persistent problem is one row I watch evolve, not forty alerts I have to correlate by hand at 2am.
The edge case that actually taught me something
A missing data source has to suspend threshold evaluation. It cannot be silently read as “healthy.”
Absence of data and confirmation of health are not the same event. Treating them as the same event is exactly how a real problem hides behind a gap nobody noticed. The first version of this system got that wrong, and it took a genuine near-miss to see it. Silence isn’t a signal. It’s the absence of one, and the system has to say so out loud instead of defaulting to green.
The part of the system that’s clever, the detector, is exactly the part I don’t trust to act. The part that’s dumb, a table with a status column, is the part the whole thing hinges on. Build the boring seam first.