Down for Hours Because of a Job That Wasn't Even an HTTP Request
A batch job took down the web service whose entire purpose is to answer requests. The fix was one afternoon. The 24 hours after the fix is the real story.
My web service went down for hours because of a job that wasn’t even an HTTP request.
Sit with that for a second. The one thing a web service exists to do is answer requests. Mine could not answer a single one, and the thing holding it hostage never made a request in its life. It was a batch job, running quietly inside the same process, and it ate the whole service alive.
The architecture fix took an afternoon. The 24 hours after the fix is the part worth telling.
The convenient shortcut
The daily ETL ran as FastAPI BackgroundTasks, inside the web service, in the same process as the request handlers.
It was convenient. You return a response, you kick off the background work, it runs after. It worked in testing, because in testing the job was small and nobody was hitting the API while it ran. So it shipped. That is usually how these things get in: they are not wrong on the day you write them. They are wrong on a day you are not thinking about.
The morning it went hostage
The batch run iterated about 40 clients, and it monopolized the single uvicorn worker for the whole run.
Every HTTP request queued up behind it. The admin panel, the API, and the health check, all of it, waiting on a job that had no intention of yielding. Latency climbed past a minute, then past an hour, then the 504s started. The health check itself timed out, which meant every automated system watching the service also thought it was dead, because from the outside it was. A job that was never a request had taken the process hostage, and everything that was a request got to wait.
The fix was an afternoon
The fix is not subtle once you have seen the failure. Get the batch work out of the web process entirely.
Same container image, different entrypoint, run the ETL as a fire-and-forget batch job on Cloud Run Jobs instead of inside the server. Batch work and request serving cannot share a process. A long-running job in a request worker is a self-inflicted outage that is just waiting for the day the job gets big enough to trip it. That part, the actual lesson everyone remembers, took an afternoon.
Then I spent a day fixing the fix.
Then I spent a day fixing the fix
Three traps, stacked, and each one cost me hours.
The first was a dependency landmine. A file:/// local path had been committed to requirements.txt at some point. It resolved fine on my laptop, because on my laptop the file was right there, so nothing local ever complained. It broke the cloud build four times in a row before I understood what I was looking at. A dependency that only exists on the machine that wrote it is a landmine that lies dormant until the exact moment you need to ship a fix under pressure.
The second was the tooling default that hid the truth. I was querying the build list to see why the deploy kept failing, and I kept getting nothing back. I read “nothing” as “no builds ran.” What had actually happened was that I queried without specifying a region, the default region was empty, and the command cheerfully returned an empty result for a place nothing was happening. A wrong default in a tool can hide your root cause for a day while you chase ghosts in the wrong place.
What it taught
The outage taught me one architecture rule in an afternoon: never co-locate batch work and request serving. That is the headline, and it is the least of it.
The three humility rules took the next day, and they are the ones I actually carry now. Verify a dependency resolves somewhere other than the machine that wrote it. Distrust an empty result from a tool until you have ruled out that you asked it the wrong question. And treat a confident fix, from an agent or from yourself, as a claim to check rather than an answer to believe.
The fix is never the hard part. The hard part is everything that goes wrong while you are certain you already know the answer.