The Five Things an Agent Framework Won't Give You

I have now built the same missing layer around enough agents that I do not think of it as optional anymore.

The framework changes. The missing pieces do not. Every time the work gets close to a real account, I end up building the same five things: durable state, idempotency, approval gates, traces, and evals. They are not the exciting part of the demo. They are the part that decides whether the demo can survive Tuesday.

This is not an argument against frameworks. I use them. They are useful. I just had to learn, a few times, that the framework’s job ends earlier than I wanted it to.

1. Durable state that survives a restart

A demo keeps its state in memory. A production run does not get that luxury.

The process gets redeployed mid-task. The container gets evicted. The human closes the laptop. If the agent cannot resume from the last committed step, every long-running task is one deploy away from starting over.

The primitive you actually need is a checkpoint keyed by a stable run identifier, written transactionally with the side effect it describes:

# the smallest durable step that is safe to retry
async def execute_step(step, store):
    if store.already_done(step.id):
        return store.result_for(step.id)

    result = await step.run(idempotency_key=step.id)
    store.commit(step.id, result)   # state + effect, one transaction
    return result

Notice there is nothing clever here. That is the point. The hard part is deciding what counts as a step, not inventing a fancy place to store it.

2. Idempotency on every external effect

The moment an agent can retry, it can retry after it already did the thing.

That sounds obvious written down. It was not obvious enough to me at first. I was worried about the model saying the wrong thing, and I had not spent enough time worrying about the runner doing the right thing twice.

An idempotency_key on every outbound call is the difference between “the run recovered” and “the customer was billed twice.” Treat it as non-negotiable for anything that mutates the outside world.

An autonomous system you can’t safely retry is not autonomous. It’s a single point of failure with a chat interface.

3. A human-in-the-loop that is real

“Human-in-the-loop” is usually drawn as a box with a checkmark. That drawing hides almost all of the work.

In practice it is a suspension of the run. The agent reaches a gated action, persists its context, emits a request for approval, and stops, possibly for days, until a person responds. That is not a button. It is a distributed state machine, and the framework will hand you maybe half of it.

• The gate must be declarative: which actions require approval is policy, not code buried in a node.
• The suspended run must be fully serializable, including pending tool calls.
• Resuming must be idempotent, because approvals get clicked twice.

4. Observability built for non-determinism

You cannot debug an agent from request logs alone. I tried to reason from the outside too many times before accepting that.

You need the full trace: the prompt, the tools considered, the arguments chosen, the result, and the branch taken. Capture it even when nothing went wrong, because “nothing went wrong” is exactly what you will be asked to prove later.

The config that earns its place looks boring on purpose:

# the orchestrator depends on the port, never the vendor
orchestrator:
  driver: engine-a        # swap target: engine-b
  checkpoint: postgres
  resume: by_run_id
  trace:
    capture: "always"     # not just on error
    sink: otel

5. Evaluation you own

The framework will not tell you whether your agent got better this week.

That belongs to your own regression suite: small, fast, run on every change. Public benchmarks do not know your tenant rules, your approval policy, your failure modes, or the weird little questions your customers actually ask. Your evals have to know those things.

I’ll make that full argument in a later piece. For now, assume that if you cannot measure a change, you cannot ship one with a straight face.

Where this leaves the framework

Use the framework for the graph, the tool-calling, the retries it does give you. Build the five primitives above against your own interfaces, and keep the framework on the far side of a seam you control.

That way, when the next orchestrator arrives, you port an adapter. You do not rewrite the product.

That seam is the subject of the next essay.