Before My Data Agent Could Answer One Question, It Had to Pass 30
I built an agent that answers questions about marketing data in plain English. Before it could answer a single real one, it had to pass 30 it had never seen. The hard part was never the model. It was writing down what a correct answer even is.
Before my data agent could answer a single question, it had to pass 30 it had never seen.
I built an agent that answers questions about marketing data in plain English. Ask it what you spent last month, or which campaign drove the most leads, and it writes the query, runs it against the warehouse, and answers. Useful. Also dangerous, because a confident wrong answer reads exactly like a right one.
The hard part was never the model. It was writing down what a correct answer even is.
The temptation
The easy version of this is a demo. You wire a language model to a BigQuery warehouse, hand it the schema, ask it a few questions, and the answers look right. Ship it. That is how most of these get built, and it is why most of them are quietly wrong.
I did not want a thing that looked right on the questions I happened to try. I wanted a thing I could put in front of a client’s real numbers and trust. Those are different problems. The first one is a weekend. The second one is the actual work.
The gate instead
So before the endpoint went in front of anyone, it had to clear an eval. Three parts.
- A per-client metric contract. The canonical definition of every number that matters: spend, cost per lead, cost per acquisition, what counts as a lead, which date window, which attribution rule, and the known gotchas for each platform. Written down, not assumed.
- 20 to 30 golden questions with known answers, across a spread: simple lookups, comparisons, multi-source questions that join two platforms, edge cases, and adversarial ones meant to break it.
- A hard bar. At least 85% accuracy to pass, and 100% of injection attempts blocked. The second number has no tolerance. 85% right is a judgment call. One leaked cross-client row is not.
A question fails if the answer is wrong. It also fails if the answer is right but the agent took a path I would not allow in production.
The boring assertions
Here is the part that gets skipped, and it is the whole point. I do not grade the answers with another language model.
Using an LLM to judge an LLM feels rigorous and is mostly vibes. It is non-deterministic, it costs money, and when it disagrees with you, you cannot tell whether the answer was wrong or the judge was. So the spine of the eval is plain code.
- Structural SQL checks, not string matching. I do not care whether the agent’s query is byte-for-byte mine. I care whether it groups by the right dimension, filters the right date range, and reads from the right tables. Two correct queries can look nothing alike.
- Answer-range checks. The number has to land in a defensible band, not equal a brittle golden string.
- Budgets. Under 10 cents a question, under 15 seconds. A correct answer that costs a dollar and takes a minute is a failure of a different kind.
Then the adversarial line, where everything has to hard-block: prompt injection buried inside a question, a request for another client’s data, and the nasty one, a schema-changing statement dressed up to look like a question. None of those is allowed to succeed, ever. That is the 100% bar.
Where evals stop
One honest caveat, because I almost got this wrong. The deterministic checks I described are cheap and stable, so they run on every change, like a test suite. The richer, model-in-the-loop evaluation is not that. It is non-deterministic and it needs an API key, which means it costs money and can flake. So I do not gate the build on it.
That line matters. Characterization tests gate the pipeline: fast, repeatable, and a red one stops the merge. The LLM-backed eval informs my judgment instead. I run it by hand when I have changed something that could move quality, and I read the results like a reviewer, not like a green checkmark. Wiring a non-deterministic judge into your CI is how you end up with a pipeline that fails for reasons nobody can reproduce. Keep the flaky thing out of the gate and in your hands.
What it taught
The eval harness was not a one-time hurdle to clear before launch. It became the regression test for every prompt change after. Every time I tweaked an instruction, the 30 questions told me whether I had made it better or quietly broken something three questions over.
The lesson I keep coming back to: writing down what “correct” means is the actual AI work. The model is the easy part. It will hand you a confident, well-formatted, wrong answer, and it will do it in plain English so it reads as true. The only thing standing between that and a number a client acts on is an answer key you were disciplined enough to write first.
The eval is usually the part that turns a demo into something a client will trust with their real accounts.