AI Agent Development Agency with Production LLM Evals: What to Demand

Plenty of agencies will build you an AI agent. Far fewer will hand you the thing that tells you whether it still works next month. That gap — a working demo with no evaluation harness behind it — is why so many agent projects stall after the pilot.

If you are hiring an agency for an LLM agent, the evals are not a nice-to-have line at the bottom of the SOW. They are the deliverable that makes every other deliverable verifiable.

Why an agent without evals is unshippable

An agent is non-deterministic software that calls tools and takes actions. That combination breaks the assumptions your normal QA rests on:

• The same input can produce different output. A passing manual test proves the agent worked once.
• The failure mode is confident wrongness. A traditional bug throws an exception. An agent invents a plausible answer or calls the wrong tool with well-formed arguments.
• Changing anything changes everything. A new model version, a reworded system prompt, a re-chunked knowledge base — any of these can silently degrade a behaviour that was fine yesterday.

Without a repeatable eval suite you cannot answer the only question that matters after launch: did that change make it better or worse? You end up frozen on a model version you are afraid to upgrade.

What an eval harness actually is

Not a spreadsheet of vibes. A production eval harness has four parts, and you should be able to run it yourself with one command:

1. A golden dataset. 100–300 real cases drawn from your actual domain — real support tickets, real order exceptions, real policy questions — each with an expected outcome or a rubric. Synthetic-only datasets flatter the system; they lack the messy phrasing real users produce.
2. Graders. A mix of deterministic assertions (did it call create_order with the right SKU? is the cited document ID in the retrieved set?) and model-graded rubrics for open-ended text (faithfulness to sources, tone, refusal correctness). Deterministic checks first — they are cheaper, faster, and never disagree with themselves.
3. Gates in CI. The suite runs on every pull request that touches a prompt, a tool definition, a retrieval parameter, or a model version, and it fails the build below a threshold you agreed in advance. Evals that only run when someone remembers are not gates.
4. Production tracing. Every live run logged with inputs, retrieved context, tool calls, and outputs, so real failures can be pulled back into the golden dataset. The dataset should grow from production, not from imagination.

Tooling here is mature enough that nobody needs to build it from scratch — RAGAS, LangSmith, TruLens, Braintrust, and OpenAI Evals all cover parts of this, and a plain pytest suite covers more of it than people expect. The tool matters far less than whether the four parts exist.

What to measure for an agent, specifically

RAG metrics alone are not enough once the system can act. For an agent, measure at three layers:

• Retrieval (if it uses RAG): recall@k, precision, and whether the answer’s citations actually appear in the retrieved set.
• Tool use: correct tool selected, arguments well-formed and semantically right, and — most importantly — correct refusal to act when the request is out of policy or the information is insufficient. Over-eager tool calling is the expensive failure in any agent that touches an ERP or a payment.
• Task outcome: end-to-end success on the real job, plus cost per task and p95 latency. An agent that succeeds 95% of the time at 40 seconds and $0.60 a run may be worse for your use case than one at 90%, 4 seconds, and $0.04.

Then add the operational guardrails that keep a bad day from becoming an incident: a step limit per run so a loop cannot spin indefinitely, a spend cap per task and per day, human approval required for any irreversible action, and structured logs on every tool invocation.

What to require in the SOW

Ask for these explicitly. An agency that builds evals as standard will not blink; one that does not will negotiate them out:

• The golden dataset and the graders as source in your repository, not in a vendor’s SaaS account you lose access to.
• A named acceptance threshold per metric, agreed before the build starts — that is what “done” means for a non-deterministic system.
• Evals wired into your CI, running on your infrastructure.
• Observability and tracing deployed and documented, with a runbook for triaging a bad output.
• A model-upgrade procedure: the steps to re-run the suite against a new model version and compare, so you are not stranded on a deprecated model.
• Infrastructure as Terraform in your cloud account, so the whole system is reproducible.

If evals appear in the proposal only as a bullet under “testing”, that is your signal. Ask to see an eval report from a previous build — redacted is fine. The presence of a real one, with metrics that are not all 100%, tells you more than any case study.

How this changes cost and timeline

Building the eval harness alongside the agent adds roughly 15–25% to a first engagement. It is not overhead. It is what converts a demo into something you can change without fear — and it is dramatically cheaper than discovering, three months in, that nobody can safely touch the prompt.

Expect the dataset curation to need your domain experts for a few hours. No agency can label what a correct answer looks like in your business. That collaboration is the highest-leverage time you will spend on the project.

Work with us

We build AI agents and RAG systems as fixed-scope engagements, with the eval suite, the tracing, and the Terraform in your repository at handoff — and a named pass threshold in the SOW rather than a demo at the end. If you want to talk through what the eval set for your use case would need to contain, get in touch.

Need this built? Our team does fixed-scope rag and agentic workflows — scoped, priced and delivered in weeks.