Eval-Driven, Spec-Driven, Experiment-Driven: What Each Asks of You

Imagine a standup where three people use three phrases. One says the team should be “eval-driven.” Another wants to be “spec-driven.” A third says what is really needed is “experiment-driven.” They nod as if they agree, and each has a different workflow in mind.

These labels are not settled standards. This article gives working definitions, says what each asks of you in practice, and shows how they combine. We use these meanings and flag where others exist.

Three definitions

Eval-driven development (EDD). You write the evaluation before, or alongside, the agent change: a set of examples plus a way to score outputs. You improve the agent until the evaluation says it is good enough. It is the agent counterpart of test-driven development (TDD).

Spec-driven development (SDD). Careful: this term has two meanings. In coding-agent tooling it often means writing a specification that an AI coding assistant then implements. Here we mean something different: a machine-checkable specification of the agent itself, its allowed configurations, its objectives and its constraints, written before experiments run, so a run can be validated against it.

Experiment-driven development. You treat each change as a hypothesis: “this model or prompt will reduce cost without lowering quality.” You compare candidates under controlled conditions, repeat runs, and decide by a rule fixed in advance.

Why agents need more than TDD

TDD works because a test is a deterministic yes or no. That carries over for the deterministic parts of an agent: tool-call formats, input validation, refusals that must always happen. Keep writing those tests first.

It breaks down for quality. The same input can pass on one run and fail on the next, so a single assertion is not evidence. From machine learning you borrow datasets and statistics: score many cases, repeat, read the spread. The three approaches below build on that.

What each asks of you

	EDD	SDD (this article’s sense)	Experiment-driven
Core question	Is it good on representative cases?	Is the candidate allowed, and what must hold?	Is B really better than A?
Main artifact	Evaluation set plus scorer	Written spec of options, objectives, constraints	Experiment plan and result log
Who leads	QA, data scientist	Software engineer, architect	Data scientist
When it runs	Every change	Before running; reject invalid candidates early	When choosing between candidates
Typical failure	Evaluating on the cases you tuned on; weak scorer	Spec drifts from reality; becomes ceremony	Too few runs; deciding after seeing the numbers
Evidence needed	Scorer agrees with human review	Spec catches real invalid configurations	Decision rule fixed beforehand; repeated runs

The practice: combine them in order

1. Write the deterministic tests first (TDD). Tool calls, schemas, hard refusals.
2. Build the evaluation (EDD). Collect cases, including hard ones. Write the scorer. Check the scorer against a human reviewer on a sample.
3. Split your data. Keep cases you tune on separate from cases you report on. If you adjust a threshold on the same cases you then report, the report describes the fitting, not the system.
4. Write the spec (SDD). List the options you will vary (models, prompts, retrieval settings), the objectives (quality, cost, latency), and the constraints (never exceed a cost cap, never use a disallowed tool). Even a short YAML or doc counts.
5. Plan the experiment. State the comparison, the number of runs, and the decision rule before you run.
6. Run, read the spread, decide by the rule. Record the agent version, dataset version and scorer version with the result.
7. Feed failures back. Cases that surprised you join the evaluation set.

A worked example (illustrative)

Imagine a team choosing a model for a summarization agent. This sketch is pseudo-code, not a particular tool.

spec:
  vary:        model in {small, medium}, temperature in {0.0, 0.5}
  objectives:  quality (maximize), cost (minimize)
  constraints: cost per task <= cap; no tool calls outside allowed list

eval:
  dataset:     summaries-v2  (tune on 60 cases, report on a separate 40)
  scorer:      rubric-v1  (agreement with human reviewer checked on 20 cases)

plan:
  runs per candidate: 5
  decision rule: pick lowest-cost candidate whose worst run is within 2 points of best

The spec rejects a candidate that uses a disallowed tool before any run is spent. The evaluation gives each candidate a score distribution. The plan stops the team from picking whichever run looked best. The numbers in the rule are invented and would be set per project.

Where each one pays off first

If you can only start one thing, start with the evaluation. Without a set of cases and a scorer you trust, a spec has nothing to be checked against and an experiment has nothing to measure. Add the experiment discipline next, because it protects you from the most common mistake: picking the run that looked best. Add the spec when your option space grows large enough that invalid or unsafe combinations start wasting runs. Small teams with two or three options can keep the spec to a paragraph.

What this doesn’t solve

• No method here is proven. These workflows are sensible and increasingly discussed, but strong comparative evidence is thin. Evidence that would help: trials where comparable teams use one approach versus another and are compared on defect escape rate and time to fix.
• A bad scorer defeats all three. If the evaluator is wrong, you will efficiently optimize toward the wrong thing.
• Small sets limit what you can claim. Run-to-run noise can swamp small gains.
• Specs can become paperwork. If nobody validates against the spec, it is documentation, not a safeguard.
• Overfitting is quiet. Repeated tuning against one set slowly gives optimistic numbers. Keep a final holdout you rarely touch.
• These sit on top of, not instead of, engineering. Deployment, security and monitoring still apply.

---

See it in practice

Traigent publishes a Python SDK for the experiment loop. Per its public README, you point a decorated function at an evaluation dataset, declare objectives, and it suggests configurations, runs them against the dataset, scores them with your evaluator and records results. It supports custom evaluators and multi-objective runs. See traigent.ai.

---

Who this is for: software engineers, QA engineers and data scientists who build, test and compare agents.