Agent Engineering Isn't Software Engineering or ML. It Needs Both.

Imagine a support agent that answers refund questions. In the demo it was flawless. Two weeks after launch, a prompt tweak fixes one complaint and quietly breaks another. A provider updates its model and tone shifts. Someone asks, “Is it better than last month?” and nobody can answer with more than a feeling.

The software engineers say the code is fine and the tests pass. The data scientists say the model is fine and the metrics looked good. Both are right, and the agent is still failing customers. That gap is where agent engineering lives.

Why neither discipline is enough alone

Software engineering gives you habits that work well on deterministic systems: unit tests, contracts between components, code review, CI, version control, rollbacks. They carry over to agents, and you should keep every one of them.

Machine learning gives you habits for systems whose behavior is learned from data: datasets, train and test splits, statistics, evaluation metrics, error analysis. These also carry over.

What neither fully covers is the combination an agent creates:

• Nondeterminism. The same input can produce different outputs on different runs. One passing run proves little.
• Tools and side effects. The agent calls APIs, writes records, sends messages. A wrong answer is no longer only a wrong answer.
• Changing data and models. Customers change what they ask. Providers change models under you.
• Uncertain evaluators. Often the judge of quality is itself a model or a rubric, and it can be wrong.
• Many interacting knobs. Model, prompt, tools, retrieval settings, temperature, retry rules. Changing one shifts the others.

Agent engineering is the practice of building agents that stay dependable despite these five conditions. It draws on both older fields and replaces neither.

Three things that evolve together

A conventional application has code, and you test the code. An agent has three assets that all change over time:

1. The agent implementation: prompts, model choice, tools, configuration, orchestration code.
2. The evaluation dataset: the examples that represent the work the agent must do, including hard and rare cases.
3. The evaluator: the code, rubric or judge model that decides whether an output is good.

If you change only the first, you may improve against a stale yardstick. If you change the dataset or evaluator without noting it, last month’s scores are no longer comparable to this month’s. Any of the three can be wrong, so you need to track versions of all of them.

Experiments are what connect the three. An experiment says: this version of the agent, on this version of the dataset, scored by this version of the evaluator, produced this result. Without that sentence, a number is just a number.

The lifecycle, stage by stage

It helps to have a shared map. This site organizes the agent development lifecycle into six stages:

Stage	The question it answers
Foundations	What makes agents different, and what vocabulary do we share?
Requirements	What must the agent do, and how will we know?
Agent design	Which models, tools, prompts and configurations will we choose, and why?
Evaluation	Do we have representative data and evaluators we can trust?
Experimentation	Which candidate is better, and how sure are we?
Release and operations	Is the evidence strong enough to ship, and how do we notice drift?

This is a loop, not a waterfall. Production monitoring feeds new examples into the dataset, which changes what you test next.

The practice: start here on Monday

1. Name your three assets. For one agent, write down where the implementation, the dataset and the evaluator each live, and who owns each.
2. Version them together. Record which dataset and evaluator version produced every reported score. A spreadsheet row is enough to begin.
3. Keep your software tests. Test tool calls, input validation and error handling the usual way. Deterministic parts deserve deterministic tests.
4. Add a small evaluation set. Collect 30 to 50 real or realistic cases, including a few you expect to fail. Treat it as code: reviewed, versioned, never edited casually.
5. Run repeats. For each candidate change, run the set more than once and look at the spread, not one score.
6. Write the decision rule before the result. “We ship if quality is at least as good and cost does not rise” is easy to follow. A rule invented after seeing the numbers is easy to bend.
7. Feed production back. Pick one failure per week from real traffic and add it to the dataset.

A worked example (illustrative)

Imagine a team changes the refund agent’s prompt. Here is how a log entry changes when it records all three assets:

Field	Before	After
Agent version	prompt v7, model A	prompt v8, model A
Dataset	refunds-set v3 (40 cases)	refunds-set v3 (40 cases)
Evaluator	rubric v2	rubric v2
Runs	5	5
Correct answers per run	31, 33, 32, 30, 33	34, 29, 35, 28, 35

The average rose slightly, but the spread widened. A reader with only the averages would call it an improvement. A reader with the spread would ask whether v8 is less stable, and would look at the cases that flip between runs. The numbers here are invented for illustration. The point is the habit of recording and reading them.

What this doesn’t solve

• Small datasets give weak evidence. Forty cases can show a large regression but rarely a small gain. How big a dataset you need depends on the effect you care about, and no formula removes the need for judgment.
• Evaluators are not ground truth. A judge model has its own biases. Spot-checking its verdicts against human review is still necessary.
• Methods are young. Practices such as eval-driven development are promising and widely discussed, but their long-term benefits are not yet established by strong comparative evidence. Evidence that would help: studies comparing teams or projects with and without a practice, using agreed outcome measures.
• It does not remove the need for the older fields. An agent with good evaluation but sloppy deployment still fails.

---

See it in practice

Traigent, the company behind this publication, makes a Python SDK built around these ideas. Per its public README, you decorate a function, name your objectives and point it at an evaluation dataset, and it runs trials comparing configurations. You can try it without provider calls using its mock mode. Start at traigent.ai.

---

Who this is for: engineering leaders, software engineers and data scientists getting oriented in agent engineering.