How Agent Engineering Changes Your Job, Role by Role
Seven roles, seven new challenges. For each one: what changes when you ship an agent, one habit to adopt this week, and one question to ask your team on Monday.
Imagine a team shipping its first customer-facing agent. The product manager writes a spec the way she always has. The engineers build it. QA runs the test plan. The data scientist is brought in at the end to “check the quality.” Three weeks later nobody agrees on whether it works, because each person measured something different.
Agents do not remove anyone’s job. They shift where each job’s hard parts are. This article walks through seven roles. For each: the new challenge, one habit to adopt, and one question to ask your team. Agent engineering draws on software engineering and machine learning; it adds to both and replaces neither. If you want the big picture first, read why it needs both disciplines.
Engineering leaders
New challenge: “Is it ready?” no longer has a yes/no answer from a test suite. An agent can pass today and behave differently tomorrow.
Habit: Ask for a release decision in the form “here is the evidence, here is what it does not cover, here is who accepts the risk.” Insist on all three parts.
Question for your team: What would make us roll this release back, and how would we find out within a day?
Product managers
Browse product manager articles.
New challenge: Requirements cannot say “the agent answers correctly.” They have to say how correctness is measured, on which examples, and what level is good enough.
Habit: For every feature, write one acceptance criterion with a threshold, a dataset and a measurement method. Separate must-never-happen rules from quality targets.
Question: Which three user requests would embarrass us most if the agent got them wrong, and are they in the test set?
Software engineers
Browse software engineer articles.
New challenge: Your code may be correct while the system is not. Behavior also depends on prompts, model versions and tool descriptions, which are easy to change and hard to review.
Habit: Treat prompts, tool schemas and model settings as versioned configuration. Change one thing at a time, and record which version produced which result.
Question: If this agent behaves differently on Friday than Thursday, can we say exactly what changed?
Data scientists
Browse data scientist articles.
New challenge: There is often no training run and no fixed label set. You are asked to support claims about a system that varies from run to run and is judged by an imperfect evaluator.
Habit: Report spread, not just averages. State what comparison you made, how many runs, and what difference you would consider meaningful before you look at results.
Question: How do we know our evaluator agrees with a careful human reviewer, and on how many cases did we check?
Data engineers
Browse data engineer articles.
New challenge: The valuable datasets are now evaluation datasets and traces: real conversations, tool calls, outcomes. They are messy, sensitive and constantly growing.
Habit: Build a path from production to a reviewed evaluation set, with clear rules for what may be copied, redacted or excluded.
Question: Can we take a bad production interaction and turn it into a regression test case in under a day, without exposing customer data?
QA engineers
New challenge: The same test can pass and fail on consecutive runs without any code change. “Expected output equals actual output” often does not apply.
Habit: Split checks into two kinds. Deterministic checks (did it call the right tool, did it refuse a forbidden action) get strict pass/fail. Quality checks get repeated runs and a pass rate.
Question: Which behaviors do we require on every single run, and which are we content to measure as a rate?
Architects
New challenge: The design decisions that matter include tool boundaries, permissions, fallbacks and cost ceilings. A tool the agent can call is a capability it can misuse.
Habit: Give each tool the narrowest permission that works, and decide in advance what happens when the model, a tool or the network is unavailable.
Question: What is the worst action this agent could take with the access we gave it, and what stops it?
A worked example (illustrative)
Imagine the team from the opening, rebuilt around shared habits. One page, owned jointly, holds:
| Item | Who keeps it | Review rhythm |
|---|---|---|
| Acceptance criteria with thresholds | Product manager | Each release |
| Must-never-happen checks | QA and architect | Each release |
| Evaluation dataset (versioned) | Data engineer and QA | Weekly additions |
| Evaluator and its agreement check | Data scientist | Monthly |
| Prompt, model and tool versions | Software engineer | Every change |
| Release evidence and risk owner | Leader | Each release |
None of this requires new tooling to begin. A shared document and a weekly thirty-minute review is enough to expose where roles are measuring different things.
The practice: try this week
- Pick one agent your team owns or plans to build.
- Have each person on the team answer their role’s question above, in writing.
- Compare answers in one meeting. Disagreements are the finding.
- Assign the owner for each row of the table above.
- Adopt one habit per person. Not seven; one.
A note on reading this list
You will notice that the habits overlap. Every role ends up caring about the same few things: a saved set of realistic examples, a written definition of “good,” a record of what changed, and a named person who decides. That overlap is the point. Agent engineering works when these shared artifacts exist, because they give the product manager, the tester and the data scientist something common to argue about. Without them, each role optimizes a private idea of quality.
What this doesn’t solve
- Role boundaries vary. In a small team one person holds four of these hats, and the habits still apply.
- A habit is not a guarantee. Writing thresholds down does not make them the right thresholds.
- The practices here are reasonable and widely discussed, not proven. Evidence that would strengthen them includes before/after comparisons across teams using agreed outcome measures, such as incident rate or time to detect regressions.
Who this is for: engineering leaders, product managers, software engineers, data scientists, data engineers, QA engineers and architects working on or alongside agents.
Filed under: