30 September 2026 · Essay

The agent is not the model

A model answers the next local question it is given. An agent is what you get after that model is placed in a system. Reporting the system's behavior as the model's capability changes the subject of the measurement.

Two objects

Zaharia, Khattab, Chen, Davis, Miller, Potts, Zou, Carbin, Frankle, Rao and Ghodsi drew the distinction cleanly in February 2024. An AI model, in their terms, is a statistical model: a transformer that predicts a token. A compound system tackles a task with several interacting parts, including repeated model calls, retrievers and external tools. The control logic may be ordinary code, or it may itself be a model calling tools. Either way, the thing that acts is no longer the weights.

An agent is a compound system with a loop. At each step the model is asked for an action, the environment answers, and the next call sees that answer. Yao and colleagues made the loop explicit in ReAct: reasoning traces and task actions are interleaved, and the action's observation comes back into the context. The trace is not a property of the pretrained model. It is a property of the model plus the instruction that demanded the trace, plus the tool that produced the observation, plus whatever was allowed to remain in context.

Write the observed behavior as a function of the parts that can actually change it:

B = A(m, i, t, μ, e, f, p, o)

m is the model. i is the instruction. t is the set of tools. μ is memory, including whatever the scaffold keeps or discards between steps. e is the environment. f is feedback, including tests, rewards and human corrections. p is permission. o is orchestration: who is called, in what order, and when the loop stops. A is the agent, meaning the whole arrangement. B is what a log, a benchmark or a user can observe.

This is a map, not an identification theorem. The terms are not independent. A tool definition changes the instruction. A permission changes which tools exist in practice. Memory changes what the model is shown, which is a change to the input, not to the weights. The point of the map is narrower. If two runs differ in any argument other than m, a difference in B is not evidence about m.

What you are looking at

Turn parts off. The model stays the same. The behavior you would be scoring does not.

The same model, a different system

Yang, Jimenez, Wettig, Lieret, Yao, Narasimhan and Press built SWE-agent around one claim: a language-model agent is a new kind of user, and the interface it is given changes what it can do. Holding GPT-4 Turbo fixed, their agent-computer interface resolved 12.47% of SWE-bench. On the Lite split, that interface solved 10.7 percentage points more instances than the same model given only a Linux shell. They also report a 64% relative increase over that shell-only setting, again with the same model. Linting inside the editor, the shape of search results, the size of the file window and how much history is kept each moved the result. None of those are training runs.

The interface was designed by watching failures on a development split. That is the ordinary way an engineer improves a system, and it is also how a result stops being a pure measurement of the model. The paper is honest about the object it measured: an agent using a particular interface. It does not say that GPT-4 Turbo, as a model, "scores 12.47% at software engineering."

The other direction matters just as much. Kapoor, Stroebl, Siegel, Nadgir and Narayanan showed that several agent architectures reported as state of the art on HumanEval did not beat much simpler baselines once cost was held in view. Retrying, warming and escalating a model produced no significant accuracy gain over the best of those architectures in their comparison, at lower cost. They had not found a paper that had run those simple baselines against the complex agents. A gain attributed to an agent design can be a gain from calling the model again.

Those two results do not cancel. SWE-agent is evidence that the surrounding system can dominate on a hard, interactive task. Kapoor and colleagues are evidence that a leaderboard gain can be the model, used more than once, wearing an architecture's name. You cannot tell which paper you are in unless the comparison says what was held fixed.

What the slogan gets wrong

"The model does not matter" is the equal and opposite mistake. A weak model inside a careful scaffold still proposes the action. SWE-agent's own transfer is uneven: the interface built for GPT-4 Turbo, run with Claude 3 Opus, resolved 10.5% of SWE-bench, not the same figure. Kapoor and colleagues' other warning is the pair of this one. Benchmarks built to compare models, and benchmarks built to choose a system for a real task, answer different questions. A downstream choice cares about the dollar cost of the whole loop. A model comparison that folds an agent around each model has measured A, then talked about m.

Tool use can also be moved into training. Schick and colleagues' Toolformer trains a model to decide when to call a calculator, a search engine or a translation system, and to continue from the result. After that, "the model" already contains a habit of tool use. The map still applies. It tells you to say which part moved. Fine-tuning m is not the same intervention as editing t at inference, even when both increase B.

What this does not establish

• It does not estimate the share of recent agent progress that belongs to models, scaffolds, tools or evaluation practice. That would be a factorial comparison, model by system, on a task whose holdout was not used to design the system.
• SWE-agent's margin is one benchmark family and one generation of models. A later model can shrink a scaffold gap, or move it somewhere the scaffold was never tested.
• HumanEval is a weak stand-in for software engineering. Kapoor and colleagues say so. A result there is evidence about that benchmark's agents, not about agents in general.
• The eight-term map will not survive contact with a real log unchanged. Use it to force the question "what else changed?", not as a taxonomy to file work into.
• This is not a Sekiya result. Sekiya's public programme on autonomous-agent behavior has not yet established findings. Nothing here should be read as that programme.

The comparison worth running

The useful next measurement is boring on purpose. Fix m. Vary one of i, t, μ, e, f, p, o. Report B, and report the cost of producing B. Then fix the system and vary m. The two tables answer different questions: what the scaffold is worth, and what the model is worth inside that scaffold. A single leaderboard number answers neither.

Until that table exists, the disciplined sentence is the long one. Name the model. Name the interface, the tools, the memory policy, the permissions and the stop rule. Say which of them the comparison changed. Then say what was observed.

Sources

1. Zaharia, Khattab, Chen, Davis, Miller, Potts, Zou, Carbin, Frankle, Rao and Ghodsi. "The Shift from Models to Compound AI Systems." Berkeley Artificial Intelligence Research Blog, 18 February 2024.
2. Yao, Zhao, Yu, Du, Shafran, Narasimhan and Cao. "ReAct: Synergizing Reasoning and Acting in Language Models." International Conference on Learning Representations, 2023. arXiv:2210.03629.
3. Yang, Jimenez, Wettig, Lieret, Yao, Narasimhan and Press. "SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering." NeurIPS, 2024. arXiv:2405.15793. Figures used here: 12.47% resolve rate with GPT-4 Turbo; 10.7 percentage points over the shell-only agent on SWE-bench Lite; 64% relative increase versus that shell-only setting; 10.5% with Claude 3 Opus.
4. Kapoor, Stroebl, Siegel, Nadgir and Narayanan. "AI Agents That Matter." arXiv:2407.01502, 2024.
5. Schick, Dwivedi-Yu, Dessì, Raileanu, Lomeli, Hambro, Zettlemoyer, Cancedda and Scialom. "Toolformer: Language Models Can Teach Themselves to Use Tools." NeurIPS, 2023. arXiv:2302.04761.