As AI evolves from responding to prompts to completing tasks, the focus of infrastructure optimization has expanded from the models themselves to the entire workflow, encompassing planning, tool invocation, result verification, and task execution.
At 8:42 a.m., a developer opens her laptop to find a failed build, test errors, and a code branch waiting to be deployed. In the past, she might have had to spend a significant amount of time sifting through logs one by one, comparing code commit records, and tracing the repository history; but now, all she needs to do is issue a simple command to the AI coding agent:
"Fix the failed tests. Find the bugs, submit a patch, rerun the tests, and summarize the changes.”
With just a few keystrokes—the time it takes to take a sip of tea—the agent returns a patch that passes the tests, accompanied by a concise explanation. For the user, the effect is like instant magic: input a request, get a result. But behind the scenes, the process of achieving this result is far from simple.
A single prompt can trigger request parsing, task creation, policy checks, repository searches, context concatenation, token encoding, model invocation, tool verification, sandbox routing, file editing, test execution, telemetry collection, and final result verification. While model invocation is certainly critical, it is only one part of the workload; what truly demonstrates engineering capability is the coordination and orchestration of the task graph.
The AI agent performs planning, retrieval, execution, verification, retries, and feedback. As AI shifts from isolated prompt-response interactions to continuous workflow modes, performance metrics have evolved from token-level metrics to the delivery of complete tasks.
Model invocation is no longer synonymous with the entire workload
For a long time, AI infrastructure has been primarily evaluated based on model execution metrics: prefill, decode, tokens per second, first-token latency, throughput, batch efficiency, memory usage, and accelerator utilization. While these metrics remain important, with the rise of agent-based AI, the system’s critical path is no longer limited to model execution alone.
Traditional inference requests have clear boundaries: from the prompt to the output result. Agent workflows, however, may involve task planning, context retrieval, state management, tool and API calls, sandbox execution, result validation, telemetry collection, and retries. The final output could be an answer, a code change, a database query, a tool invocation, or a decision to gather more information.
In short, agent-based AI has transformed inference into a distributed systems problem. GPUs remain indispensable for intensive model execution, but the work surrounding GPUs is increasingly becoming the decisive factor in system performance: which tasks are running, where they are running, what context they operate in, which tools are permitted, and whether the results meet user needs.
This surrounding technology stack adds multiple layers to the model, with much of the coordination work handled by the CPU—including orchestration, memory and retrieval, tool invocation, runtimes and sandboxes, policy systems, and observability. The model provides intelligence, reasoning, and generation capabilities, while the peripheral systems translate these capabilities into actionable steps.
Why Traditional Inference Benchmarks Can No Longer Comprehensively Measure System Performance
For chatbots, token throughput reflects a model’s response speed and generation efficiency, as well as the number of users the system can serve. However, for agent-based AI, this evaluation metric is not comprehensive enough.
Developers do not perceive system performance in terms of raw tokens per second. What they truly care about is whether bugs are accurately pinpointed, whether patches are applied correctly, whether all tests pass, whether permissions are compliant, and whether results are reliable and verifiable. The same holds true for agents in enterprise, research, security, operations, and the physical world—users care about the final outcomes.
This means that the era of AI agents requires the adoption of workflow-level performance metrics, including: the cost of completing a single task, tool invocation latency, retrieval latency, sandbox startup time, and the number of agents per node. The core evaluation criterion shifts from “How fast is the model generated?” to “How efficiently does the system complete tasks?”
The CPU Plays a Decisive Role
The value of the CPU in intelligent AI agents is reflected in every stage surrounding model invocation. Before the model runs, the system parses requests, initializes the runtime environment, loads policies, reads project files, queries the retrieval system, organizes the context, counts tokens, and prepares the request. During model invocation, the CPU handles routing, tokenization, batch processing, streaming, and schema configuration. After the invocation is complete, the CPU verifies the output, routes tool invocations, manages sandboxes and subprocesses, collects logs, categorizes failures, updates status, and integrates telemetry data.
In the example of a coding agent, some of the most CPU-intensive tasks often occur after the model generates a fix: running tests, invoking the compiler, launching subprocesses, classifying failures, and autonomously selecting retry logic. These operations place strain on CPU cores, caches, memory, storage, and orchestration software.
This is precisely the critical computational work hidden behind the agent’s AI seemingly “completing tasks in an instant.”
The head node is evolving into the control center of the entire system
As agent systems scale, the role of the head node becomes strategically significant. An AI head node is not merely a CPU sitting next to an accelerator. It is the central control hub responsible for enabling heterogeneous AI systems to operate in a unified and reliable manner: routing requests, managing state, preparing context, coordinating accelerators, invoking management APIs and tools, executing policies, collecting telemetry, and maintaining workflow observability.
The essence of orchestration lies in determining which tasks to execute, where to execute them, what context to provide, which tools to invoke, and how to determine when a task is complete. The head node serves as the infrastructure control center that underpins this capability, responsible for coordinating resources across CPUs, GPUs, accelerators, memory, storage, networking, runtimes, databases, rule management systems, and observability tools, and for ensuring these orchestration decisions are enforced throughout the entire system.
As AI infrastructure becomes increasingly heterogeneous, models, tools, memory, accelerators, APIs, sandboxes, policy management, tracing, and evaluation loops must all remain synchronized and collaborate seamlessly. The head node serves as the control center that unifies the execution and operation of this coordination capability.
Opportunities for Arm: End-to-End Agent Workflow Optimization
For Arm, agent AI represents a platform-level opportunity. Its infrastructure requirements align with the key capabilities that matter most to cloud infrastructure builders: performance, energy efficiency, scalability, software maturity, and freedom of choice. Industry evaluation criteria are no longer limited to token generation speed; instead, they focus on whether a platform can efficiently complete more workflows under real-world constraints such as latency, power consumption, cost, resource utilization, stability, and development iteration efficiency.
Arm provides a computing platform designed for heterogeneous infrastructure. In the era of agents, AI systems consist of a collaborative ecosystem of CPUs, GPUs, accelerators, memory, storage, networking, runtimes, APIs, tools, observability systems, and rule-based control. Arm’s core value lies in helping the entire system achieve greater efficiency and scalability, while making system-level optimization easier.
This value stems from Arm’s deep technical foundation: a computing platform built on flexible Arm Neoverse IP and the Computing Subsystem (CSS), along with a software support ecosystem and partner network, which provides AI infrastructure builders with diverse paths to design and deploy infrastructure tailored to specific workloads. Agent-based AI makes this flexibility of choice even more critical, as no single computing configuration can efficiently handle every stage of a workflow.
This is why Arm’s value should be measured at the workflow level. Superior agent-based AI infrastructure should be reflected in the following outcomes: lower per-task costs, lower latency, higher resource utilization, and more predictable execution results. These key metrics directly reflect the value of the CPU and head node, closely linking them to business outcomes.
Arm extends this philosophy by introducing the Arm AGI CPU—a proprietary chip designed for AGI infrastructure—offering an alternative option for deploying Arm architecture computing power. In this solution, the CPU coordinates model-centric tasks: control, memory, retrieval, tools, APIs, runtimes, sandboxes, and observability.
The goal is not to diminish the role of GPUs or other accelerators, but to enable the entire heterogeneous system to work together more effectively. In the era of AGI, the infrastructure that truly gains the upper hand will be the one capable of optimizing the entire task graph—not just the token stream.
The Next-Generation Benchmark: Task Completion at the Core
Returning to the developer mentioned earlier who was waiting for the results of a fix, what she needed was not a token, but a fix for the bug. This is the infrastructure transformation brought about by agent-based AI. In the era of traditional prompt-driven question-answering, model performance was clearly the primary focus. In the era of agents, systems must optimize the entire closed-loop chain: model intelligence, memory, tools, state management, rule management, execution, verification, and observability.
The criteria for evaluating next-generation AI infrastructure will be its ability to complete tasks quickly, accurately, securely, observably, and efficiently. Against this backdrop, the Arm AGI CPU offers AI infrastructure builders a new option for deploying Arm-architecture computing power to handle critical workloads such as orchestration and task execution, helping to transform model outputs into truly valuable actions.
English
