A scheduler that plans around AI agents' tool calls cuts serving delays

AI agents do not spend all their time thinking: between bursts of work by the language model, they wait for the external tools it has called to respond. Researchers affiliated with AIOR argue that the systems which run such agents should plan around that waiting. In a preprint posted to arXiv, which has not yet been peer reviewed, they describe TAO, a scheduling framework that decides which agent sessions get time on a graphics processor, and on which one, partly by anticipating when each session's tool results are about to come back. Across the test settings they report, the authors say TAO cut latency by up to 26.8% relative to the strongest existing system they evaluated.

An agentic service is one in which a large language model does not simply answer a prompt once but works through a task in steps, calling on external tools and folding their outputs back into its work. According to the abstract, such services alternate between model inference on clusters of GPUs, the specialised chips that run the model, and tools running on separate, distributed CPUs, the general-purpose processors found in ordinary servers. The authors identify two weaknesses in existing serving systems. First, they ignore whether a session's tool results are ready when judging which work to serve, which wastes limited resources. Second, they rely on rules of thumb for admission, meaning which waiting sessions are allowed onto the GPUs, and for placement, meaning which GPU each one lands on.

TAO replaces those rules of thumb with a scoring scheme the authors call a value interface, which weighs three factors. The first is near-term tool readiness: whether a session's tool call is about to return, so that the session will shortly have model work to do. The second is a cache credit specific to each GPU node. Language-model serving systems typically keep intermediate computations from a session's earlier steps on the machine that produced them, so returning a session to that machine can spare it from redoing work; in TAO this credit fades over time, a feature described as time-aware decay. The third is token footprint, the amount of text, counted in the word fragments known as tokens, that a session brings with it.

Those scores feed a dynamic program, a classic optimisation technique that solves a large problem by building up answers to smaller, overlapping ones. It decides jointly which waiting sessions to admit and where to place them, while respecting the capacity limits of the hardware. Making both choices together, rather than as separate rule-of-thumb steps, is what the title's ‘joint scheduling’ refers to.

The authors tested TAO in 12 settings that varied the model, the workload and the concurrency, or number of sessions running at once. Measured against the best-performing state-of-the-art baseline for each metric, they report that TAO reduced latency by up to 26.8%, reduced mean time to first token, the wait before a model starts producing its reply, by up to 36.3%, and raised decode throughput, the rate at which output tokens are generated, by up to 29.5%.

These are best-case figures across the reported settings, and the abstract does not say how large the gains were in a typical setting or which baseline systems were compared. All results are reported by the authors and have yet to face peer review. Whether similar gains hold beyond the tested models, workloads and levels of concurrency is a question the abstract leaves open.