A turn's input tokens grow with the square of its rounds

A model API remembers nothing between calls. Every time an agent calls the model, it sends the system prompt, the tool definitions, and the whole conversation context so far. Call a turn everything from one question to its answer, and a round one model call plus running the tool calls in its response. A turn usually takes several rounds, and the tenth round's request carries every tool call and result from the nine before it. If each round adds a similar amount, the total input the model reads in one turn grows with the square of the number of rounds.

This post covers where that growth comes from, and what OwlAI reduces and measures because of it. We get the same work done in fewer rounds, shrink large results that would otherwise be reread many times, and count cost from the full request each call actually sent. “OpenAI stores responses by default, so we resend the conversation” explains why we send the conversation ourselves every time instead of leaving its state with the provider, and “Keeping old answers out of edited questions with a conversation tree” covers the tree that stores the conversation context.

#The request for round N is the request for round N-1 plus two things

In chat, each round's LLM call is stored as one node. A node holds what that call newly sent together with what the model returned. To send the next round, OwlAI starts at the node at the end of the conversation, follows parent links back to the start, concatenates the items along the path in time order, and appends this call's new items. Work without a conversation, such as playbooks, table analysis and meeting summaries, keeps a message list in memory and does the same thing. Only the storage differs. Either way, every round resends everything before it.

So the request for round N is the request for round N-1 with two things appended: the tool calls the model returned in round N-1, and the results of running those tools. The system prompt and tool definitions go out again every round.

To get a sense of scale, take an assumption. Suppose the system prompt and tool definitions come to 20,000 tokens and each round adds 5,000 tokens of tool results. The tool-call items themselves are small, so leave them out. The first round's input is 20,000 tokens; the tenth round's is 65,000. Across ten rounds the inputs add up to 425,000 tokens. The distinct content in the requests is only 65,000 tokens, yet the model reads 6.5 times that.

With a fixed part F, a per-round result D and R rounds, a turn's total input is R×F + D×R(R-1)/2. The second term grows with R squared, so the more rounds there are, the larger the ratio of what the model reads to what the requests contain.

Fourteen rounds is the p95 for chat turns observed in production (“When every tool call succeeds but the results repeat, stop the agent”). One turn in twenty goes deeper than that.

#Finishing the same work in fewer rounds cuts what gets reread

The most direct way to shrink R is to put tool calls that don't depend on each other into one round. On every call that has tools, OwlAI lets the model call several tools in one response (parallel tool calls), and runs the calls from one round concurrently.

Consider a question that needs information from three tables, each looked up separately. Using the same assumption, the fixed part is 20,000 tokens and each lookup returns 5,000.

The lookup results are the same, and so is what the model sees when it writes the final answer, but the total input is half. Two fewer round trips also mean less waiting. The model decides which calls to group, because calls that need an order cannot be grouped, as when the next SQL depends on the previous lookup's result.

The round limit also counts LLM round trips, not tool calls. Ten tools called in one round count once, and the full request is resent once. “When every tool call succeeds but the results repeat, stop the agent” covers the limit's value and its basis.

#The earlier a large result lands, the more times it is reread

D is not the same every round. In real turns a few results are usually far larger than the rest, such as a query that scans a whole table or a single long document. Once such a result is in, it goes out again in every round until the turn ends.

Take a 50,000-token query result in a 10-round turn. If it comes from a tool called in the first round, it is sent in rounds 2 through 10, nine times, for 450,000 tokens. If the same result comes from a tool called in round 9, it is sent once, in round 10. The cost of one result is its size times the number of rounds left after it lands.

So large results are reduced before they go in. When a single tool result exceeds the size cap, it is reduced and the model is told that it was. When several tools run in one round, the same cap applies to the total of the results that round adds (“Shrink giant tool results before the provider cuts them, and say so”). When a long conversation is compacted, old tool results are trimmed first (“Compaction thresholds come from the input limit, not the context window”). Both aim at the same thing: large chunks that would otherwise be resent in every remaining round.

#The reread part matches the previous request, so it can come from the cache

The request for round N is the request for round N-1 with items appended, so its front matches byte for byte. Provider prompt caching and vLLM prefix caching read a matching front from the cache instead of recomputing it. Cached tokens skip input processing (prefill), and providers bill them below the list price. On a self-hosted model, GPU compute time goes down.

In the 10-round example, 360,000 of the 425,000 tokens overlap with the previous round's request. If the cache holds every round, only 65,000 tokens are computed fresh, the same as the distinct content. Appending to the conversation context instead of editing it is the condition that makes this arithmetic work. “One system prompt edit zeroes cache reads, so changing blocks go last” covers how changing one line in the system prompt forces everything after it to be recomputed, and how block ordering avoids that.

The cache reduces compute and price, though, not the amount read. Requests approach the model's input limit just as fast. So cutting rounds and shrinking large results still matter with a cache.

#Cost is measured from the full request each call sent

What you count decides whether you see this growth. If you count only the items added this round, each round shows a similar number and the buildup of the earlier conversation disappears. So for every call OwlAI records the character count of the full request it actually sent, including the system prompt, the tool definitions and the input items. Each round also logs input tokens, tokens read from the cache, and tokens newly written to the cache.

When you add up input tokens, the numbers mean different things per provider. In OpenAI's usage, input tokens are the total input and cached tokens are a subset of it. In Anthropic's, input tokens are only the part not served by the cache, and cache reads and cache writes are separate numbers. Add up Anthropic's input tokens alone and the turns that used the cache best look the smallest. So the usage ledger and the round log apply one rule to bring both formats to "total input including cache reads and writes" before counting.

Calls the user stops mid-answer are counted too. The provider bills a stopped call for the input it received and the output it generated. But OpenAI's streaming responses report usage only in the final event, so a call cut off partway has no usage. In that case the ledger estimates input from the accumulation itself. This call's input is the previous call's input and output plus the items newly appended this time, and since the front matches the previous request, that part is treated as read from the cache. If the estimate exceeds the count taken from the characters actually sent, it is capped at that count. Characters convert at three per token. In real conversations mixing Korean text and tool JSON, OpenAI used 2.2 to 4.2 characters per token (measured 2026-09-30), and 3 is the middle of that range. Output is counted from the characters received over the stream.

#Design choices

We chose to resend the full request every round. That keeps all conversation state in OwlAI's database and lets any provider's model be called the same way (“OpenAI stores responses by default, so we resend the conversation”). The cost is that deep turns grow their input with the square of the round count. Caching, parallel tool calls, the result size cap and compaction reduce that cost; they do not remove it.

Parallel tool calls are allowed, not forced. Whether calls are independent often depends on earlier results, and forcing calls that need an order into one round makes the model invent the next SQL without the result it needed. We leave the number of rounds to the model's judgment and bound the cost of that judgment with reduced results and measurement.

The cost of a stopped call is an estimate. Input is capped at the size of the request actually sent, but output counts only the characters that streamed in, so on OpenAI models, which do not stream reasoning tokens, the reasoning share is missing. The ledger's figure for such a call sits close to a lower bound on the real bill.

The cache only removes the reread cost when its conditions hold. If the user steps away for a few minutes, the cache lifetime runs out, and switching models mid-conversation or running a compaction changes the front so the cache is written again from scratch (“One system prompt edit zeroes cache reads, so changing blocks go last”). If your users pack deep, result-heavy work into one long conversation, check those turns' total input in the per-round log first; splitting long work into new conversations gives more accurate answers and costs less. The customer manual also advises starting a new conversation when one gets very long.