Shrink giant tool results before the provider cuts them, and say so

Sometimes a single tool result blows past the model's input limit in one go: a query that returns hundreds of thousands of rows, or an MCP server that sends back an enormous JSON blob. Send that request as is, and the provider either rejects it with a 400 or cuts the input and returns a 200. The cut is the more dangerous outcome. Nothing in the response says anything was dropped, so the model believes it received the whole result and writes numbers for rows it never saw. Compaction, which summarizes the front of a long conversation (“Compaction thresholds come from the input limit, not the context window”), can't help here either.

So OwlAI caps the size of tool results before they reach the provider. A result over the cap is shrunk according to its shape, and the result carries a note that it was shrunk and what the model should do next. This post covers where the cap sits, what gets shrunk first, and what the model is told.

#Automatic truncation returns a 200 with no sign of the cut

The OpenAI Responses API has a truncation parameter. Set to auto, it drops items from the beginning of the conversation when the input exceeds the context window; set to disabled, it rejects the request with a 400 (official reference, checked 2026-11-03). We sent the same request under both settings to a model with a 272,000-token input limit, carrying a tool result of about 1.29 million tokens (measured August 2026).

The appeal of auto is that the request never fails, so the user always gets an answer. The catch is that no one knows what that answer was based on. A 400 is something the system can catch and report to the user. An answer written from a truncated input reaches the screen looking like a normal answer. So OwlAI pins truncation: "disabled" on OpenAI requests. Anthropic rejects an oversized request with a 400 from the start, and vLLM rejects it with an error saying the maximum length was exceeded. An overflow 400 is classified as a context overflow and reported to the user without a retry.

#Compaction can't shrink a request that has already overflowed

The compaction parameter made no difference because of the order compaction works in. To compact text, the model has to read it once, and text over the limit leaves no room to read it. Server-side or OwlAI's own, compaction shrinks the next request. Getting the request about to be sent under the limit has to happen separately, before compaction.

That is the job of the result size cap. A single tool result is capped at 50,000 tokens by default, adjustable with the install setting compaction.singleToolResultMaxTokens. In one round (one model call plus running the tool calls in its response), tools called in parallel can each return results under the cap while their total overflows. So the total of the results newly added in that round gets the same cap, and when it overflows, the largest results are shrunk first. That order saves the same number of tokens while touching the fewest results.

The total cap equals the single-result cap for a reason. The headroom between the compaction threshold and the input limit is set to the larger of 10% of the input limit and the single-result cap (“Compaction thresholds come from the input limit, not the context window”). On any model, that headroom is never narrower than 50,000 tokens, so keeping what one round adds under 50,000 tokens means the input lands inside the headroom and gets compacted instead of jumping past it.

#A reducer that knows the result's shape shrinks it first

A result over the cap goes to a reducer. A reducer is a function that knows the shape of a given tool's results and picks what to trim. It first shrinks the result to preview level, and if that is still over the cap, to skeleton level.

When a SQL result is shrunk to preview, the truncation flag the result already carried is left alone. Cutting at 80 rows is a display trim, a different fact from the database returning fewer rows because it hit a limit (“RAG can't count or sum rows, so we ask the source database in SQL”). So a separate marker says "kept 80 rows, dropped this many." Mixing the two facts in one flag would make the log disagree with what the model actually saw when you trace a wrong answer back.

Tools without their own reducer go to a generic reducer that works from the shape alone. Every customer connects different MCP servers, so per-tool reducers can't be written in advance. At each cut, the generic reducer records the original size: the total item count and how many were kept for a list, the original character count for a string. Emptying a result entirely because its shape is unfamiliar leaves the model not even knowing what it queried, and it calls the same query again.

#Check that the result actually shrank before calling it shrunk

A reducer that meets an unfamiliar shape may not shrink the result at all. Results exchanged with the client, such as tools on the user's PC, are one example: the reducer found nothing to trim and only added a marker saying it had emptied the result, so a 142,924-character result became 143,589 characters. Left that way, the record says the result was shrunk while the model receives something larger than the original.

So the shrunk result is checked twice. First, is its character count smaller than the original? If so, is its token count under the cap? If either check fails, only short values of 200 characters or fewer (the tool name, a success flag and the like) and the original size are kept, and the body is removed entirely. Whatever shape the body has, this step guarantees the result gets smaller. Cutting a string in the middle is not an option. JSON broken mid-way can't be read by the model, and the model can't tell it was cut either.

Token counting is kept cheap too. Every token is at least one character, so if a result's character count is under the cap, its token count is too, and counting is skipped. Most results fall into this case, so the round trip to the provider's token counting API (measured at 284–479 ms) is paid only for large results near the cap.

#A shrunk result carries the fact and the next step

The original problem was data disappearing silently, so a fix that is also silent fixes nothing. A shrunk result carries a marker saying so, and inside it the original token count, the cap, the reduction level, the reason (one large result or a large round total), and guidance for the model.

Tell the model only "this was truncated," and it makes up an answer from the part it received. So the guidance includes the next action. Three things appear in every variant: the model did not see the full result, it must not report the received row count as the total, and it must say in its answer that part is missing. On top of that comes one next action suited to the situation.

The variants exist because a model given the wrong next step tries it and fails. Tell a playbook step with no file-reading tool to "read the file," and the model calls a tool that isn't there. Tell it to "query again" when the original is already on disk, and it runs the same query a second time.

The original is on disk thanks to the spill. Any tool result over 1,000 characters is first written by the server to a sandbox file right after the tool runs (“Query results go to a file so the model never retypes them”), so a result that hits the 50,000-token cap already has its original in a file before it is shrunk. The guidance includes that file's path and the row and byte counts the server actually wrote. When the model answers "N records in total," it uses those numbers.

#The cap is applied before the result is saved to the conversation tree

OwlAI saves each model call as one node and assembles the next request from that conversation tree (“Keeping old answers out of edited questions with a conversation tree”). The result cap is applied before newly added tool results are saved to a node. The shrunk result is what the node keeps, so the next round and the next turn assemble their requests from it too. Shrinking only the copy about to be sent would make this one request smaller, while the original stayed in the tree and overflowed the cap again every round.

Shrinking before saving also suits prompt caching. Each result is judged once, when it is first added, and its bytes never change after that. The front of the next round's request matches the previous round's, so the cache keeps getting read (“One system prompt edit zeroes cache reads, so changing blocks go last”). The cap sits at the entry point every model call passes through, so chat, playbooks and deep research follow the same rule. On the vLLM path, where OwlAI compacts on its own, the first compaction step uses the same cap (“Compaction thresholds come from the input limit, not the context window”).

#Design choices

The price of truncation: "disabled" is that an overflowing request ends in an error instead of an answer. We gave up auto, which always returns an answer, because a failure you can detect is easier to handle than a normal-looking answer with nothing behind it. Keeping that error from reaching users is the job of the result cap and compaction.

If token counting fails, the result is sent unshrunk. Treating "couldn't count" as "over the cap" would turn a fault in the counter into the loss of a perfectly good result. If the request really did overflow, the provider says so with a 400. In that case the conversation's end node is kept at the last successful node rather than the failed one. The failed node may hold the giant result that caused the overflow, and pinning it as the end would assemble the same input on every later turn, so the conversation could never move forward again. This costs that round's tool results, but the conversation goes on.

The reason is logged as either "single result exceeded" or "round total exceeded" because the remedy differs. One large result means that tool's query needs narrowing; a large total means fewer parallel calls in one round. A tool that often shows up in the logs as shrunk by the generic reducer is a candidate for a dedicated reducer.

If your agent's tool results can grow to hundreds of thousands of rows or millions of characters, check two things first: that automatic truncation is not on at the provider or gateway, and that results are shrunk before they are saved to the conversation record. If the first is on, the model answers from data it never saw. If the second happens after saving, the shrunk result comes back as the original in the next round.