Compaction thresholds come from the input limit, not the context window

As a conversation grows, its requests eventually reach the input limit the model will accept. Before that happens, older turns have to be summarized or trimmed; this is called compaction. Compaction starts when the input crosses a threshold. If you set that threshold as a fixed ratio of the context window, it can land above the input limit. Then compaction never starts, and the conversation ends with a request rejected with a 400. The context gauge next to the chat input fills to 100% and nothing happens.

This post covers how we set the threshold from the input limit instead: how we derive the input limit, how much room we leave between the threshold and the limit, what OwlAI trims first within that room when it compacts on its own, and how the context gauge reads the same numbers as the compactor.

#The context window includes the output share

The context window is the combined size of one request's input and output. The input can use the window minus the output share, and that is the input limit. A threshold set as a ratio of the window misses this difference.

Here are the boundaries we measured directly on a model declared with a 400,000-token window and a 128,000-token maximum output (measured in August 2026).

The threshold in the last row sits 43,000 tokens above the input limit. As the conversation grows, the input crosses the limit before it ever reaches the threshold, so under this formula compaction has nowhere to happen.

It is tempting to take the output share from the max_output_tokens value sent with each request: ask for less output, and it seems you could send more input. On the same model, we lowered max_output_tokens to 256 and measured again. 260,024 tokens still passed and 300,000 were still rejected. The limit does not move with the requested value, so the output share is the maximum output the model declares.

#The threshold is the input limit minus one round's worth

OpenAI and Anthropic offer server-side compaction. When a request carries a threshold, the provider's server summarizes the earlier part of any request whose input crosses it. On this path the threshold takes three lines.

• Input limit = context window − maximum output the model declares
• Headroom = max(input limit × 10%, size cap for a single tool result)
• Threshold = input limit − headroom

Server-side compaction happens only on a request whose input falls in the [threshold, input limit] interval. To compact, the server's model has to read the original text once, so a request already over the limit can't be rescued by compaction. If one round (one model call plus running the tool calls in its response) adds more than the headroom, the input jumps over the interval and goes straight past the limit. A single tool result has its own size cap (50,000 tokens by default; see “Shrink giant tool results before the provider cuts them, and say so”), so headroom at least that wide guarantees that any one result under the cap is caught inside the interval. Models with large context windows can receive several large results in one round through parallel tool calls, so we let the 10% term grow larger there.

For the model above, the input limit of 272,000 minus 50,000 gives a threshold of 222,000. A model with a 1,050,000-token window and a 128,000-token maximum output has an input limit of 922,000; 10% of that, 92,200, is larger than 50,000, so the threshold is 829,800.

The same token count goes into compact_threshold under context_management for OpenAI and into the trigger value under context_management.edits for Anthropic. When compaction happens, OwlAI stores the compaction block the server returns as a compaction node (a new parentless node holding the compacted context) and assembles the conversation context from that node from the next round on (“Keeping old answers out of edited questions with a conversation tree”).

#OwlAI's own compaction measures before sending

For providers without server-side compaction, such as vLLM, OwlAI compacts on its own. It takes the spec.contextWindow set per model in the on-premises configuration (vLLM's maximum length), subtracts the output headroom outputHeadroomTokens (8,192 by default), and multiplies by the threshold ratio thresholdRatio (0.85 by default). For a model with a contextWindow of 131,072, the input limit is 122,880 tokens and the threshold is 104,448 tokens.

This path measures at a different point from server-side compaction. Before sending each call, OwlAI counts the tokens of the input, including the tool results this round will add, and if the count crosses the threshold, it shrinks the input before sending. The methods run cheapest first; after each step OwlAI measures again and stops once the input is under the threshold.

Step A comes before summarization because in business queries the biggest share of the input is database query results, not the conversation. Step A turns old query results into a skeleton that keeps only the column list and row count, and because it calls no model, the same input always gives the same result. The summarization call in step B reads only the earlier part and the summary instruction. Compaction is most needed when the whole conversation is close to the limit, so asking the model to reread all of it would make the summarization call overflow first. Step C is the last step, used when the input still overflows after summarizing or when summarization fails.

#The summary's output limit includes a reasoning share

The summary body is set at about 10,000 tokens. After compaction this summary is the only context left from the earlier conversation, so whatever is trimmed for brevity is gone. The output limit for the summarization call is that budget plus 6,000 tokens for reasoning, 16,000 tokens in all.

The reasoning share is added because the output limit counts reasoning tokens and body text together. Set the output limit to the body length and reasoning uses up the limit first, so the body comes back cut off or empty. When we had a self-hosted model summarize a 66,000-token conversation with an output limit of 2,000, 4 of 5 summaries came back empty and the remaining one was cut off. An empty summary skips step B and falls to step C, which drops old turns outright.

OwlAI's own compaction also stores its result as a compaction node. Without storing, compacting again on every call would produce a slightly different summary each time, the front of the request would change every round, and the cache would never be reused. Pinning the result as a compaction node means the cache is written once right after compaction and hits again from then on (“One system prompt edit zeroes cache reads, so changing blocks go last”).

#The compactor and the context gauge read their numbers from one function

The threshold formula differs between the two paths, so if the UI and the compactor each computed it, the two would split the moment only one side was changed. Computing separately has the upside that neither side needs to know about the other, but every formula change would have to land in two places. So there is a single function that returns the context budget, and both the compactor and the UI read it. It returns three numbers per model, window, input limit and threshold, and "threshold ≤ input limit ≤ window" always holds. The function picks the path from the provider type, and only the three numbers go out through the API.

The context gauge is drawn from these numbers.

• Numerator: the input tokens this conversation actually read on its last call (including cache reads and writes; see “A turn's input tokens grow with the square of its rounds”)
• Denominator: the threshold of the model that will run the next turn
• Display: hidden below 50%, caution color from 75%, warning color from 90%

The numerator uses the last call's input rather than the turn's total, because a turn resends the conversation context every round, so the total for a turn that used five tools is several times the real context. The input of the summarization call is left out too, since it is not the conversation context. The denominator comes from the next turn's model, so switching models shows the same conversation at a different ratio. That is correct, because the point at which the next turn compacts really does change.

The window and the input limit arrive as data but are not shown. On the model above, putting the threshold next to the window reads 222,000/400,000, or 56%, and users would ask why compaction happens just past halfway. The only maximum that means something to users is the threshold, so the popover shows only Current and Max (the threshold). For a model that is not subject to compaction, the function returns nothing and the gauge is hidden. Drawing an estimated threshold that does not exist would leave users waiting for a compaction that never comes.

#Design choices

The gauge's numerator uses the measured value the provider returns instead of an estimate. The cost is that the gauge runs one turn behind. Once the next question and its tool results are added, the real input is larger, so if a large query lands while the gauge sits at 90%, compaction happens within that turn. We chose a gauge that lags by a turn over one that shows a wrong number.

When a model spec does not declare its window or maximum output, conservative defaults apply. The server-side path assumes a 400,000-token window and a 128,000-token output share; a vLLM model without a contextWindow is treated as 131,072 tokens. A threshold that errs low only makes compaction happen a little early, while one that errs high climbs above the limit and stops compaction again. So the defaults keep the window small and the output share large, which makes the threshold err low.

Server-side compaction is always on, with no setting to turn it off. A setting would let you turn off this one feature on an account that has trouble with it, but in an on-premises product every setting multiplies the combinations to validate. The feature does have to be enabled on the provider account, so on the account you deploy with, it is worth confirming once with a long conversation that compaction actually happens.

If you attach a vLLM model, setting contextWindow to the serving engine's actual maximum length is what keeps the gauge and the compaction point aligned. Set it larger than the real value and the threshold climbs above the limit, producing exactly the symptom at the top of this post.