Measure the vLLM concurrency limit with production input lengths, not short prompts

Benchmark a vLLM server for a self-hosted model with 1.1K-token prompts, and the numbers tell you to raise the number of requests processed at once (max_num_seqs) from 4 to 32, because total throughput rises 92%. Measure the same server at the input lengths of real business queries, and the answer flips. Going from 4 to just 8 adds only 4% throughput, while time to first token (TTFT) rises 65% and the speed at which each response produces tokens drops 56%.

This post explains why input length flips the result, how to choose benchmark inputs, and the other values that follow from a concurrency limit of 4: weight precision, the number of input tokens processed per step, a server-side output limit, and how many requests OwlAI sends to the server at once. All numbers come from a reference configuration: an Int4 (GPTQ) quantization of a 35B-class MoE model (3B active parameters per token) on two RTX 4090s (48 GB total) with tensor parallelism of 2, measured on vLLM 0.26.0.

#With long inputs, throughput stops growing at concurrency 2 to 4

Each request runs in two phases. Prefill computes keys and values for every input token and fills the KV cache. Decode then produces output tokens one at a time. A single decode step reads the entire set of weights from GPU memory to produce one token, so memory bandwidth runs out before compute does. Batching several requests into one step produces several tokens from that same read. That is why, under a load of short inputs where decode dominates, throughput keeps rising with concurrency.

Long inputs change this. In the reference configuration, prefill is already compute-bound at about 11,400 tokens per second. A single 40,000-token input keeps the GPU computing flat out for more than three seconds, so accepting more requests leaves no spare capacity to share that work. The extra requests wait for earlier prefills to finish, and TTFT grows. Responses that are already streaming slow down too, because they share the GPU with other requests' prefill.

Business queries have long inputs, because table schemas, document excerpts, tool results, and conversation context pile up in one request. Observed in production, the median input length was 39,812 tokens, and 61.7% of inputs exceeded 32K. At these lengths, total throughput flattens at concurrency 2 to 4, so we set the limit at 4. Raising it to 8 buys 4% throughput and costs every user extra waiting before the first character appears.

#Take benchmark input lengths from production records

A benchmark matches production only if its conditions resemble the production load. The measurements behind the reference configuration kept to these conditions.

• Set input length from production records. OwlAI records the input token count of every LLM call in its usage ledger, so look at its percentiles first and size the benchmark prompts to match.
• Give each request a different beginning. If you send the same prompt concurrently to a server with prefix caching on, the requests share the KV cache, throughput is inflated, and a 31K input reports a TTFT of 0.13 seconds.
• Measure the same condition several times. Even at temperature 0, results shift with batch composition, so a single run is not enough to draw a conclusion.
• Leave max_num_seqs wide open and vary concurrency on the client side. The value is only a scheduler limit, so you get the same curve as restarting the server for each value, and you skip a restart that takes three to four minutes each time.
• Check whether concurrent requests × average context exceeds the number of tokens the KV cache can hold. When it does, vLLM starts preemption: it sets running requests aside and recomputes them later.

#To hold four requests at full window, KV cache room comes before precision

Once the limit is 4, the GPU memory split follows. For four requests to fit without preemption even when each uses nearly the full 262,144-token context window, the KV cache must hold at least four windows.

The model's official quantization is FP8, and on precision alone it beats Int4. But the FP8 weights take 34.9 GiB, which leaves only about 5 GB of KV cache per card on these GPUs. The Int4 weights take 22.7 GiB and leave about 10 GiB per card. The Int4 configuration's KV cache holds 7.58 full-window requests; FP8 leaves about half that KV cache, which by rough arithmetic falls short of four windows. So on these GPUs we chose Int4. With enough GPU memory to load the FP8 weights and still hold four windows, choose FP8.

The number of tokens prefilled per step (max_num_batched_tokens) stays at 8192. Raising it to 32768 looks like it should allow bigger prefill batches, but throughput barely moved, from 82 to 84 tokens per second, and the KV cache shrank 17%, from 9.7 GiB to 8.1 GiB. Prefill already uses all the compute, so bigger batches don't run faster, and the working memory they need comes out of the KV cache.

#A server-side output limit ends repetition that ties up one of four slots

With a concurrency limit of 4, the server also has four slots for generating responses at once. If one of them is tied up for a long time, a quarter of the server's capacity is gone for that whole time.

Occasionally a model repeats the same phrase and does not stop. If the model's generation config file has no output limit, vLLM's effective limit is "window minus prompt," and a request stuck repeating generates to the end of the 260K-token window, holding one slot for tens of minutes.

So we put two server defaults into vLLM's override_generation_config.

32768 is the output length the model card recommends for most queries, and 1.5 is the value it recommends for general tasks in thinking mode. Depending on the vLLM version, the server may silently drop presence_penalty as a default. Of the versions we tested, 0.29 dropped it and 0.32 onward accepted it. After upgrading, check that the startup log has the "Default vLLM sampling parameters have been overridden" line and that both values appear in it. A response cut off at the output limit comes back without an error, so OwlAI logs these responses separately as a WARNING (“On-premises OwlAI is observed through one JSON log, joined to the worker by a request ID”).

#OwlAI sends only as many requests as the server has slots

Requests beyond the server limit just queue in front of the server, and other users' chat turns wait in the same queue. So wherever OwlAI sends several requests at once, it checks whether the target is a self-hosted model.

The step that polishes deep research reports is one such place. It splits the report's paragraphs and list items into blocks and calls the model once per block. On a cloud model it calls 12 blocks at once; on a self-hosted model, only 4. Block calls on a cloud model are short, 2 seconds on average, so more concurrency shortens the wait just before the report appears. In our measurements, the wait for the slowest block was 18.2 seconds at concurrency 4 and 6.5 seconds at 12, and barely dropped further at 16, to 5.7 seconds. A self-hosted model has four server slots, so sending more just holds up other users' requests. Both concurrency values are configurable, so if you change the server's max_num_seqs, change the self-hosted value to match.

#Design choices

We chose time to first token over throughput. Raising the limit to 8 processes 4% more on the same GPUs. In exchange, every request's TTFT rises 65% and each response's token speed falls below half. People wait in front of the screen for business answers, so a late start costs more than 4% throughput. If users grow enough to lengthen the queue, add GPUs instead of raising the limit.

We chose KV cache room over one step of precision. The official FP8 is closer to the original model than Int4. But choosing FP8 on 48 GB means four long requests no longer fit together, preemption follows, and under a load dominated by long inputs that loss outweighs the precision difference.

We put the output limit on the server, globally, instead of leaving it to each request. The cost is that legitimate outputs longer than 32,768 tokens are cut too. Relying only on the value the client sends means a single call that omits a limit can hold a slot for tens of minutes. With the limit on the server, every call ends within the same bound.

We chose startup stability over the generation speed speculative decoding (MTP) would add. On vLLM 0.26, enabling it with a quantized MoE model kept the server from starting.

If you serve long-input business queries on one or two consumer GPUs, these values are a reasonable starting point. If your traffic is mostly short chat, decode dominates the load, so re-measure the concurrency curve first at the input lengths in your own production records.