When document search ranks lookalike chunks first, rerank them

Ask "How many days of annual leave can I carry over to next year?" against a company's policy documents with embeddings alone, and the leave request procedure, a guide to leave types, and last year's notice come out on top. All three are about annual leave, so they sit close to the question's vector. The paragraph that states the carryover limit may be around rank 7. The model only receives the top few results, so when the answer falls outside them, the model answers with the number from last year's notice.

OwlAI's document search decides the order in two stages. The first stage gathers candidates broadly with semantic search and keyword search. In the second, a reranker reads the question together with each candidate and sorts them again. If the reranker is slow or fails, search returns the first-stage order right away. This post covers why a reranker can fix the order, where it sits and when it gives up, and how we pick the model.

#Embedding distance measures whether the topic is similar

Embedding search turns the question and each chunk into separate vectors (a bi-encoder). Chunk vectors are computed in advance, when a document is uploaded, so when a chunk is squeezed into one vector, nobody knows what question will come. The topic, annual leave, survives in the vector. Whether the chunk contains a sentence answering "how many days can I carry over" gets blurred. Chunks on the same topic end up packed close together, and their order within that cluster has little to do with which one answers the question.

There are two common responses. One is a better embedding model, and it does help. Swapping only the embedding model, with no reranker, raised the quality of the top five results (nDCG@5) from 0.420 to 0.500 (measurement conditions are in the measurement section below). But the best model differs by language, and a better model still packs same-topic chunks together. The other is to send the model more. Sending the top 20 makes it likelier that the answer gets in, but it costs context in proportion, and other paragraphs on the same topic come along, any of which the model may pick as its source.

So the first stage's only job is to get the answer into the candidate list. A different model decides the order.

#The first stage gathers candidates broadly

Semantic search takes 50 chunks by distance between the chunk vectors stored in pgvector and the question vector. Keyword search takes 50 with PostgreSQL full-text search. The two lists are merged with RRF (reciprocal rank fusion): the two scores are on different scales and can't be added, but ranks can (“Wrong-table SQL still runs, so search narrows the candidates first”). “Korean particles break full-text search, so we join words with OR” covers how keyword search turns a Korean question into a query.

The merged list is cut to the number of results the model asked for in its tool call (8 by default, at most 30). With the reranker on, it keeps the larger of that count and the reranker's candidate count (topKInput). The goal of this list is inclusion, not order. If the carryover paragraph made it in at rank 7, the first stage has done its job.

#The reranker reads the question and the chunk as one input

A reranker reads the question and a chunk together as one input and scores whether this chunk answers this question (a cross-encoder). "How many days" in the question and "up to 10 days may be carried over" in the chunk see each other inside the same input, so the reranker can tell a chunk that only shares the topic from one that holds the answer. It can't be precomputed the way chunk vectors are, though: the model runs once per candidate, so it runs only on the narrowed list.

The reranker block in the configuration file sets up this stage.

Any server that exposes a Cohere-style /rerank endpoint can serve as the reranker. On-premises installations use a self-hosted model served with vLLM, and the SaaS configuration, which has no GPU, uses managed Cohere. Remove the reranker block and search runs in first-stage order. Either way, search returns the number of results the model requested.

The reranker only reads candidates that have already passed the permission check. Search and reranking run only within the document collections the user can see, and the rows that come back are filtered by collection permission once more before they reach the model. Reranking everything and filtering afterward could leave fewer permitted results than the model requested. Organization collections the user can't access are searched separately without reranking, and only the document name and collection name are reported, with no content.

#With the reranker on, the embedding gap almost disappears from the top five

We measured with the whole app running in an isolated environment, through the product's own search path (September 2026). We copied 8,073 document chunks from a production environment and used 150 queries that users actually asked. Judging pooled the top results of several configurations, hid which configuration produced each, and graded them on three levels: 0, 1, and 2. Significance came from a bootstrap paired by query. We didn't write separate benchmark code because a change in any one of keyword search, the RRF weights, the permission filter, or the reranker's input format would make the measurement diverge from the product.

Here is nDCG@5 for two embedding models (the self-hosted Qwen3-Embedding-8B and one cloud-managed model), with the reranker off and on. The reranker is Qwen3-Reranker-0.6B, with 20 candidates.

The difference in embedding quality is real, but with the reranker on it shrinks until it is statistically indistinguishable. The reranker pulls the answer back to the front from within the 20 candidates. That is the basis for giving the first stage inclusion only and leaving order to the reranker.

#A reranker earns its place only when its language and input format match

Not every reranker fixes the order. In an August 2026 measurement using conversation memory chunks and 200 model-generated queries, the general multilingual reranker Qwen3-Reranker-4B was indistinguishable from no reranking on Korean queries by MRR@5 (0.56 vs 0.52, p=0.13). A reranker further trained on Korean reached 0.70, and on English queries the two models swapped places. In the September evaluation, too, one of the managed rerankers was indistinguishable from no reranking on Korean document queries. No model is best in both languages, so the reranker and the embedding model are installation settings. The self-hosted default, Qwen3-Reranker-0.6B, matched the 4B model's quality statistically in the August evaluation and cut latency on Korean requests from 2,605 ms to 447 ms. In the September evaluation, the 0.6B model beat no reranking, 0.57 to 0.50. The conclusion differs from the August 4B result because the evaluation sets differ.

The input format has to match too. Qwen3-Reranker is a language model that answers yes or no, and its score is the ratio of the yes and no probabilities at the last token. That ratio means something only when the input is wrapped in the chat template the model saw in training, and vLLM's /v1/rerank (checked in 0.26.0) doesn't apply it. Without the template, scores bunch up between 0.65 and 0.99. For the question "What database feature stores JSON?", the sentence "The capital of France is Paris" scored 0.95, above the correct sentence at 0.82. With the template, they separated to 0.000003 and 0.89. On the evaluation set above, nDCG@5 rose from 0.34 to 0.57. bge-family rerankers go the other way: wrap their input in this template and the order flips. So the search client applies the template before sending only when the model name belongs to the Qwen3-Reranker family, and logs which format it sent.

#If the reranker is slow or fails, the RRF order goes out unchanged

Without the reranker, results are not wrong. Only the ordering drops to first-stage quality. So timeoutMs is not the time the reranker gets to finish its work; it is the point at which search stops waiting and falls back. Called from Korea, the managed reranker's median latency was around one second, but occasionally a call took 17 to 19 seconds or went unanswered for more than 30. Waiting on those would stall the whole search, so the managed example config cuts off at 5 seconds, about five times the median, and returns the RRF order already computed.

If the vendor hangs, every search would throw away 5 seconds. To prevent that, rerank calls go through a circuit breaker. After three consecutive failures in one process, the breaker opens: requests stop going out and search falls back immediately. After 30 seconds, a single trial call checks for recovery. Timeouts, transport errors, and 5xx count as failures, and so do 4xx such as 401 and 429: if the key is wrong or the quota is hit, sending again gives the same answer. Breakers on other external calls open later, because on those paths it is better to wait a little longer and succeed. Reranking is a stage whose absence still yields a result, so it gives up early.

Each search result records whether reranking ran and, if it fell back, why, and the model receives a fallback warning too. Managed rerankers bill per call, so each successful rerank call is recorded in the usage ledger under the name of the user whose turn it was. A call the open breaker never sent is not billed, so it is not recorded.

#Design choices

The reranker decides only the order; the model decides how many results to take. Whether the reranker is on or off, search returns the count the model requested when it called the search tool. If the model asks for 20 to look more widely, it gets 20. The reranker reads the larger of the requested count and topKInput, so with the default request of 8, the reranker reads 20 and returns 8 of them. Reading more widely than the request is what lets an answer sitting at rank 9 to 20 in RRF order move up to the front. The cost is longer latency as the reranker reads more chunks per call, which is why topKInput is an installation setting.

We made the reranker a stage search can do without. A reranker outage doesn't become a search outage, but while search is falling back, the quality of the top five (nDCG@5) drops without notice, from 0.57 to around 0.50 in the measurement above. That is why the fallback and its reason go into the result and the logs, so searches whose quality dropped can be found later.

The input format is chosen by model name. That holds for the Qwen3 and bge families we checked, but before connecting a reranker from a new family, check what input format it expects. A wrong template doesn't make the call fail; it only makes the scores meaningless, so failure logs won't show it.

Chunks go to the reranker as they are, untruncated. If a chunk exceeds the reranker model's input limit, that call fails and search falls back to RRF order, so it keeps working. In an installation where documents are cut into the default 4,000-character chunks, if you switch to a reranker with a smaller input limit, shrink the chunk size too, so reranking actually runs.