Korean particles break full-text search, so we join words with OR

A table note says 「반품 주문은 ORD_RTN 테이블에 따로 쌓인다」 ("returned orders pile up separately in the ORD_RTN table"), and a user asks 「지난달 반품 주문 건수를 알려 줘」 ("tell me how many returned orders we had last month"). A person sees right away that the two belong together. PostgreSQL full-text search finds nothing. The simple configuration does no morphological analysis, so 「주문은」 ("orders" with a topic particle) and 「주문」 ("orders") are different tokens, and a question joined with AND only matches a note that also contains 「지난달」 (last month), 「건수를」 (count, with an object particle), 「알려」 (tell), and 「줘」 (me). The same problem hits every keyword search that looks up table cards, documents, and past conversations.

OwlAI does not add a Korean morphological analyzer. It solves the problem on the query side: it splits the search text into words, strips symbols, and joins the words with OR. The net is wide so that rows that should be found are not missed, and the ranking functions and semantic search put the candidates in order. This post covers how the query is built, what fills the gap that remains, and what gets weaker when the embedding server stops.

#In the simple configuration, each spaced word is one token

PostgreSQL full-text search splits text into tokens and normalizes them with the dictionaries of a language configuration before indexing. The English configuration adds stemming that turns cats into cat. All four of OwlAI's keyword searches use the simple configuration, which only folds case and leaves word forms as they are. “Wrong-table SQL still runs, so search narrows the candidates first” explains why that configuration was chosen.

In Korean, the difference shows up in full. Index two notes with simple on PostgreSQL 16 and these are the tokens:

Each spaced word, particle or ending included, becomes a single token. 「주문은」, 「주문도」, and 「주문」 are three different tokens, so a search for 「주문」 matches only the second note.

English identifiers behave differently. The default parser breaks ORD_RTN at the underscore into ord and rtn, and an ORD_RTN in the query becomes the phrase condition "ord immediately followed by rtn," so it still matches. Finding a table name or code value from the question, character for character, works fine under simple. Korean words are what break.

#A natural-language question joined with AND matches nothing

The easiest function for turning search text into a full-text query is plainto_tsquery. It joins every word with AND. 「지난달 반품 주문 건수를 알려 줘」 becomes 지난달 & 반품 & 주문 & 건수를 & 알려 & 줘, and only rows containing all six tokens match. Both notes above drop out. The first note has 「주문은」 rather than 「주문」, and neither note has 「지난달」, 「건수를」, 「알려」, or 「줘」.

Questions always carry words unrelated to what is being looked for: requests such as "tell me" or "summarize," periods such as "last month," and aggregate words such as "count" with a particle attached. Notes and document chunks rarely contain all of them, so with AND, the longer the question, the closer the result gets to zero rows. When keyword search returns nothing, hybrid search is left with semantic search results alone, and the part that matched table names and code values by their characters disappears.

#Morphological analysis and n-gram indexes need another PostgreSQL extension

The textbook answer is morphological analysis. An analyzer splits 「주문은」 into 「주문」 and the particle 「은」 at index time, so word forms in the query and in the text line up. This is what Elasticsearch's Nori analyzer does. PostgreSQL ships no Korean configuration, so morphological analysis means installing an extension and dictionaries separately. Internal system names and abbreviations missing from the dictionary have to be added to a user dictionary before they split correctly.

Character n-grams are another option. With the pg_trgm (three-character) or pg_bigm (two-character) extension, 「반품」 finds 「반품된」 ("returned," as a modifier). The index grows, though, and the search operators change. Korean has many two-syllable words, and three-character units are coarse for them: on the same PostgreSQL 16, similarity('반품', '반품된') from pg_trgm was 0.4.

The only PostgreSQL extension OwlAI's metastore requires is pgvector. The metastore has to run on the PostgreSQL bundled with the on-premises product and on managed PostgreSQL, and managed services each fix the set of extensions you can install. Every extra extension adds "can this extension be installed?" to the installation requirements. So the index stays on the simple configuration, which works with the default distribution, and the problem is solved on the query side.

#Split the search text into words and join them with OR

The keyword searches over table cards, table notes, document chunks, and past conversations all build their query with the same function.

1. Split the search text on whitespace.
2. In each word, keep only letters (including Hangul), digits, and underscores, and drop every other symbol.
3. Remove empty words and duplicates.
4. Join the remaining words with OR. If no words remain, skip keyword search.

The example question becomes 지난달 | 반품 | 주문 | 건수를 | 알려 | 줘, and both notes match: the first through 「반품」 (returns), the second through 「주문」 (orders).

Step 2 picks the characters to keep rather than the symbols to remove, because to_tsquery reads operators such as &, |, !, and <->, along with parentheses, quotes, and colons, as syntax. If search text such as "products with price<100" goes in as is, to_tsquery raises a syntax error and that whole search fails. Removing only a handful of operators fails the same way as soon as one symbol is missed. Keeping only letters, digits, and underscores means no symbol can reach the query grammar. Underscores stay so that identifiers such as CUST_STAT_CD remain whole. The joined query string is never embedded in the SQL; it is passed as a bind parameter.

#Ranking filters the candidates OR lets in

OR trades precision for recall. In the example, the second note matched on 「주문」 alone and has nothing to do with returns. Each note matched exactly one token, so ts_rank on PostgreSQL 16 gave both the same score (0.0101). Keyword search alone cannot tell them apart.

So keyword search only produces a wide set of candidates, and later stages put them in order:

• The full-text ranking function puts rows with more matching tokens first, and each search cuts the list at a fixed count.
• RRF merges the result with the semantic search ranking, so rows ranked high in both searches come first (“Wrong-table SQL still runs, so search narrows the candidates first”).
• In document search and past conversation search, on installations with a reranker configured, the reranker reads the question and each candidate together and re-sorts them (“When document search ranks lookalike chunks first, rerank them”).

If the first note, closer to what the question means, ranks high in semantic search, it overtakes the second note in the merged ranking.

#Semantic search connects different forms of the same word

Some cases OR cannot connect. Search for 「반품된 주문 테이블」 ("the returned-orders table") and the query is 반품된 | 주문 | 테이블, while the first note's tokens are 반품, 주문은, and 테이블에. Nothing matches. Only the second note, which contains 「주문」, comes back. Judged by keyword search alone, the note being looked for is missing and an unrelated note ranks first.

No amount of query cleanup closes this gap inside keyword search. Stripping 「된」 from 「반품된」 requires knowing that it is an ending, and making that call is morphological analysis. Semantic search covers this part. Embeddings compare 「반품된 주문」 and 「반품 주문은」 by meaning, not by word form, so the first note becomes a candidate on the semantic side. In hybrid search, keyword search matches characters such as ORD_RTN exactly, and semantic search connects Korean words whose particles and endings differ.

#Korean search gets weaker when the embedding server stops

Embeddings come from a separate model server. When it stops responding, semantic search drops out, and keyword search carries the gap above on its own. Without a query embedding, table note, past conversation, table, and document search continue on keyword search alone:

Because the query uses OR, keyword search still does not come back empty. Ask about 「반품 주문」 and it finds notes that contain 「반품」. Ask only with a changed form such as 「반품된」 and it misses them. While the embedding server is down, Korean search is weaker by that much.

#Design choices

We put recall ahead of precision. The OR query avoids missing rows that should be found and lets noise into the candidates; the ranking functions, semantic search, and the reranker restore the order. AND would keep the candidates clean, but for conversational questions the clean set is empty.

We did not bring in a morphological analysis extension. The metastore's installation requirement stays at pgvector alone, and the price is that the same word with a different particle or ending remains a different token in keyword search, leaving that gap to semantic search. When the embedding server stops, the gap shows.

For symbols, we pick the characters to keep instead of the ones to remove. The price is that price<100 becomes a single token, price100, which matches neither price nor 100 in the text. Missing one word was preferable to the whole search failing on a syntax error.

This combination fits when the searched text is Korean, questions arrive in conversational form, a morphological analysis extension is hard to add to the installation requirements, and semantic search is available alongside. If Korean documents have to be found with keyword search alone and no embeddings, a morphological analysis extension should come first.