Korean text leaking into English installs is blocked at merge by a source check

OwlAI is an on-premises product, so each installation runs in one language. The supported languages are Korean and English. If server code holds Korean copy as a string, that copy looks correct on a Korean installation, and on an English installation that one spot comes out in Korean. However carefully the development team tests on a Korean installation, the defect does not show, and walking the normal path on an English installation usually misses it too. This kind of copy clusters where a turn fails or a request is rejected.

This post describes how we block the defect with two checks instead of relying on people's attention. Defects that leave a trace in the source (Korean strings, prompt sentences that fix the language, rejections without an error code) are blocked by a CI check that runs on every merge request. Defects that leave no trace in the source, meaning code that doesn't know the locale and falls back to the default, Korean, are caught by a nightly test that runs real turns on an English account.

#Three defects that tests on a Korean installation cannot see

The defect takes three shapes.

With the second shape, even when the administrator sets the language to English, that path alone answers in Korean, and nothing on the settings screen points to the cause.

The third shape is the hardest to find. When code rebuilds user info from only a few fields and drops the locale, every path downstream that receives that info builds its copy in the default, Korean. The answer body picks its language through a different path and comes out fine in English; only the failure and tool messages beneath it turn Korean. On a Korean installation even the result is correct, so there is no trace.

#Grepping for Hangul catches the comments too

The first idea is a regular expression that finds Hangul in the source. It doesn't work in this codebase. Comments and docstrings in the backend hold tens of thousands of lines of Korean, and in the frontend, too, most lines containing Hangul are comments. Comments are written for developers and never reach customers, but if the check blocks them, it can't be turned on until every one of them is cleaned up.

Review and manual testing are weak here as well. Korean copy reads as a perfectly good sentence, so it doesn't look wrong to a reviewer. Manual testing covers only the paths someone walks, and this defect clusters in failure and rejection paths that have to be produced on purpose.

#Every merge request checks the strings that reach customers

One checker covers the Python code of the backend, the server-side sidecars, and the updater. It reads the source as an AST (abstract syntax tree) and checks only string literals. For the TypeScript of the frontend and the desktop app, a separate checker reads the AST through the TypeScript compiler API and does the same job. The two are split only because the language tooling differs, and they check different files.

Rule 1. No Hangul in string literals. Comments and docstrings are left out of the verdict. The criterion fits in one line:

If a string stands alone as a statement, it is documentation.

This criterion drops every docstring and leaves only strings that are assigned to a variable or passed to a function. Copy the screen can't regenerate and that is stored or sent out (turn failure messages, web push, meeting minutes section titles, audit log summaries) moves into the server's per-language copy catalog.

Rule 2. Don't fix the language in code. The checker looks for expressions such as in Korean, 한국어로 ("in Korean"), ko-KR, and lang="ko" inside strings. The output language is passed as a response language value, and the locale comes from the user and installation settings.

Rule 3. API rejections carry an error code. A rejection response must carry an errorCode alongside the human-readable sentence. The screen builds its copy from the code, so the sentence the server sends may stay in English. Without a code, the screen can only tell reasons apart by HTTP status. For example, "email already registered" at sign-up and "account has activity and can't be deleted" both use 409. If the screen goes by status alone, an administrator who tried to delete an account with activity sees "This email is already registered." This rule checks the whole API layer.

Only directories that have been cleaned up go into the check's scope. Turning on a scope that still has violations blocks every merge request, and the check ends up switched off again.

#Exceptions carry a reason, and CI fails once one is no longer needed

Exceptions happen. In a module that measures Korean writing style, Korean patterns such as 를 통해 (a stiff "by way of" construction) are the detection target itself; translated into English, the module would have nothing left to do. Code that understands a schedule phrase like "every Monday" typed in Korean needs Korean words too. The checker accepts exceptions at three levels, narrowest first.

A file exception lets new Korean in that file pass, so for a line or two we use a word exception first. Exceptions to rule 2 are per line rather than per file for the same reason. Exempting by expression would let a newly added Always answer in Korean. in the same file pass, so each exemption names the context snippet the line must contain. An example is a style-rule prompt written once in English that splits by language with a conditional clause such as "When writing in Korean: …". That clause doesn't choose the output language; it reacts to a language that has already been chosen.

Every exception needs a reason, and the checker's self-test rejects entries without one. It also rejects exceptions that are no longer useful. If an exempted file no longer contains Hangul, or an exempted path no longer exists, CI fails and says to delete the entry. Left in place, it would let a new file created at the same path start out exempt. The frontend checker records an allowed count for each file not yet migrated. The run fails if the count goes up, and it also fails if the count goes down while the allowance stays the same. The allowlist shrinks only as the code is cleaned up, and it never grows back.

Targets we decided not to check (tests, developer CLIs, the per-language manual source, documents written for people) are also listed with their reasons, which separates a newly introduced gap from "something we never checked."

#Instruction files the model reads are checked down to the comments

Rule 1 looks only at string literals in Python files. The skill instructions the model pulls in when it needs them have the extensions .md and .js, so that check skips them. These instructions are seeded into the database and loaded verbatim into prompts. If the instructions for creating Word, PowerPoint, or HWPX documents are written in Korean, the model on an English installation reads that Korean too.

So skill instructions and scripts get a rule of their own. Python comments are excluded on the grounds that people read them. In a skill instruction file, the comments and everything else are model input, so that reasoning doesn't hold. There is one line:

In skill instructions, the only place Korean may stand is an inline backtick quote.

Korean inside backticks is either a pattern to detect or a proper noun. The font name 맑은 고딕 (Malgun Gothic) has to go into the document file under exactly that name; translated into English, the font can't be found.

Code blocks don't count as quotes. The model copies example code from the instructions verbatim when it builds a user's document. If the example in the Word document instructions contains new TableOfContents("목차"), a document created on an English installation also gets "목차" as its table-of-contents title. So code blocks are checked. Skill scripts get the same rule, since the model opens and reads them in the sandbox.

This rule has chunk-level exceptions too. Government forms put spaces between characters to pad a label to width (기 관 명), and the form-editing instructions keep that label as is because the example has to show that the code finding a cell by its label tolerates the spacing.

Scripts that generate skill instructions are added to the check scope one file at a time. If a generator shares one Korean header between both language editions, a Korean sentence lands in the English instructions, and the cause lies in the generator.

#Fallbacks that leave no trace in the source are caught by real turns on an English account

The third shape is invisible to source checks. Code that drops the locale field contains no string to flag. So we use two things together.

First, when the copy catalog receives a locale it can't resolve, it produces the copy in the default and logs a warning. Places where the locale truly can't be known, such as spots with neither a session nor a user, use a separate fallback that logs no warning, with the reason written in the code. If deliberate fallbacks also logged warnings, the real paths that dropped the locale would be buried among them.

Second, a live test that runs every night on the on-premises test environment produces failing turns on an English account on purpose. A member whose preferred language is English starts a turn that calls a slow test tool, presses the stop button while the tool runs, and then checks that the stored failure message and the last answer on screen contain no Hangul. The runner counts, in the server logs for the test window, the warnings about unresolved locales and puts the count in the run summary. The count must be zero; otherwise the notification carries a language warning. How we check facts so that tests hold up even though the model's wording changes every time is covered in “LLM answers vary by run, so E2E tests check what the server recorded”.

#Design choices

We chose a check that reads the AST and looks only at string literals over a Hangul regular expression. A regular expression doesn't care about language or file type, so one check would cover everything. But it also catches Korean in comments, and then the check can't be turned on. The cost is one checker per language, plus a separate rule for skill instructions, where comments and prose can't be let off.

We use word, context, and chunk exceptions before file exceptions. A file exception means fewer entries to maintain, but new Korean in that file slips through. Narrow exceptions cost more entries and keep catching new violations in the same file.

The error code rule checks the whole API layer instead of only the changes. Change-based checking lets you turn the check on while old violations remain, but it switches the rule off for local runs without a base commit and for release merge requests. So we cleaned up every existing rejection and then moved to a full check. Code that passes the rejection body through a variable can't be judged statically, so it passes, and review covers it.

Instead of switching the default locale to English, we log a warning and make callers pass the locale. Switching the default to English makes the English installation look fixed, but the same defect then shows up as English copy on a Korean installation. The right default is the installation language, and the copy catalog is a low-level module that doesn't read the database, so it doesn't know that value.

The fallback check runs every night. It uses real LLM turns, which cost too much money and time to run on every merge request. In exchange, a fallback the source checks missed shows up in the run summary within a day of the merge.

If your development team works in a different language from your customers, the same combination carries over. Apply the string check only to code that reaches customers, put a warning log on every place that falls back to a default, and look at that warning count every day alongside failing turns on an account in the customer's language. As long as the default is the development team's language, the team's own tests will not see this defect.