Get started

Engineering NotesStructured data and NL2SQL

DDL doesn't say what a column means, so we read its values first

The order table's STATUS_CODE CHAR(1) holds S, D and C. If the model guesses that C means complete, "revenue from orders completed last month" comes back with canceled orders mixed in. The SQL runs without an error and only the number is wrong, so neither the database nor the model notices the mistake. That C means canceled appears nowhere in the DDL. It lives only in the values actually stored in the column and in the code table they pair with.

So we work out what columns mean before any question arrives. When an administrator runs schema analysis, OwlAI reads, for each table, the statistics, value distributions, sample rows and actually executed SQL the database has already gathered, and an analysis agent opens code tables only for the columns whose meaning is still unsettled. The result is stored as a table card that puts facts and meaning side by side for every column, and chat pulls that card out at question time.

A column guessed from its name runs fine even when the guess is wrong

On business databases, columns whose names don't tell you their meaning usually come in three shapes.

Column What the DDL doesn't say When the guess is wrong
Code column STATUS_CODE CHAR(1) That S, D and C mean shipping, delivered and canceled Canceled orders land in completed revenue
Free-text column REMARK VARCHAR(400) That support agents type notes into it by hand, and its most common value is a single period The model counts "orders whose remark contains 'canceled'" and reports that as the cancellation count
Undeclared reference PROD_TP That it links to the product type code table (no foreign key) The model lists raw codes, or joins to another table with a similar name

All three execute successfully. A syntax error gets rejected by the database, which gives the model a chance to fix it, but SQL that misreads a column's meaning goes straight into the answer. The only way to stop it is for the model to know what the column means before it writes the SQL.

Investigating at question time means looking at the same column again and again

The closest alternative is to look at values after the question arrives. There is nothing to pay up front and the values are always current, but the round spent asking "what does S mean?" is repeated for every question and lands on the response time each time. Nothing stops the model from skipping the investigation and guessing from the name, either.

Another option is to count the source tables directly, the way data profiling does. Running COUNT(DISTINCT) and GROUP BY on every column is exact, but on a table with hundreds of millions of rows each column becomes a full scan, and that load falls on the production database. Creating missing statistics with ANALYZE or DBMS_STATS is a write that changes the state of the source database.

So we work out meanings once, before any question, and take the inputs from what the database has already gathered.

The analysis reads only statistics the database already has

The first step gathers facts without making any judgments. Each input has a fixed place it is read from.

Input Where it is read from
Columns and types, keys and constraints, indexes, table and column comments The catalog
Row counts, distinct value counts, NULL ratios, minimums and maximums Optimizer statistics
Value distributions (the most common values and their share of rows) Optimizer histograms: most common values from pg_stats on PostgreSQL, FREQUENCY, TOP-FREQUENCY and HYBRID on Oracle, FREQUENCY on Tibero, singleton on MySQL, popular values from JSON_HB on MariaDB
Statistics status When statistics were last collected and whether they are stale. It distinguishes never collected from unreadable for lack of privileges
SQL actually run against this table pg_stat_statements on PostgreSQL, performance_schema on MySQL and MariaDB, V$SQL on Oracle and Tibero

A value distribution keeps up to 50 values per column, along with whether it lists every distinct value and what percentage of rows the listed values cover. A column without a histogram gets no value distribution, and a missing statistic is left empty rather than filled with 0, because the model has to tell "0" apart from "unknown." From the executed SQL, statements OwlAI itself sent are removed. What remains serves as examples of how this table is joined and filtered.

The agent checks values only for columns whose meaning is unsettled

The gathered facts go into the table analysis agent's input from the start: comments, keys, statistics, value distributions, statistics status, 5 sample rows and 5 executed SQL statements. The agent runs on the same agent loop as chat (“One agent loop for every feature, with the differences in hooks”) and has three read-only tools.

Tool What it does
Referenced-table read Reads rows of a referenced table, such as a code table, to confirm which label goes with which code
Conditional sample Tests a hypothesis such as "rows with C are canceled orders" as a WHERE condition
Table list Returns every table name the connection account can see

There is no tool that counts values. The value distributions are already in the input, and a tool running GROUP BY would bring back the full scans the first step avoids. The target table is fixed in the tools, so the agent passes only columns and conditions, and the two tools that send SQL attach a row limit. A single table gets 40 tool calls in total.

The system prompt tells the agent to use tools only on columns the catalog leaves unsettled. For a matching code table, the agent confirms the real name in the table list, then reads it. A name that isn't in the list is refused by the referenced-table read tool before any SQL is sent.

The agent returns one result of a fixed shape.

Item What it holds
Purpose The table's purpose and role, in two or three sentences
Column descriptions One line per column. For code columns, the meaning of the main codes goes in
Table role One of: master, time-stamped transactions or events, code table, many-to-many link, view
Code table details If this table is itself a code table, its code column and label column
Code decoding For each code column in this table, the code→meaning pairs and the source code table
Columns to avoid Columns that, despite their names, must not be used for classification or filtering, and why
Inferred joins Join relationships not declared as foreign keys, with evidence and a confidence of low, medium or high. High is used only for what was verified by value overlap or by reading the referenced table

Before anything is saved, code filters the result once more. Code table details that point to columns the table doesn't have are dropped, and inferred joins that duplicate a declared foreign key are removed. These checks give the same answer every time, so they are not left to the model.

One card puts facts and meaning side by side for every column

The last step merges the gathered facts with the agent's result into a table card. For each column, the card writes the type, key role, NULL ratio, distinct count and value range on one line, followed by the comment the customer wrote in the database, the meaning the agent wrote and the most common values.

One table card: database facts and analyzed meaning, column by column
Figure 1. One table card: database facts and analyzed meaning, column by column

If a column has 20 or fewer distinct values in total, the card lists them all. Otherwise it lists the top 10 with the share of rows they cover and notes that other values exist, so the model doesn't conclude that a value missing from the list doesn't exist. The card's header carries the purpose and role, the row count and the statistics status, and when statistics have never been collected, it says to read the row count and NULL ratios as unknown, not zero.

The same card is used in two places. Rendered as text, it is what table search covers (“Wrong-table SQL still runs, so search narrows the candidates first”), and the table card tool returns the same card to the model as structured data. What the model sees in search therefore never disagrees with what it sees when it opens the card.

Chat treats code decoding as a hint and joins the code table

The code decoding on a card is a sample taken at analysis time. If a new code is added to the code table or a label changes after the analysis, the card still holds the old content. So the chat system prompt tells the model not to decode codes from it directly. When a code column appears in a result, the model LEFT JOINs the code table and shows the label alongside, and when it filters by a label, as in "orders being shipped," it puts the condition on the code table's label column.

That join needs a path saying the order table's STATUS_CODE leads to the ORDER_STATUS table. Even without a foreign key, if the agent confirmed that every value in the two columns overlaps, it records the path as an inferred join with high confidence.

Columns to avoid work the same way. If the most common value in REMARK is . and the rest are hand-typed sentences, the agent puts the column on the list with the reason "order status is in STATUS_CODE." The chat system prompt tells the model not to put WHERE or GROUP BY on columns in this list and to use the column they point to instead.

Decisions that need several tables at once come from the database summary

The table analysis agent looks at one table only. Whether a table is a backup or staging copy, or a partition, of another table can only be known by looking at several tables together, so after table analysis finishes, the database summary makes that call and writes it into each table's analysis result. Search uses it to put copies behind the primary table (“Wrong-table SQL still runs, so search narrows the candidates first”).

The database summary also collects code table values. Only for code tables whose row count statistic is 1,000 rows or fewer does it read every code and label with DISTINCT; larger tables and tables with an unknown row count are not read. The collected values go onto that code table's card.

Analysis cost scales with table count

The agent loop runs once per table, so cost scales with the number of tables. Across 14 analysis jobs (about 520 tables) run in October 2026, observed in production, each table took around 15,000 input tokens (roughly half of them prompt cache hits) and around 1,000 output tokens. The largest request finished some 270 tables in a little over 16 minutes. Tables within one request are analyzed 5 at a time by default, which you can change with the installation setting analyze.parallelism.

So administrators set the scope. On the database screen they can pick only the tables they need and rerun only the failed ones. Every query the analysis sends is read-only and goes through the same time budget and cancellation path as chat (“Read-only and query cancellation have to reach the database server”). For databases connected through OwlDB, analysis queries go through OwlDB too, so if there is a standby node, the primary node is not touched (“AI queries pick their database node at run time, not at registration”).

Design choices

Value distributions are read from the database's statistics, never counted on the source. The analysis puts no full scans on the production database, and in exchange a column without a histogram has no value distribution, and the values that are there are sampled approximations. Default collection also differs by engine. Oracle's defaults build histograms only on columns used in WHERE clauses, and MySQL builds one only after ANALYZE TABLE … UPDATE HISTOGRAM is run for each column. So when a value distribution is not complete, the card says so, and for columns with no distribution the agent checks sample rows and conditional samples.

Code decoding is stored, but chat doesn't use it in answers. The SQL carries one more join, and in return codes added after the analysis still come out with their current labels.

Analysis results stay as they are until reanalysis. Schema checking finds tables whose columns were added or changed and marks them, and the administrator reanalyzes only those tables (“Schema analysis goes stale, so only changed tables get reanalyzed”). The cost of this choice is that the old card is used in the meantime.

This approach fits business databases where many column names look like codes, foreign key declarations are rare and free-text columns such as remarks are mixed in. Before analyzing such a database, check first that the source database is collecting statistics and histograms. The value distributions the analysis reads are only as accurate as those statistics.