On-premises OwlAI is observed through one JSON log, joined to the worker by a request ID

An on-premises product runs on the customer's servers. When something breaks, the vendor often has nothing to look at but the logs the customer has collected. You can ship OpenTelemetry instrumentation and a Prometheus /metrics endpoint in the product, but if the customer never stands up a collector and a metrics server to receive and store those values, nobody reads them. Logs, on the other hand, every customer already collects. So OwlAI is observed through a single structured JSON log written with structlog.

For one log to be enough to trace an incident, three things have to hold. Each line has to be a fact a machine can filter. A single ID has to run from the request the web process accepted to the LLM calls the worker made. And user input and secrets must not leak into the log. This post covers how we met each of those, and how much log the product keeps on disk.

#A metrics server and a trace collector are each one more system for the customer to run

Putting instrumentation into the product is the easy part. The hard part is on the receiving end. Storing metrics takes a Prometheus server, storing traces takes a trace collector and a backend, and looking at either takes dashboards. On-premises, the customer installs each of those tools, sets its retention, puts it through security review, and upgrades it.

Logs carry none of that cost. Line-delimited JSON on standard output and in files plugs straight into the log collector the customer already runs (Promtail, Fluent Bit, and so on). Facts that must last are not left to the log; they go into database tables. Per-call tokens and costs go into the usage ledger (“Reconcile the LLM cost ledger with the bill, and count cache tokens by response format”), and who touched what goes into the audit log (“Auditing an AI agent: log every query it runs, not the conversation”).

#Each line is one event name plus fields

Output is JSON in production and development alike. Every line automatically carries a UTC timestamp, the level, the logger name, the service name, and the environment.

The message slot holds only a fixed, dot-separated event name such as http.request or llm.round, and values go separately into snake_case fields. A sentence like "2 of 3 requests failed" reads well for a person, but a collector cannot filter or sum it by failure count.

A line's level follows from what the line says.

Every fallback branch logs, not only except blocks. A common failure in LLM and external calls is "200, but empty," so a branch like if not result: that falls back also logs, along with the basis for the decision (measured values, caps). A fallback that leaves no log makes a feature look fine while its quality drops. Decision functions, which take input and return a plan, don't emit WARNING internally, because the same warning would pile up on every loop iteration. They return the observed value, and the caller logs it once.

Lines with no signal are kept out of INFO. The chat and meeting screens poll for progress every few seconds. Logged at INFO, those successful responses add up to 105,028 lines of meeting-progress polling and 22,579 lines of chat-progress polling over seven days, half of the backend log (observed in production). So successful responses on polling paths drop to DEBUG, and health checks are left out of the access log. On polling paths, 4xx responses and responses slower than one second still log at INFO. A 409 from meeting polling is the signal to stop recording, and without that line there is no answer to "why did the recording stop?"

#One request ID runs from the web access log to the LLM round

The web process assigns a request ID (requestId) to every HTTP request and attaches it to every line that request produces. If the front proxy sent an X-Request-Id header shaped like an ID, that value is used; otherwise a new one is generated. Values containing line breaks or that are excessively long could forge or bloat a log line, so they are not accepted. The same ID goes into the response header and into the body of error responses. The customer's proxy log and OwlAI's log share the ID, and a single error response a user received is enough to find that request's logs.

A chat turn (one question in, one answer out) is accepted by the web process and run by a worker. The web process puts the accepting request's ID into the job it queues, and the worker picks that ID up when the turn starts and attaches it to its own logs. Lines the worker writes, and lines from the LLM calls inside that turn, all carry the same request ID. Work that runs without an HTTP request, such as analysis, gets a job ID (jobId).

The LLM call kernel writes one llm.round line per round (one round trip of calling the model and getting a response). Playbooks, schema analysis, meeting minutes, and document conversion run the same tool loop as chat, so every kind of work produces lines of the same shape. A chat turn also writes a call record to the database for every round, but that record hangs off a conversation, so work without a conversation cannot go in. That is why round observation goes into the log.

Each line carries four groups of fields.

• What it belongs to: stack (chat, playbook, analysis, meeting, conversion), unit type and ID, node, parent unit, nesting depth, round number
• What it called: provider, model, number of tools passed and a sample of their names, whether structured output was requested, tool_choice
• How it ended: latency, status, names and count of the tools the model called, body length
• Tokens: input, output, cache read, cache write

The caller sets "what this call belongs to" as a scope, and the kernel reads the scope to fill in the fields. A call that arrives without a scope is logged with the stack name unscoped. That value is how a new call path that forgot its scope gets found.

Status takes four values: ok, error, cancelled, and incomplete. Cancellation, where the user stopped the turn or the wait for a response expired, is not an error and logs at INFO; errors log at WARNING. Incomplete means a response arrived but was cut off at the output cap. No exception is raised, so it looks the same as ok; we raise it to WARNING and record the reason for the cutoff. A round in which the model repeats itself until it hits the output cap and returns an empty body has this shape (the server-side cap is covered in “Measure the vLLM concurrency limit with production input lengths, not short prompts”).

Token fields are counted by the same function as the usage ledger (“Reconcile the LLM cost ledger with the bill, and count cache tokens by response format”). The same line also carries hashes of the start of the request, which are used to find why the prompt cache broke (“One system prompt edit zeroes cache reads, so changing blocks go last”). If emitting the event fails, the LLM call does not.

#User input is stripped where logs are serialized, not where they are written

Secrets and users' original text are not stored in the log. Being careful in every piece of code that writes a log is not enough, because libraries and the database write lines of their own.

The queue library (Procrastinate) puts a job's full arguments into the message and fields every time a job starts and finishes. A chat turn job's arguments contain the user's question verbatim. Trimming arguments task by task leaks again with every new task. So queue logs are filtered by an allowlist at the one JSON serialization point that standard output and the files share. What survives is job information such as task name, job ID, queue, status, and attempt count, and return values are stripped too. When the library adds fields, the new ones are not on the allowlist and drop out.

The metastore PostgreSQL logs statements that take longer than one second, and at its default it logs the bind parameter values with them. That is a path for user questions and prompts to pile up in the collector by way of the database log. So the compose file pins log_parameter_max_length=0. The access log does not record query strings, and for paths that contain a token, that segment is masked.

#Log files inside the product are a buffer until the collector picks them up

Logs are kept in two places. Container logs rotate through Docker's json-file driver, 20MB per file, up to 5 files. They disappear when the container is removed, so they are for checks right after a deployment.

The persistent copy is written to files in a host directory (by default /var/log/owlai/<service>) and survives container recreation. Each process writes its own file. The web service runs several processes, and if they shared one file, their log rotations would collide and lose lines. The file name includes both the process ID and the container hostname, so names don't collide even when several copies of the same service run. Each file rotates at 20MB with 5 more kept, and at startup the files of past processes untouched for more than 14 days are deleted. If the directory is not writable, the process logs one warning and keeps running on standard output alone.

Queries spanning months go to the customer's collector, which ships these files or standard output. The customer sets the retention, and queries can span deployments no matter how many times containers are recreated.

#Design choices

• We chose log aggregation over metric time series. A time-series store answers a query like "average latency over the last hour" directly. But a metrics server in the product is one more observability tool for the customer to run. The same values come from aggregating the access log's elapsed time and llm.round's latency field in the customer's collector.
• We chose a request ID over distributed tracing. A single request ID ties the web process, the queue, and the worker together, and llm.round gives per-round timing. That answers which request took how long and where, without a collection backend. There is no trace view stitching spans together; log search fills that role.
• Log files inside the product are capped at 120MB per process. That keeps logs from filling the host disk and stopping the service, at the price of losing older lines the collector hasn't shipped. Facts that must last go into tables from the start.
• llm.round records at most 40 tool names as a sample, with the count logged separately. Printing the whole tool catalog every round would make the log as large as the prompt. A changed definition for a tool outside the sample is still caught by the hash.

The same choices transfer to a product that customers install and operate themselves, where each customer already runs a log collector and facts that must last can be kept in dedicated tables.