Deploy success is judged by failed turns after the deploy, not by health checks

A deployment can be broken while /health returns 200 and every container shows healthy. The old version may still be running instead of the one you approved. The containers may be up while every chat turn (one question in, one answer out) throws the same exception. After a database restart, a process that keeps a dead connection in its pool fails only on the first request that takes that connection out. A health check only asks whether the process is alive, so it passes all three.

After the on-premises updater replaces the services, its startup check looks at four things in order: the version label on the running image, container state compared with before the update, exception markers in the backend and worker logs, and the turns and jobs that failed after the update started. If any one of them trips, the update fails, and the updater rolls back the code or stops by the rules in “When an update fails, the schema change decides between rolling back code and restoring the dump”. This post explains what each check uses as evidence to call a failure.

#The health check stays light, and deployment verification runs once on its own

The backend's /health always returns ok while the process is running. The handler answers straight from the event loop. If it ran in the thread pool, it would queue behind in-flight chat turns, time out, and mark a perfectly healthy process unhealthy. The readiness probe, /health/ready, is separate: it sends SELECT 1 to the metastore (OwlAI's own PostgreSQL) and returns 503 if that fails.

You could make the health check ask for the version and ping the database and the queue. That puts every verdict in one place. But Docker calls the health check every 15 seconds, and once a heavier check runs slow, a healthy container turns unhealthy and the rolling replacement stops. The bigger problem is that a health check only sees the present moment. Process state cannot tell you whether the requests that arrived after the deployment actually failed.

So the health check stays light, and deployment verification is a separate startup check that runs once, right after the deployment. Its reference time is when the updater started applying the update. Logs and turns are read only from that time on.

#The running version is checked against the image label

At build time, the backend image gets its release version in the standard OCI label org.opencontainers.image.version. After the replacement, the updater reads that label from the running backend container and passes only if it matches the approved version. A missing label or a different version fails. The web server and worker use the same image, so the backend alone is checked.

Operators can see the same fact on screen. The "Running version" line on the admin System status screen shows the release version that is running and the first 12 characters of the build commit. An image built without a tag shows "Untagged build" instead of a version. The update screen's current version also shows such a build as "Development build · commit …" and does not compare it with new versions. If you see that on a production installation, that alone is worth investigating.

#Container state counts restarts as the change since before the update

docker compose up -d creates the containers and returns immediately. If a service dies right after starting, the deployment script still finishes successfully. So after the deployment, the updater reads the installation's containers every 5 seconds and puts the result into one of three outcomes.

If containers are still waiting after 5 minutes, the check fails with the list of what is left.

Restarts are counted as the difference from the value recorded before the update. A container that lives across deployments, such as postgres, carries its old restart count, and an absolute count would flag a healthy container as a crash loop. Containers that run once and exit with code 0, such as migrations, are left out.

#One traceback or AdminShutdown line in the logs fails the update

A container can be healthy while one code path throws. Once the containers pass, the updater searches the backend and worker logs since the update started for two markers. One is the head of a Python traceback (Traceback (most recent call last)). The other is AdminShutdown, which the driver logs when PostgreSQL closes a connection by administrator command. OwlAI logs are one JSON object per line, and exceptions go into the exception field as the raw traceback, so a single string search finds them. In 24 hours of logs from our internal production environment, both markers appeared zero times. That is why a single line counts as a failure.

AdminShutdown is a marker because of resources the health check never touches. When the database restarts, the server closes every connection it had. If the connection pool does not check a connection when handing it out, the dead connection stays in the pool. The readiness probe sends SELECT 1 through a different session, so it keeps returning 200. The failure only shows up on the first request that uses that pool.

The connection pool the web server uses to put jobs on the queue is one such place. That pool runs an empty query each time it hands out a connection, discards dead ones, and checks the next. On the worker, the queue library runs the same check on its own pool. Even so, after any work that stops the database (a PostgreSQL major version migration, a dump restore, or an engine upgrade on a managed database), the operating procedure is to restart the web server and worker and confirm zero AdminShutdown lines in their logs. A pool check has to be set on each pool, while a restart opens every connection the process holds fresh. The on-premises PostgreSQL major version migration runs inside the deployment script before the web server and worker are replaced, so that restart comes with it.

#Failed turns are sorted into four groups by error message

Unlike a health check, a failed turn is a failure a user actually hit. OwlAI records each chat turn's status and failure message in the metastore. The startup check counts turns started since the update began, by status, and sorts the failed ones into four groups by their message.

• Regression: a traceback, a message shaped like a built-in Python exception, or a turn that ended without a runner result or whose stream stopped without finishing. The code is broken.
• Interrupted: the worker stopped, the worker was lost, the turn never started, or the watchdog closed it. The rolling replacement took long; it does not count as a code defect. “Worker restarts: every running turn answers or records its failure” covers how work survives the replacement.
• Legitimate refusal: reason codes such as model, credential, or quota problems, and HTTP errors from the provider.
• Unclassified: a message that fits none of the above. Samples go into the verdict.

The update fails on one or more regressions or three or more unclassified failures. You can adjust both thresholds with deployJudge.regressionFailureLimit and unclassifiedFailureLimit in the config file. The unclassified threshold is higher than the regression threshold because some paths lose the exception type. When the runner keeps an exception only as a sentence, built-in exception messages such as list index out of range cannot all be recognized by shape, and those failures land in unclassified. One could be chance, so the signal is when they pile up.

Meeting and playbook jobs that failed in the same window are sorted into the same four groups. Jobs, though, are counted by when they failed, not when they started, because meetings and scheduled runs often start before a deployment and finish after it. Job regressions are added to turn regressions against the same threshold. Unclassified jobs appear in the verdict as a count only and are not held to a threshold. A failed meeting records a user-facing notice as its reason, so its messages cannot be sorted by shape the way turn messages can.

#A check that cannot reach a verdict also fails

A one-off helper container started from the new version's backend image runs the turn count. It uses the same database connection settings and configuration as the app and writes its result as a single JSON line. The count only reads the metastore.

The updater handles the three ways a count can fail to reach a verdict as follows.

Only a missing verdict line leads to another run. The count only reads, so rerunning a run whose result was lost gives the same answer. If "cannot decide" counted as a pass, a new version that cannot reach the database would be recorded as a success, so it fails the same way a regression does.

If the window has no turns and no failed jobs, the count passes. The updater then writes a warning to the update progress log that there was no evidence. You read the progress log in the admin console and through the host's update CLI. The version label, container, and log checks still apply in this case.

OwlAI SaaS deployments run the same count with the same 10-minute limit and the same retry rule, and roll back the deployment if it fails. Both deployment paths use one verification standard.

#Design choices

• We chose a verification step that runs once after each deployment over a heavier health check. The health check stays light enough for Docker's 15-second interval, and deployment verification can take up to 10 minutes. The cost is that verification only runs at deployment time. If the database alone restarts with no deployment, the operating procedure (restart the web server and worker, confirm zero AdminShutdown lines) has to catch dead connections.
• We sort failures by the shape of their messages and count the unknown ones as a signal, instead of storing exception class names. Class names would sort more accurately, but every path where the runner records an exception as a sentence would have to change, and new exceptions would still be missing from the list. The unclassified threshold catches defects that are not on the list. The cost is that people read the samples in the verdict and add rules.
• Groups with weaker evidence get a higher threshold or none, because a wrong verdict rolls back a production update. A single unclassified failure passes, and unclassified jobs appear only as a count. So a job defect that leaves no traceback shows up only in that count.
• When the window has no turns, we chose to leave a warning instead of running a test turn ourselves. A test turn adds model cost on the installation and an audit log entry someone has to own, and it cannot run on an installation where no model is configured yet. If your installation updates when nobody is using it, the cost of this choice is checking the update log and failed turns once more during the first busy hours after the update.

If you run a service where one request takes minutes, as with LLM agents, and failure reasons are stored as sentences, you can carry this verification over. Leave the health check to confirm the process is alive, and add a step to your deployment that sorts and counts failure messages right after each deployment.