Worker restarts: every running turn answers or records its failure

In OwlAI, workers run chat turns (one question through to its answer), not the web server. The web server only accepts a question and puts it on a job queue. A single turn calls tools several times and takes minutes, and meeting transcription or a playbook (an automated job that calls tools in a set order) runs longer still. Changing one line in a settings file still means restarting the workers, and if the container is taken down as is, a running turn is cut off by SIGKILL while the user watches a spinner that never stops.

OwlAI runs deployments and manual restarts through the same script. The script starts one extra container, waits until it is healthy, and only then takes an old one down. The old worker takes no new jobs and, within a fixed time, either finishes its running turns or records their failure itself. This post covers that order and its time budgets, the two things that need separate handling while old and new containers run side by side (the heartbeat and the database connection count), and why updates that change job formats empty the workers first.

#Compose commands take every container down at once

The easiest restart is docker compose restart or up -d --force-recreate. The production compose file runs two web containers and two workers by default, and these commands take both of each down together. Questions that arrive in the meantime have nowhere to go. A short timeout such as restart -t 10 overrides the stop grace period set on the container, so a running turn gets SIGKILL after 10 seconds. The turn cannot record its failure, and the user watches a spinner for about two minutes until the web side's heartbeat check closes it.

Setting the stop grace period to the 30-minute turn limit would mean no turn is ever cut off. But a restart would then wait as long as the slowest turn, and replacing four containers one after another could take two hours in the worst case.

So a restart replaces one container at a time and gives the container going down a fixed time to wrap up its work.

#The old container goes down only after the new one is healthy

For each service, the replacement script repeats the five steps below once per old container. Web goes first, then workers.

1. Start one more container. The old one keeps running.
2. Wait for the new container to become healthy. The default limit is 300 seconds; past it, the script deletes only the new container and stops. The old container keeps working.
3. Pause 6 seconds. The nginx in front re-resolves the web server's name every 5 seconds, so the pause is long enough for it to learn about the new container.
4. Stop and delete the oldest old container without shortening its stop grace period.
5. Pause another 6 seconds so nginx forgets the deleted container's address.

An old container goes down only after its replacement is healthy, so the number of working containers never drops during the swap. An installation that runs a single container follows the same steps and briefly has two only while swapping.

On the web side, nginx handles it. If it picks the address of a container that is already gone and cannot connect within 500 milliseconds, it sends the request to another container. A failure at the connect stage is retried even for POST; a failure after the request was sent is not retried, so a write never runs twice. A web server going down finishes its in-flight requests within 300 seconds.

A deployment puts everything that can fail ahead of this swap: the preflight check (“The preflight check only reads the host and leaves the fixes to a person”), the image pull and the metastore migration. If any of them fails, it stops without changing a single container. Recreating the PostgreSQL container drops every database connection, so it is recreated only when its settings or image contents actually changed. When only the image reference name changed, the script compares the configuration hash and image layers and leaves it running.

#The old worker finishes turns within 270 seconds, or records the failure itself

When the old worker receives the stop signal (SIGTERM), it stops taking new jobs and waits for its running jobs to finish. This is called draining. New jobs that arrive in the meantime go to the remaining workers.

Chat turns have three time budgets in a fixed relationship.

Run plus wrap-up, 300 seconds, has to be shorter than the 330-second stop grace period so a turn leaves its record before SIGKILL. 270 seconds covers the measured p95 turn length (177 seconds). The interruption is the same signal a user sends by stopping a response, so it also cancels the SQL the turn was running on the database.

What users experience falls into three cases. A running turn that finishes within 270 seconds delivers its answer as if nothing happened. A longer one immediately leaves "Could not answer because the server work was interrupted. Please try again shortly." in the conversation. A question sent during the swap is picked up by a remaining worker. With a single worker, it waits in line until the new worker is up. The web server marks the turn as in progress for 420 seconds when it accepts it, so the conversation shows it as in progress meanwhile.

Jobs such as meeting transcription, playbooks and schema analysis have no time budget of their own and run until the 330-second stop grace period. A job not finished by then is cut off by SIGKILL, and another worker finds the stopped job and handles it by kind.

• Meeting transcription overwrites its output segment by segment, so running it again from the start gives the same result. It goes back on the queue.
• A playbook run would call tools it already called (sending mail, creating tickets) if it ran again. It is marked failed, and finished steps are left as they are. The user sees the reason and the step where it stopped, and runs it again.
• Schema analysis would redo every LLM call from the start, so it is closed as failed and the user runs the analysis again.

#A stopping worker keeps its heartbeat so other workers leave its jobs alone

The job queue library (Procrastinate) records a heartbeat for each worker, and OwlAI workers handle the jobs of any worker whose heartbeat has stopped for more than 30 seconds, as described above. The library's default stop procedure, however, turns the heartbeat off first and then waits for running jobs.

Left that way, the swap creates exactly that situation. While the old worker is still transcribing a meeting, the new worker decides 30 seconds later that the old one has stopped, and either puts the transcription back on the queue to run again or marks a running playbook as failed. Waiting to finish a job ends up duplicating or failing it.

So OwlAI workers keep their heartbeat after the stop signal until every running job has finished, and turn it off only after deregistering. If a library upgrade drops this behavior, the worker writes an ERROR log. A worker killed by SIGKILL stops its heartbeat along with everything else, so another worker handles its jobs after 30 seconds.

Workers log the start of draining, each interrupted turn and the moment they are empty, each with the elapsed time since the stop signal. In a normal swap, the log shows zero jobs handled by another worker.

#Database connections peak during the swap

During a swap, one service briefly runs N+1 containers, and each container holds its own connection pools. So PostgreSQL gets closest to max_connections during a swap, not in normal operation. The budget is the sum of pool limits.

With two of each, normal operation needs 2×144 + 2×35 = 358. While a web server is being replaced, it is (2+1)×144 + 2×35 = 502. Adding admin sessions and the superuser reservation, we set the production compose default limit to 600. A limit of 300 falls short in normal operation with only two of each.

Raising the number of web processes or containers changes this sum, so the preflight check computes the same formula from OWLAI_BACKEND_REPLICAS, OWLAI_WORKER_REPLICAS and OWLAI_WEB_WORKERS in .env. If the normal-operation sum alone exceeds the limit, it reports FAIL and the deployment stops before touching any container; if only the swap exceeds it, it reports WARN.

Swap time is counted the same way. With the default 2+2 containers and nothing running, it takes 2 to 3 minutes; if every old container is holding long work, each waits out its stop grace period and it takes close to 26 minutes. If you deploy through a CI job, give it a time limit comfortably above that. A job canceled mid-swap leaves the stack half replaced.

#Updates that change job formats empty the workers first

Starting the new worker first means there is a window when old and new workers watch the same queue. For a restart of the same version, that is fine. But if the new version renames a job or changes its arguments, an old worker can pick up a job in the new format, or the other way round.

So the admin console update (“The update button only approves, and only the updater holds the docker socket”) empties every worker in its Wait for running jobs step before replacing services. The updater stops worker containers one at a time, and each finishes its running jobs within the same stop grace period described above. The updater deletes each stopped container and records the time it waited in the progress view. The deployment script then starts workers on the new version. Old and new workers never run at the same moment, so a change in job format cannot make one pick up the other's jobs.

The cost is a period with no workers at all. Jobs that arrive then wait on the queue. A chat question that no worker picks up within 420 seconds of being accepted is closed with "Could not answer because the server did not pick up the request in time. Please try again shortly." The update confirmation dialog shows the administrator the expected impact during the update, as recorded in the manifest, beforehand. What is rolled back when the service replacement step fails is covered in “When an update fails, the schema change decides between rolling back code and restoring the dump”.

#Design choices

A turn past 270 seconds is closed as failed rather than moved to another worker and continued. Continuing would mean handing the turn's mid-flight state, such as tool calls still waiting for results and the effects of external tools already run, to another process, and we judged the risk of running the same tool twice to be worse. The cost is that a very long turn has to be resent, but the user gets a try-again message right away instead of a spinner.

We sized the run window to the measured p95 turn length rather than the longest turn. Restart time is not dragged out by the slowest turn, but a turn that does not finish within 270 seconds of the stop signal is closed as failed.

A stopped job goes back on the queue only if running it again from the start gives the same result; everything else is closed as failed. Requeuing would spare users a second click, but repeating an external action is the bigger risk. If a new job kind is left out of this classification, a build-time test blocks it.

For updates, we chose emptying the workers first over running old and new side by side. Jobs wait on the queue for the few minutes with no workers, but in a version that changes job formats, old and new workers never pick up each other's jobs. A manual restart of the same version does not change formats, so it overlaps them.

If your service runs long jobs on queue workers, the same order carries over. Take the old container down only after the new one is healthy, give each job kind a time budget that fits inside the stop grace period and a defined outcome when it does not finish, and keep the heartbeat running while the worker stops.