When an update fails, the schema change decides between rolling back code and restoring the dump

When an update fails, the operator's first move is to redeploy the previous image. Redeploying an image rolls back the image only; the database schema stays as it is. If the new version's migration dropped a column, the old code still starts, but every screen that reads that column errors out. Meanwhile writes keep arriving through the other screens, and the moment you restore the database from a dump, all of them are lost. It looks rolled back on screen, but there is more to clean up than right after the failure.

The OwlAI on-premises updater picks an end state from two facts when an update fails: which step failed, and whether that version's schema change is expand (the old code can run alongside it) or contract (it cannot). Where rolling back the code alone is enough, the updater carries the rollback through. Otherwise it stops without redeploying the containers and leaves a person to restore the database from the dump taken just before the update. This post covers that decision and the restore procedure. The permission design that separates approval from apply, and the criteria for expand and contract, are covered in “The update button only approves, and only the updater holds the docker socket”.

#An image rollback works only after expand changes

The manual deployment path has a one-step rollback to the previous digest recorded in the image pin file (“Deploys and air-gapped loads are checked by digest, not by tag”). It changes only the images and leaves the settings in the install directory and the database schema alone. Whether it is a real rollback therefore depends on which migration revisions went up in between.

An expand revision leaves the columns and constraints the old code reads and writes as they are. The old code never reads the new columns, so it keeps running on the new schema. Backend startup knows this case. When the database's Alembic revision is one the code does not know, startup treats it as a schema raised by newer code, skips migration, writes one warning line and comes up. Without that check, alembic upgrade head stops with an error on the unknown revision, and a code-only rollback fails at startup.

After a contract revision, this rollback does not work. A column the old code read is gone, or a newly added NOT NULL or CHECK constraint rejects the old code's writes. Startup succeeds, but the first read of that column or write to that table fails. The only way back is restoring the database from the pre-update dump.

The common alternative is to ship a script that reverses the schema with every revision (a down migration). Alembic supports downgrade, and done well it rolls back the schema without a dump. But the down script for a revision that dropped a column brings the column back, not the values that were in it. Reversing an expand revision with its down script loses whatever accumulated in the new columns. And if that script fails the first time it runs on a customer installation, the restore itself becomes a new incident.

#The dump is taken on every update, whatever the version

To leave a way back from a contract version, there has to be a dump. The updater takes it on every update, not only for contract versions. The signed manifest carries a separate value saying whether this version needs a backup, but the updater does not skip the backup when that value is false. Expand and contract are a judgment a person records for each release (“The update button only approves, and only the updater holds the docker socket”), and there has to be a way back even when that judgment is wrong.

These rules apply to the dump.

• It covers the metastore (OwlAI's own PostgreSQL holding users, conversations and settings) and is taken in pg_dump custom format.
• Only the two most recent dumps are kept. If you need an older one, move it elsewhere before approving the next update.
• The dump includes user accounts and stored credentials, so the file is created as 0600 and the directory as 0700. Narrowing permissions after creation would leave a window in which another account could open the file, so the file is opened with these permissions from the start.
• While it is being written, the name ends in .partial. It gets its real name only after writing finishes and the dump format header (PGDMP) checks out. An interrupted file never shows up as a backup.
• Before anything in the installation changes, the updater checks for 1.2 times the database size free in the backup location. If it is short, the updater stops right there and the current version keeps running.
• If the app's database is not in the postgres container of the same compose project, the updater stops without a backup. On an installation that uses an external database, dumping that container would back up a database the app does not use and let the update carry on. Such installations back up the database themselves and update by hand.

The dump holds the database only. Updates do not change the object storage that holds documents, meeting audio and sandbox files (“Files that must survive a redeploy live only in object storage”), so the database is all that is needed to return to the previous version. Files uploaded after the update stay in storage as files the restored database does not know about. Before approval, the confirmation dialog states this scope: "File storage such as documents and generated files is not included in the backup."

#The failed step and the schema change decide the end state

The updater runs the apply as a series of steps, from signature verification to the startup check (the step list is in “The update button only approves, and only the updater holds the docker socket”). The end state splits four ways, depending on which step failed and on that version's schema change.

If it fails before the images are loaded, nothing in the installation has changed. The signature, upgrade-condition, file copy, disk, unpack and backup steps fall here. The current version keeps running; fix the cause and approve again.

If it fails after loading but before the services are swapped, the updater restores the image pin file, the install kit and the settings file from their pre-update copies. The containers are still on the old version. If the worker had already been brought down (“Worker restarts: every running turn answers or records its failure”), the updater redeploys with the old pins to bring it back. The schema has not changed yet, so the old code comes up cleanly.

If it fails during the service swap or the startup check, the updater looks at the version's schema change. The signed manifest carries "the oldest version that code alone can roll back to." If this version is expand and the current version is within that range, the updater restores the files, redeploys with the old pins and runs the startup check once more. The end state is Rolled back. The schema stays on the new version, and the old code runs on top of it.

For a contract version, that manifest value is the version itself, so no older version falls within range. The updater restores only the files and stops without redeploying the containers, because starting the old code on the new schema produces the errors described at the top. The status keeps the dump path and the name of the app database that was dumped. At the top of the admin console, a banner that cannot be closed reads "Stopped because this update cannot be rolled back automatically. Restore the database from the backup taken just before the update." It goes away only when the next update is approved.

There is one exception. If the deployment script reports through its exit code that it stopped before changing any container, as when the pre-install check or the image pull fails, the schema is untouched too. In that case the updater redeploys with the old pins to bring the worker back, even for a contract version.

What the startup check looks at to call a failure is covered in “Deploy success is judged by failed turns after the deploy, not by health checks”. Even when the containers came up fine, failing that check counts as a failure of this step.

If a reboot or a Docker engine restart cuts off the deployment helper, that does not count as a failure right away. The deployment script gives the same result when run again, so the updater runs it once more and carries on. If the rerun is also cut off or fails, the rules above apply.

#Restore from the server, and restore the database before deploying

When the updater stops because it cannot roll back on its own, a person restores from the dump. The admin console gives the guidance as server commands, not a button, because the service is half-changed and the console cannot be counted on to load. Even when it won't open, tools/ops/update.sh in the install directory calls the same updater. Status, per-request progress logs and rollback all use the same state file as the console.

The restore goes in this order.

1. Read the dump path and the name of the dumped app database with tools/ops/update.sh status. The app database name can differ between installations.
2. Copy the dump out of the updater volume.
3. Stop the backend, worker and frontend.
4. Drop and recreate the app database, then run pg_restore as the schema owner account.
5. Deploy with tools/ops/restart_stack.sh deploy. The updater has already restored the pin file to the old version, so the old version comes up, and the worker that was down comes up with it.
6. Start the updater again with tools/ops/restart_stack.sh updater.

The one thing this order must not get wrong is running step 5 before step 4. The pin file already points to the old version, so deploying first starts the old code on the new schema. If you would rather not lose the writes made after the update, you can instead fix the cause of the failure and approve the same version again.

tools/ops/update.sh rollback serves a different purpose from this restore. It rolls the code back to the previous version when you find a problem in the new version after a successful update. It works only when the last update succeeded, that version is the current one, and it is within the range where code alone can roll back. Otherwise it refuses and points to the dump path. The rollback uses the pin and kit copies that update left behind, and those copies are deleted when the next update request comes in. The updater itself stays on the new version. It can judge old manifests too, so there is no reason to roll back its own bug fixes.

The confirmation dialog shows this verdict before approval: "If something goes wrong, you can roll back to the previous version right away." for a version where code alone can roll back, and "To roll back, you need to restore the backup taken just before the update." otherwise.

#Design choices

We chose returning to a single dump taken just before the update over down migrations. Down migrations have a real advantage: they can keep the writes made after the update. In exchange, every version needs a reverse script written and tested, and even then the values in a dropped column do not come back. Restoring a dump discards the writes made after the update, so we have the startup check catch failures right after apply, which keeps the discarded writes to that short span.

We keep only the two most recent dumps. An update dump's job ends at leaving a way back to the previous version; long-term retention and file backup belong to the customer's regular backups. Dumps don't keep eating disk version after version, and the cost is that if an older dump is needed, the operator has to move it before approving.

If waiting for the deployment helper times out, the updater rolls back nothing. The helper may still be changing containers, and starting a rollback then would have the changing side and the undoing side modify the same installation at once. The console shows "Could not start the rollback because an earlier task has not finished.", and the operator waits for the helper to finish and decides from its log. That is one fewer automatic recovery, and in return a half-changed installation never gets half-overwritten again.

A product that updates itself on customer hosts can adopt this split as is. Record for every release whether the schema change can run alongside the old code, take the dump regardless of that value, and use that value to decide whether to roll back the code automatically.