The update button only approves, and only the updater holds the docker socket

Put an update button in the admin console of an on-premises product, and something has to stop the containers and start them again on new images after the click. If the web server does that itself, it needs the docker socket. A process holding the docker socket can start a privileged container with host paths mounted, so it is effectively root on the host. In that setup, one vulnerability in the admin console or one stolen administrator session is enough to take over the host, and an attacker's own image can be installed as an "update."

OwlAI on-premises therefore gives approval and application to different containers. The web server writes one request file with the version the administrator approved and the hash of the manifest the confirmation dialog showed. Only the updater, a container with no network, holds the docker socket. The updater does not take the request at its word: it re-judges everything from the signed manifest before applying. This post covers that permission boundary, the order of checks, the steps that resume after an interruption, and the migration rule that lets every installation go straight to the latest version.

#Give the web server the docker socket and a console bug becomes host root

There are two common approaches. The first mounts the docker socket into the web server, or into an update container that takes web requests. It saves a component and keeps the implementation simple. But the web server is the component that takes requests from the whole internal network. Mounting the socket read-only (:ro) does not stop docker API calls, so any vulnerability in the web server turns straight into host root.

The second leaves updates to a person who logs into the server and runs commands. There is no permission problem, but it misses what customers ask for, which is to finish with one button, and it leaves room for mistakes such as getting the version order wrong or skipping the backup.

A process that holds the socket is root, and that cannot be changed. So we made exactly one container hold the socket and narrowed what that container accepts.

#The web server hands the updater one request file

When an administrator clicks Update in the confirmation dialog, the web server first re-authenticates them: it asks for the current password, plus an authenticator code if the account has two-factor authentication on. After that, all the web server does is write one request file. The file has exactly five fields:

• The version to install (X.Y.Z form)
• The sha256 of the manifest the confirmation dialog showed
• How the release file arrives (online download or file upload)
• A request ID
• Who approved it

If the file has any field the updater does not know, the updater rejects the request. There is no field for a command string or a path, and the version is used only to pick the delivery directory name.

The request location is a separate volume from the updater's state, so the web server cannot reach the updater's backup dumps or working copies. The only other way the web server reaches the updater is a Unix socket that accepts status reads and license file replacement. A test that reads the compose file checks that no service other than the updater mounts the docker socket, the signing trust keys, the installation directory or the updater's state.

Only one update is accepted at a time. If two administrators approve almost simultaneously, the second approval is rejected with "Another update has not finished", and the updater also rejects new requests while one is running.

The updater container runs with no network (network_mode: none) on a read-only root filesystem. Online downloads are therefore handled by the worker, which puts the release in the release storage (“Files that must survive a redeploy live only in object storage”). A release file (owlai-<version>.owlai) uploaded through the console on an air-gapped installation lands in the same place. The updater reads that location read-only.

#The updater re-judges everything from the signed manifest

Each release ships a JSON file called the manifest, and its raw bytes carry a detached Ed25519 signature. The manifest holds these values:

Signing happens only in pipelines for protected release tags; builds on ordinary branches cannot reach the signing key. The updater checks the signature against public keys baked into its own image and does not trust any key shipped inside a delivered file.

The updater checks in this order:

1. It reads the manifest and signature once, verifies the signature against those bytes and keeps them as a copy. If the manifest's sha256 differs from the value in the request, it stops. It will not apply a manifest other than the one the administrator saw in the confirmation dialog.
2. It reads the version label of the running image and decides whether the target version can be installed from there. The same or an older version, an installation below the oldest direct-upgrade version, a waypoint version in between, or a pending manual step are all rejected, with the reason sent back to the console.
3. It reads the image bundle and installation kit once, hashing them while writing the same bytes to a working copy. The delivery location is writable by the web server, so its contents could change during verification. From here on the updater never reopens the originals and unpacks exactly the bytes whose hash it checked.
4. When unpacking the copy, it accepts only regular files and directories at relative paths, and stops if it finds a link or device file. A signature only says the files came from us, so what the root process doing the unpacking will accept is decided separately.
5. It compares the image digests in the bundle's table with the signed values in the manifest. The check of image IDs after loading is covered in “Deploys and air-gapped loads are checked by digest, not by tag”.

The signature covers the manifest, the manifest covers the bundle hash and image digests, and the digests pin the loaded images. Even if a compromised web server writes whatever request file it likes, an unsigned version cannot get past step 1.

The confirmation dialog uses the same signed values. The impact notice is the manifest value as is, and the rollback notice is computed by the web server with the same rule function the updater uses.

#Fifteen steps, resumed from wherever they stopped

Once the checks pass, the updater runs these steps in order. The console's In progress panel shows them under the same names, with the time each one took.

1. Download binary or Upload binary (confirm the release file has fully arrived)
2. Check signature
3. Check upgrade path
4. Copy and verify files
5. Check disk space
6. Unpack files
7. Back up database
8. Load images
9. Replace installation files
10. Convert configuration file
11. Wait for running jobs
12. Replace services
13. Check startup
14. Clean up
15. Update the runner

The installation itself changes only from step 8. If the update stops before that, the current version keeps running. Step 7 dumps the metastore with pg_dump every time, regardless of the manifest, and keeps the latest two. What that dump is used to restore is covered in “When an update fails, the schema change decides between rolling back code and restoring the dump”, and how step 11 finishes and stops running jobs is covered in “Worker restarts: every running turn answers or records its failure”.

The updater writes its state file after each step under a temporary name and renames it, so each update is atomic. If a reboot or power loss cuts the process, the next start resumes from the recorded step, which is why every step gives the same result when run again. The copies used for rolling back and the list of files to overwrite are fixed once, the first time. The helper container that swaps services is named after the request ID, so if a container with that name already exists, the updater waits for its result instead of starting another.

Service swaps are done by a one-off helper container. The helper runs tools/ops/restart_stack.sh deploy from the installation directory in the host namespace, so it goes through the same script and the same pre-install check (“The preflight check only reads the host and leaves the fixes to a person”) as a manual deployment. The helper runs one of four subcommands, deploy, startup judgment, old-image cleanup or updater replacement, and the request cannot choose among them.

An update cannot be stopped from the console once it starts. Stopping mid-swap would leave a half-changed state, so the updater either finishes or rolls back on its own. The updater replaces itself in the last step, and only after the update has succeeded. Replacing itself first would cut its own process off in the middle of applying.

#Going straight to the latest version means migrations cannot create waypoints

Updates from the console always go to the single latest version, but one customer updates every release and another updates once in six months. Squashing the migration revision chain into a single baseline forces any installation older than that boundary to install a waypoint version first. So metastore revisions stopped being squashed after the 0.6.0 baseline and only accumulate, and any version from 0.6.0 on goes to the latest in one step. Revisions are written to run on the database of any earlier version, with idempotent statements such as IF EXISTS, and data conversions are fixed logic inside the revision. If someone changes the models and forgets a revision, a CI check that applies every revision to an empty database and compares the result with the models blocks the merge.

Databases older than the last squash boundary are the one exception. The backend keeps the list of revisions that squashing removed and refuses to start on a database at one of them. If it read an unknown revision as "a schema from newer code" and skipped ahead, the service would come up on the old schema. The updater also rejects such an update at step 2 with the reason that a waypoint is required, and the administrator installs the waypoint version first and then the latest.

One value is written by a person for every release: whether the revisions since the previous release are expand changes that old code can run alongside, or contract changes it cannot. Dropping or renaming columns or constraints, adding NOT NULL, CHECK or foreign keys, or moving values that old code reads all make a release contract. The judgment is recorded together with the last revision it covered, and if more revisions land before the release is tagged, manifest generation stops before signing. This value decides the rollback notice in the confirmation dialog and what the updater does on failure, which “When an update fails, the schema change decides between rolling back code and restoring the dump” continues.

#Design choices

Instead of giving the web server the socket, we added a separate updater that holds only the socket. That is one more component, and because the updater has no network, downloading falls to the web server. In return, receiving and judging are split, so a compromised receiver cannot skip the judgment.

The updater keeps only the two container capabilities needed for file ownership, and its input is closed down to a single five-field request file. For a container that holds the socket, narrowing what it accepts sets the limits of what it can do more than trimming its capability list does.

When a version has manual steps the console cannot perform, the confirmation dialog locks approval and points to the documentation in the installation directory. We chose not leaving a half-applied state over a one-button experience.

Because the helper runs the deploy script in the host namespace, the reference runtime is rootful Docker. On rootless Docker or podman, a person updates by unpacking the kit and running the same restart_stack.sh deploy by hand.

Replacing the updater last, and only after success, also has a cost. If the updater itself has a bug, a console update cannot fix it: you first swap in the updater from the new version's bundle by hand, then approve the same version again. We judged that one procedure safer than an updater that might cut itself off mid-update.

If your installable product updates itself on customer hosts, this split carries over directly. Make one process hold the socket, close what that process accepts down to a fixed shape, and let a signed file decide what gets applied.