Deploys and air-gapped loads are checked by digest, not by tag

On-premises OwlAI runs on the customer's host as nine container images. A test result only carries over to a customer install if the image that passed in the test environment and the image running on the customer's host are the same bytes. If you pull images by a tag such as backend:1.4.2, you can't claim that, because a tag can be moved to a different image. Air-gapped networks with no registry are harder still. On Docker 29, docker load accepts a bundle with missing or corrupted blobs, exits with code 0 and leaves an image record behind.

So CI builds each image once, and every deployment path uses the one set of digests (the sha256 hash of the image manifest) returned at push time. Deployments that pull from a registry accept only digest references. The air-gapped bundle carries the bytes behind those digests unchanged, and after loading, the load script compares each image's Id and creates a container from it. This post explains how a digest travels from the build to the customer's host, and what an air-gapped load has to check before it counts as done. Host checks before installation are covered in “The preflight check only reads the host and leaves the fixes to a person”, and updates from the admin console in “The update button only approves, and only the updater holds the docker socket”.

#Pull by tag and a different image runs under the same name

There are three ways different content can run under the same tag.

Turning on immutable tags in the registry (such as tag immutability in Amazon ECR) closes the first path with a single setting. But that setting only holds inside that one registry, every customer has a different registry, and an air-gapped install has no registry at all. The third path stays open even with tags locked. A rollback to an old tag also fails to bring back the old release if the tag was overwritten in the meantime. So we pick what to deploy by content, not by name. When you pull by digest, the pull fails if the registry serves different content, so a different image cannot run under the same digest.

#CI builds once, and the digest comes from the push output

Deployment hosts do not build images. The production compose file has no build: blocks, and every image field is a variable of the form image: "${OWLAI_IMAGE_BACKEND:?…}", so compose stops and names the variable when it is empty. CI builds eight of the nine images itself, including the metastore PostgreSQL image, which adds pgvector on top of the official image. The remaining one, an open-source file storage gateway, is pulled from a source pinned by release tag and digest, copied to our own registry, and deployed by the digest of the copy. That way a deployment does not change when upstream pushes the same tag again.

The build job reads each digest from the push output. Asking the registry for the tag's digest after the push could return a value from a tag that was overwritten in between. The job writes each value to a digest file as a <registry>/owlai/backend@sha256:… reference, together with the commit SHA it built. It builds all nine images before pushing any of them, and pulls each digest once before writing it down. A build that breaks on the fourth image does not leave the first three published.

The deploy job moves this file into the host's image pin file (.env.images) and checks three things on the way.

• A reference not of the form <repository>@sha256:… is rejected. Tag references cannot get in.
• The commit in the file must match the commit of the checkout (compose, configuration and scripts). This stops new images from running with an old compose file.
• If a CI variable with the same name holds a different value from the file, the job stops and names that variable.

The third check exists because of GitLab CI's variable precedence. A variable of the same name defined on the project or the pipeline takes precedence over a job artifact's value, so the digest this pipeline built could be replaced without any warning. So deployments read image references from the file only.

Tags remain only as names for people to look things up. A release tag (such as 0.6.7) is attached to the same digest only after that commit passes gate re-verification and the image security scan.

#The air-gapped bundle carries the registry's bytes unchanged

The import bundle (owlai-images-<version>.tar) starts from the same digest file. The job that builds it does not rebuild images; it exports the images it pulled by digest as an OCI image layout. A layout is a directory in which every blob is named blobs/sha256/<digest>, so the receiving side can check each blob against its name. Layers shared by several images are stored once. The accompanying table (manifest.tsv) lists, for each image, its name, registry digest, config digest and the variable compose reads. Loading is done by a script in the install kit of the same release.

The job that builds the bundle checks that the digests recorded in the layout match the registry digests, and does not publish the bundle if they differ. The signed release manifest reads its values from the same digest file, so the registry, the bundle and the manifest all state the same digests.

Images in the bundle are named localhost/owlai/<image>. A name with no registry part, such as owlai/backend, is shorter, but if the image is missing it resolves to the same name on Docker Hub. On a host with partial internet access, that could pull and run someone else's image. localhost/… does not resolve to any public registry, so a missing image stops the deployment with a pull failure.

#docker load exiting with 0 does not mean the load is done

On an air-gapped host, tools/ops/images.sh load from the install kit loads the bundle. It runs on bash alone, with no git or python, and goes through five steps in order.

1. Verify the checksums of the table and the layout's top-level files.
2. Compute the sha256 of each blob's content and check it against the file name. Reject links.
3. Load with docker load.
4. Compare each loaded image's Id with the value in the table.
5. Create a container from the reference that will be pinned (docker create --pull never) and delete it right away.

Each step catches something different. Step 2 catches blobs that changed in transit, but a blob that is missing entirely slips past, because there is no file to hash. docker load accepts such a bundle with exit code 0 and leaves an image record, so a missing blob shows up in step 5, when the container cannot be created. Step 4 catches a broken load, or a different image left under the same name. Links are rejected because hashing reads the link's target while the tar handed to the engine carries the link itself, so the bytes that were verified and the bytes that were loaded would diverge.

Once all five steps pass, the script writes the pin file compose reads. What it pins to depends on the host's image store. The script detects the store with docker info and records which one it found in the log and in the pin file's header.

The containerd store keeps the registry digest as the Id after docker load, so the pin uses the same digest as the test environment. This store resolves a digest reference only when a tag exists for the same name, so the loaded :<version> tag must not be deleted. On the classic store, docker load loses the registry digest and there is no way to form a digest reference. So the pin is a tag, and the content is confirmed by the earlier blob hashes and the config digest comparison. Because the name starts with localhost/, there is also no way for an external registry to overwrite that tag.

Loading the same bundle again still repeats the Id comparison and the container creation. Before replacing any container, the deploy script checks that every pinned image is present, and if one is missing it stops without changing any container.

#Rollback uses the digests kept in the pin file

Each time the pin file is written, the previous pin goes to .env.images.prev, and every pin is appended with a timestamp to .env.images.history. Rolling back one step takes two lines.

cp .env.images.prev .env.images
tools/ops/restart_stack.sh deploy

A digest is an address determined by content, so rolling back runs an image identical, byte for byte, to the one that ran then. This procedure rolls back images only and leaves the database schema as it is. That is why the official update path for customer installs uses a separate rollback procedure that takes a database dump before the update (“The update button only approves, and only the updater holds the docker socket”). A digest that the registry's retention period has already removed cannot be pulled, and the deployment host also cleans up images other than the current and previous pins after each deploy. An air-gapped install loads that release's import bundle again.

#Design choices

Instead of building a separate bundle for the import environment, we chose to export the registry's bytes unchanged. A separate build could tailor images to the import environment, but rebuilding even the same Dockerfile changes the digest, and the evidence that the tested image and the imported image are the same is gone. The cost falls on the side that builds the bundle. On the classic store, docker save unpacks and rewrites layers and no registry digest survives, so bundles can only be built on a build host that uses the containerd store.

Instead of unifying pins on tags, we split pinning in two to match the image store. Unifying would leave a single procedure, but it would throw away the digest the containerd store preserves. The cost: if a customer host switches its store from classic to containerd, the earlier images disappear from view and the bundle has to be loaded once more.

Digests and signatures answer different questions. A digest fixes, byte for byte, what runs; the signature on the release manifest proves that the image is a release we published (“The update button only approves, and only the updater holds the docker socket”). Because no single mechanism has to answer both questions, each can be verified on its own.

Choosing deploy targets by digest rather than by readable tags has a cost too. To roll back, you look up the digest in the pin file's records. If you can't fully control write access to your registry, if you build the same commit separately for several environments, or if you have to move images into a network with no registry, that cost is worth paying.