Engineering NotesInstallation, operations, and quality
Code, not people, picks the files in an install kit

An on-premises product keeps the files customers receive and the files that stay in-house in the same repository. The production compose and installation docs go to customers; internal test environment settings and internal build procedure docs stay behind. When a person picks directories and tars them up, it goes wrong in two directions. A .env from the working tree, or a doc with an internal server address, gets packed and reaches every customer. Or a script that the deploy script calls is left out, and nobody finds out until an install or upgrade stops on the customer's host. Neither shows at packing time.
The install kit for on-premises OwlAI (the compose files, operations scripts and docs that air-gapped customers receive next to the image bundle, “Deploys and air-gapped loads are checked by digest, not by tag”) is selected by code. Only git-tracked files are candidates. Rules with written reasons decide what goes in, and the build stops if a rule selects nothing or if a doc has not been classified. Every selected file then goes through the same internal-coordinates check that runs on every merge request. This post explains what each of these four pieces blocks.
Packing whole directories means remembering what to add and what to leave out
The common approach is to pack the directories you need and list files to exclude. Nobody has to name each file, and a new script is picked up automatically. But it only works while someone remembers what to exclude. If someone creates a file with connection details in the working tree and never adds it to the exclusion list, it ships in the next kit. Internal operations docs that live in the same directory ship too, until each new one makes it onto the list.
Missing files follow the same pattern. Move a script to a new path without updating the packing command, and it disappears from the kit without a sound. The internal test environment deploys from a full checkout, so it never notices. The gap first shows on a customer host that has only the kit, at the moment a deploy script calls the missing file.
Only git-tracked files are candidates
The kit builder takes its candidates from git ls-files. A .env, a local config.yaml or an environment file with connection details that exists only in the working tree is never a candidate. Whatever the person who created a secret file remembers to do, the tool never sees it.
If any file bound for the kit has uncommitted changes in the working tree, the build stops, because the kit's contents must match the commit's. On the development branch as of November 3, 2026, 32 of the 6,722 tracked files go into the kit, plus three files the tool generates: a file list, a checksum file and a README.
The release pipeline builds the kit before the image bundle. The kit takes a few seconds, so if a check fails, the pipeline stops before it bakes a multi-gigabyte image bundle.
Rules decide what goes in, and the build stops when they can't
Three lists decide which tracked files go into the kit. Every entry in all three carries a one-line reason.
| List | Entries | What it does |
|---|---|---|
| Include rules | 29 | Select kit files by path or pattern |
| Internal operations docs | 10 | Name the docs in the operations docs directory that customers do not get |
| Deny list | 6 | Files no rule may ever put in the kit. If an include rule selects one, the build stops |
The include rules cover the production compose, .env.example and config.example.yaml, the deploy, check, image-loading, backup and update scripts, the script isolation profiles (AppArmor, SELinux, seccomp), database initialization SQL, and installation and operations docs. Files keep the repository's layout. The deploy script and the preflight check (“The preflight check only reads the host and leaves the fixes to a person”) treat the directory two levels above themselves as the install directory, and the production compose mounts the initialization SQL and the seccomp profile by relative path.
Operations docs go in one by one by name, not by pattern. Of the 20 docs in the operations docs directory, 10 are named in the include rules and 10 in the internal operations docs list. When a new doc arrives and neither list names it, the build stops. So an internal doc can't reach the kit without passing through that classification, even one with no coordinates in it.
The deny list holds internal-only files, such as config with internal coordinates and the internal test stack definition, plus secret and connection-detail files at any path. If this list were only a promise not to include those files, it would break unnoticed the moment an include pattern widened. So it is defined as a failure condition: if one of these files gets in, the build fails.
Missing files are caught twice. If any include rule selects no files, the build stops; that is what happens when a file moves and its rule isn't updated. Unit tests also check that every script the deploy, check and update scripts call, and every common file the host setup scripts source, is in the kit. They catch the gaps a test environment deploying from a full checkout can't see.
Each entry has a reason so the next person can tell whether the rule is safe to remove. "This doc looks harmless, let's include it" is easy to get wrong without knowing what the rule's author was worried about. The reason records what changes if the file reaches a customer.
Kit files are checked again as customer deliverables
The repository has a separate check that keeps internal coordinates (internal IP ranges, internal domains, internal database names) out of customer deliverables, and it runs on every merge request. Internal values get in because they are convenient at the moment: an internal GPU server address in an example config, an internal account as a default. One cleanup doesn't stop the next change from adding them back, so every change is checked.
Normally that check skips the operations scripts and docs directories. Many files there are internal-only, and internal coordinates in them are expected. But once a file from those directories goes into the kit, it is a customer deliverable. So the two tools call each other. The kit builder loads the coordinates check's rules and allowlist and applies them to every kit file. In the other direction, the coordinates check calls the kit builder's file selection and adds the kit's files to its usual targets.
Because of that link, a change that writes an internal address into a customer-facing operations doc fails on its merge request, not at release time. The list of kit files lives only in the kit builder, so the two checks never look at different files.
Only records of the basis for a decision go on the allowlist: a measurement source or regression case that explains why a value was chosen. Deleting one would erase the basis, so it is registered per file with a reason and let through. A growing list is itself a signal to review. Each new entry raises the question of whether it really is a basis record or a new internal value.
Database initialization SQL gets one more check. These scripts run as-is when PostgreSQL first starts on an empty volume. A statement that creates a superuser or a role with a baked-in password would create the same account in every customer installation. The kit builder stops the build when it finds such a statement in an initialization script. On an installed system, the preflight check looks at the same risk again: it reports FAIL if any login-capable superuser exists besides the metadata database's admin role (“The preflight check only reads the host and leaves the fixes to a person”).
Customer deliverables outside the kit pass the same check. Images in the image bundle are named localhost/owlai/<image> instead of carrying the internal registry address, and the sealing step strips the annotations in which the image store records the registry it pulled from. Release metadata for the public feed is published only after it passes the same internal-coordinates check, with no allowlist applied, because published metadata has no place for basis records.
The same commit produces the same bytes
The kit builder pins every tar entry value that would otherwise depend on the build environment.
| Value | How it is pinned |
|---|---|
| File order | Sorted by path |
| Modification time | The commit time for every entry; the gzip header also carries the commit time |
| Owner | uid and gid 0, user names empty |
| Permissions | 755 if the executable bit is set, otherwise 644 |
Two kits built from the same commit are therefore identical down to the byte, and a unit test checks it. To confirm that a kit a customer received came from a release commit, rebuild from that commit and compare.
The kit also contains a file list (MANIFEST.tsv: path, permissions, size, sha256 and source commit), a checksum file (SHA256SUMS) and a README. After transfer, customers confirm the files are intact with a single sha256sum -c SHA256SUMS. Because the kit is a customer deliverable, the files the tool generates are in English.
Design choices
We chose to add operations docs by name rather than by pattern. A pattern would need no edits when a customer doc appears, but it leaves a window in which an internal doc reaches customers before anyone classifies it. Naming each doc means editing a list every time someone writes an operations doc. That chore is the cost, and in exchange the person who wrote the doc decides on the spot whether customers get it.
The internal-coordinates check matches coordinates rather than product names. Domains, IP addresses and internal database names have a fixed shape, so the code reaches the same verdict every time. Product and engine names show up in legitimate code all the time; banning them would flag that code, swell the allowlist and turn the check into a list people ignore. Shapeless information, such as people's names in docs or internal matters described in prose, is left to code review, and keeping it from leaking depends on how carefully reviewers read.
Instead of writing the coordinates rules a second time on the kit side, we had the two tools call each other. Rules written in two places drift apart sooner or later, and the side that drifts gets looser without anyone noticing. When the tools call each other, changing one changes the other. The cost is that each tool has to know where the other lives, and moving one file makes the other fail to load it and stop at once.
If your on-premises product keeps customer deliverables and internal artifacts in one repository, this setup carries over as is. Start by narrowing candidates to git ls-files and by stopping the build on any rule that selects nothing. Both take a few lines of code, and you learn what slipped into the kit or fell out of it at build time instead of on a customer's host.
More in Installation, operations, and quality

Engineering Notes · Installation, operations, and quality
The preflight check only reads the host and leaves the fixes to a person
On-premises OwlAI runs as a Docker Compose stack on a single host the customer picks.

Engineering Notes · Installation, operations, and quality
People set the Docker address pool, and the check compares it with internal ranges
Install on-premises OwlAI on a host inside a corporate network, wait a few weeks, and one day an internal data…

Engineering Notes · Installation, operations, and quality
Behind a corporate proxy, list NO_PROXY hosts by name and IP, not CIDR
When you install on-premises OwlAI on a corporate network that can reach the internet only through a proxy, ca…