The preflight check only reads the host and leaves the fixes to a person

On-premises OwlAI runs as a Docker Compose stack on a single host the customer picks. The distribution, kernel, Docker version, security modules and certificates differ from one customer to the next. When dockerd won't start after compose up on install day, or every document-generation run is refused, or the desktop app can't connect, the first job is to work out whether the host or the product is at fault. The symptom shows up in the product, but the cause is usually a host condition unrelated to the install command, one that could have been fixed the day before.

This post covers the preflight script that moves that verdict to before the install. It only reads the host, and for each item it prints one line with a pass, warn or fail verdict and a command to fix it. The same check runs again before containers are replaced on every deploy and update, so host conditions that change after the install are caught before the next deploy.

#The symptom shows up in the product, the cause is on the host

The table lists conditions that stop an install, the symptom you see on the day and the check item that catches them. In every row, the symptom and the cause sit in different places.

The kernel floor for RHEL 8 is 4.18.0-425 (8.7); 8.10 is recommended. “Sandbox each script run, then prove isolation on the actual server” explains why isolation depends on the kernel and the security modules.

Certificate problems only appear when someone first uses a feature. The browser microphone is available only in a secure context (HTTPS or localhost), so on a plain-HTTP IP address the voice input button does not appear. A self-signed certificate is enough for the microphone, but not for web push. Chromium fails a service worker registration when fetching the script hits a certificate error, so the user's PC has to trust the certificate's CA. The desktop app does not ignore certificate errors either, so a server signed by an internal CA needs that CA in the Windows "Trusted Root Certification Authorities" store. The check therefore looks at the certificate's expiry (a warning from 30 days out), whether the key matches, whether the SAN contains the address users actually type, and whether it is self-signed.

#Install docs can't tell you the state of a host

If the install docs list these conditions in a requirements table and nothing more, the installer has to run commands to find the kernel build number or the cgroup delegation and compare each one against the table. That comparison happens once, on install day. When Docker or the kernel is upgraded later, or the certificate expires, the docs say nothing.

Another approach is an install script that fixes what it finds: edit daemon.json when address pools overlap, create the shared network when it is missing, change a sysctl value when it is wrong. It saves the installer work. But fixing needs context about that host. Only someone who knows the corporate network can decide which range the Docker address pool should move to, and the people who share the server decide when it reboots after a kernel update. When a customer's security officer asks whether it's safe to run the preflight, the answer has to be "it only reads." A single fixing step anywhere makes that answer false.

So the check reads the host and gives a verdict, and the fixing commands go to a person.

#The check only reads the host and prints the fix

The script writes no files, creates no networks, and changes no sysctl or daemon settings, so running it any number of times gives the same result. Fix commands that need root are run by a person as root. Run as a non-root account, checks that only root can read (the loaded AppArmor profiles, the SELinux module list) come back as skipped. One run as the install account and one as root leave no gaps.

The one exception is the item for script-execution isolation (studio.isolation). Whether isolation works depends on the combination of kernel and security modules, and reading settings can't tell you. So this item starts a throwaway container from the loaded sidecar image with the production security options and builds the isolation once inside it. The container has no network and no host-path mounts, and it does not pull images. The volume it uses for the write check is also throwaway, and when it finishes, in every case including a timeout (90 seconds) or an interrupt, the container and the volume are both deleted. On a first install, when the sidecar image is not on the host yet, this item is skipped and the health check decides after the deploy.

The check does not print secret values, and for .env it names variables only, so you can paste the output straight into a support ticket. When a tool like openssl or ss is missing, only the items that need it are skipped.

#Each item gets one verdict line and a fix command

There are more than seventy items, covering the distribution and kernel, the Docker engine and Compose, networking and proxies, the install directory's .env, config, images and certificates, and the state of a stack that is already running. Each item prints one line with one of four statuses.

• PASS: passed
• WARN: works, but is risky or outside the tested combinations
• FAIL: do not deploy like this
• SKIP: no way to judge (a required command or permission is missing)

Any FAIL makes the exit code 1. Items with something to fix get a fix: line. An install on a RHEL 8.6 host with a self-signed certificate looks like this:

[FAIL] kernel.el8               RHEL/Rocky 8 kernel 4.18.0-372.9.1.el8.x86_64 is older than 4.18.0-425 (8.7): studio's sandbox fails on 8.6 ('bwrap: Can't open source /usr: Function not implemented')
                                fix: update the kernel to 8.7 or newer (8.10 recommended; a host pinned to 8.6/EUS must move to a later minor release) and reboot
[WARN] tls.self_signed          self-signed certificate: every browser and the desktop app must trust it (web push refuses certificate errors)
                                fix: distribute the CA to clients (Windows: Trusted Root Certification Authorities)

Item IDs are fixed dotted names, and the per-distribution table in the install requirements doc lists the same ID for each requirement. One ID takes you from a check result to the doc, or from the doc to the check.

The thresholds are constants at the top of the script, each with its reasoning. For disk, 72 concurrent turns (a turn is one question-and-answer cycle, from receiving a question to producing the answer) each holding the 512 MiB per-conversation storage cap, plus 5 GiB of images and headroom, gives the 45 GiB FAIL threshold. The worst case, with every turn folder grown to 1 GiB, gives the 80 GiB WARN threshold.

#Deploys and updates run the same check before replacing containers

The deploy script runs this check before it replaces a single container, and on any FAIL it stops without changing anything. A missing secret, a directory where config.yaml should be, a port conflict or a Docker network that overlaps a corporate range would leave a half-replaced stack if it surfaced mid-replacement. Stopping before replacement leaves the previous version running. A WARN does not block the deploy; it goes to the deploy log. Updates applied from the admin console call the same deploy script in their apply step, so every update runs the same check first.

Inside a deploy it runs with --offline, which skips DNS and registry lookups. Pulling images is the very next step, and an air-gapped install has no registry to query.

Some conditions change between updates: the internal CA or the proxy is replaced, the clock drifts, or a Docker upgrade lowers the file descriptor limit. The backend and worker check these at startup and log warnings. Most are a risk that some call might fail someday, not a failure that is certain now, and blocking startup would trigger repeated restarts that hide the cause further. Only certain failures stop startup with the reason logged, such as a NO_PROXY value that makes it impossible to create any external HTTP client.

#Design choices

We chose read-only over automatic fixes. The cost is that the installer has to read the fix: lines and run the commands. In exchange, a security review can describe the check in one sentence ("it only reads"), and it changes no host settings no matter how many times deploys and updates run it. Script-execution isolation, the one thing reading can't judge, is the only item that starts a throwaway container, and that container and its volume are deleted in every case.

We chose to pass only tested combinations. On RHEL 8, kernels below the floor we established by installing on VMs (4.18.0-425) are a FAIL, which includes the untested builds between the 8.6 stock kernel and the floor. A non-stock kernel or an untested Docker version (25 to 28) gets a WARN. Passing unknown combinations would let more hosts start an install, but those combinations break on install day. The images are built for x86-64 only and signed, so what customers receive is what we tested. That is why ARM64 hosts, and FIPS-mode hosts (the crypto modules in the images are not FIPS-validated), show up as a FAIL on the first screen of the check output instead of breaking at first startup.

The check judges known failure conditions, and health checks after the deploy catch the failures nobody has seen yet. The script-execution sidecar builds its isolation with the same command a real run uses and reports the result in its health check (“Sandbox each script run, then prove isolation on the actual server”). So when a combination the check doesn't know blocks isolation, the deploy verdict still shows it.

If you ship a product that installs on hosts your customers pick, a good place to start is one check item per requirement in your install docs, with the deploy script running that check again before it replaces anything.