Files that must survive a redeploy live only in object storage

When someone records a meeting, the browser uploads a chunk of the recording to the web backend every 5 seconds, and a worker in a different container transcribes those chunks. If the web backend writes the chunks to its own container disk, the worker cannot see them. Attaching the same host volume to both works on-premises, but on OwlAI SaaS containers are replaced on every deployment and do not share disks. Original documents, reports the model creates, and conversation attachments follow the same path: one container writes, another reads, and the file has to outlive both containers.

So we set a single rule. Any file that has to survive a redeploy or container recreation lives only in S3-compatible object storage, and container disks hold only what can be wiped and rebuilt. On-premises and OwlAI SaaS run the same S3 API code; only the storage address and credentials differ. This post explains how we split the buckets to keep that rule, and how we filled in for the S3 features we gave up by using only what both stores support.

#Container disks and shared volumes fit only one deployment form

There are three common places to put files, and each breaks under some condition.

Another option is a fallback to a local directory. It has a real upside: the service keeps running while object storage is briefly unavailable. But an installation missing its storage settings would pile files onto the container disk without any warning and lose all of them at the next update. Only then would anyone learn the settings were missing. So the backend and workers check the object storage coordinates at startup and refuse to start without them. The on-premises compose file also refuses to start any service when the access keys are empty.

#Both editions point the same S3 API at different stores

The backend and workers read and write files through one S3 API. On-premises, they point at an open-source S3-compatible gateway running on the same host. The gateway uses its POSIX backend, so an object key is a file path on disk. On OwlAI SaaS, the same code points at AWS S3.

The two editions differ in four settings in .env.

The code that reads and writes files, the key rules, the cleanup jobs, and the retry policy are the same in both editions. Retries cover network errors, 5xx, and rate-limit responses, with up to 4 attempts at short intervals. The intervals are short because the other end is a gateway on the same machine or S3 in the same region.

That makes container replacement on OwlAI SaaS and restarts and updates on-premises the same event. When a container disappears, its files are still in object storage, and the new container fetches them when it needs them. However many copies of the web backend and workers run, they all see the same storage.

#One bucket per purpose, and on-premises each bucket is a volume

There are four buckets, one per purpose. Each name is the bucket prefix followed by the purpose:

• Sandbox: files the model creates, conversation attachments, and preserved tool results
• Original documents
• Meeting audio
• Imported releases: versions brought in for on-premises updates

Keys carry organization and user, document collection, or meeting identifiers instead of a purpose. Sandbox keys sit under the organization and user, and that path is the permission boundary. Right before a request goes to storage, the key's shape is checked again, and keys with segments pointing to a parent path or empty segments are rejected.

One bucket split by prefix would be simpler to manage. We passed on it because the on-premises gateway fails with that layout. During an upload, the gateway creates a temporary directory at the bucket root. If one bucket is mounted as several per-purpose volumes, that temporary directory and the final file land on different volumes. The upload then fails on permissions, and even when permissions line up, a rename within one file system turns into a copy between volumes.

Per-purpose buckets also made the on-premises migration easy. In the POSIX backend, a bucket directory is a Docker volume. Upgrading from the version that kept files on local volumes to the object storage version only takes mounting each existing volume in its bucket's place. Not a byte of data moves, and the backup targets are the same volumes as before.

#Only S3 features both stores support

Stores that share the S3 API name still differ in what they support: conditional writes (If-Match), lifecycle rules, event notifications, and ETags. Use a feature that works only on AWS S3 and on-premises breaks; depend on behavior only the gateway has and OwlAI SaaS breaks. This gateway in fact returns no ETag for files placed in the volume without going through the gateway, meaning files that were there before the migration. Code that detects changes by ETag has nothing to compare for those files.

So the app takes over the change detection, cleanup, and protection against concurrent writes that those features would provide.

Most uploads are a single PUT. The document and meeting upload cap is 500 MB, far below the S3 single-PUT limit (5 GB). The body hash is computed while reading the file in chunks, so even a large file is never loaded whole into memory. Only imported releases, which run to gigabytes, are split into parts by the admin console and joined with a multipart upload. To complete the upload, the server uses the part list and ETags it reads back from storage just before completion rather than the list the client sent.

#Only the backend and workers can reach the gateway

The on-premises gateway sits on a single internal network reserved for storage and opens no host port. Only the backend and workers, which share that network, reach the gateway by name. The container that serves the screens and the sidecar that runs scripts the model writes cannot reach the storage API, and the gateway has no route out.

The gateway process runs as a non-root user, with no Linux capabilities, on a read-only root file system. The gateway root holds only the bucket mounts, so it cannot create any bucket beyond the four the installation defines. All of its state is in the volumes, so the gateway container can be recreated on every deployment.

The gateway stores object metadata in the files' user extended attributes (xattr). The file systems holding the volumes and Docker's storage both have to support xattr; ext4 and xfs do by default. The backup tool also captures extended attributes and numeric owners when it copies the volumes.

#The container keeps one work volume, and everything in it can be rebuilt

The only thing on container disks is the work volume, and everything in it can be recreated.

Each turn folder holds a copy of that conversation's files, so the work volume grows with the number of concurrent turns. Two limits bound its size. One conversation (one run, for a playbook) can keep up to 512 MiB in object storage by default; anything beyond that is not uploaded. A turn folder can grow to 1 GiB by default during a turn; past that, only that turn fails. The first value must be smaller than the second, and a configuration that breaks this is rejected at startup, because a turn needs room to work right after its files are fetched. The pre-installation check uses these two values and the number of concurrent turns to compute minimum and recommended disk sizes (“The preflight check only reads the host and leaves the fixes to a person”).

We recommend putting the work volume on a different file system from the PostgreSQL data. If a burst of concurrent turns fills the work volume, a separate file system means only those turns fail and database writes continue. On the same file system, database writes fail too and the whole service stops.

#Design choices

We chose refusing to start over a local directory fallback. A fallback keeps the service alive while storage is briefly down, but an installation missing its settings stays hidden until the moment files are lost. Refusing to start shows the problem on the first day of installation.

For the on-premises default object storage, we chose a gateway on the same host over a separate storage cluster, so the product stands on one host. The cost is that file durability rests with the disk holding the volumes (RAID, in-house storage). In return, backup is simple, because a bucket directory is a Docker volume: copy the three volumes for sandbox, original documents, and meeting audio right after the database dump, and let your company's backup system collect them. The imported release volume runs several GB per version, and a signed version can be downloaded again, so it is left out of the backup. The dump taken right before an update (“When an update fails, the schema change decides between rolling back code and restoring the dump”) contains only the database, so it cannot replace this regular backup.

The gateway runs as a single process. More gateway instances on the same disk gain nothing, and with one process the upload, update, and backup paths can all be validated on one configuration.

We chose the features both stores share over the full S3 feature set. The app's own job has to do the cleanup that lifecycle rules would have done, and the per-turn ownership rule has to stand in for conditional writes. In exchange, on-premises and OwlAI SaaS run the same code, and a file path validated on one side holds on the other.

If you ship the same product as single-host compose and on a cloud container service, set this rule from the start. Code that assumes file paths has to be rewritten the moment it moves to the other deployment form, and by then customer files are already piled up on those paths.