Sandbox each script run, then prove isolation on the actual server

In OwlAI, Excel, Word, PowerPoint and Hangul documents are produced by python and node scripts the model writes, and scripts also post-process query results. Nobody reviews these scripts before they run. The sidecar container that runs them mounts a single volume holding every user's working folders, so wrapping the scripts in a container does not stop one run from reading or overwriting someone else's output. That is why each run gets a fresh set of namespaces from bubblewrap, and sees only its own folders.

This isolation is not self-contained in the image. The user namespaces bubblewrap relies on are a feature of the host kernel, and whether a process may create one is decided by the host's kernel settings and security modules (AppArmor, SELinux). The same image can pass on a development PC with security modules off and have every run refused on an Ubuntu 24.04 server. This post covers how one run is confined, and how we decide on the server itself, with the same command a real run uses, whether isolation holds, refusing every run when it does not.

#One container sees every user's folders

Unreviewed code can do damage in four areas.

• Files: read or overwrite another user's or another organization's outputs and attachments, or corrupt a source template.
• Network: send data to an outside address, or make requests to internal services and the cloud instance metadata address.
• Resources: stall other users' runs with a loop that never ends, an allocation that eats all memory, code that keeps spawning processes, or a file that fills the disk.
• Kernel: exploit a kernel vulnerability to gain privileges and leave the container.

None of this needs malice. A model mistypes a path or follows an instruction hidden in a document (“Permissions are enforced in code, judgment is left to the model”). Isolation has to produce the same outcome whether the behavior is a mistake or induced.

#In-interpreter monitoring and newer kernel features don't fit on-premises

Python audit hooks (PEP 578) can filter events such as file writes without any special kernel feature. But the hook decides inside the same interpreter as the script, so a script that swaps out globals can shake the decision, and writes from C libraries do not raise events at all. The Python documentation itself says audit hooks are not suitable for implementing a sandbox.

Landlock does not have this weakness: the process narrows its own access and the kernel enforces it. But the feature we need (ABI 2) arrived in kernel 5.19, so it is unavailable on the RHEL 8 family, which runs kernel 4.18. An on-premises product does not get to choose the customer's distribution.

bubblewrap builds a new mount namespace on top of an unprivileged user namespace. User namespaces have existed since kernel 3.8, so the kernel version does not rule out whole distributions. What remains are kernel settings and security modules, which we deal with further down.

#A run sees only its own folders and has no network

For each run, bubblewrap mounts system directories and the run's input folders read-only, mounts only the output folder writable, and hides the rest of the filesystem. Because the kernel narrows the view when it resolves paths, writes from C libraries and direct system calls are blocked too.

A script can read only its own run's outputs, attachments and raw tool results (a run is one conversation or one playbook run), plus the files uploaded to the playbook when it is a playbook run. Other conversations of the same user and attachments that have not been sent yet are invisible. Folders on the volume are split by organization, then person, then run, so a run in one organization never sees another organization's folders.

Besides the mounts, bubblewrap does three more things per run.

1. It gives the run its own network, PID, IPC, UTS and cgroup namespaces. The script has no network device, and the sandbox dies with the sidecar process that started it.
2. It applies a second seccomp filter to the script. The container's seccomp profile has to leave some system calls open so bubblewrap can create namespaces, so untrusted scripts get a narrower filter. That filter blocks calls that would create another user namespace and a known kernel privilege-escalation path.
3. It does not mount /proc. The script cannot read other processes' command lines or environment variables.

When a script needs data, it calls an OwlAI tool. The call goes back to the turn that started the run and passes the same permission checks and approvals as a call the model makes directly (“Tool calls a script makes still get permission and approval checks”). The script has no direct route to outside addresses or internal services.

#The container is locked down too, and the sidecar owns the limits

The container around bubblewrap runs with these settings.

• Read-only root filesystem, 128 MB /tmp without exec permission
• All Linux capabilities dropped, privilege escalation disabled (no-new-privileges)
• A seccomp profile that is Docker's default plus only the calls bubblewrap needs to create namespaces, generated by a script from the upstream default and regenerated and compared by a unit test
• On Ubuntu 23.10 and later, the bundled AppArmor profile
• An internal-only network with no gateway, reaching neither the internet nor the metastore database

The sidecar sets resource limits for each run and does not take the caller's values at face value.

node gets 2 GB because its engine reserves more than 1 GB of virtual address space at startup and dies without output under a 1 GB limit. A script that calls tools keeps the clock running while it waits for approvals and tool results, so a script whose computation never ends is cut off at 60 seconds of CPU time instead of wall-clock time.

Process count is capped by the container's PID limit, not RLIMIT_NPROC. When a container runs without a user namespace, RLIMIT_NPROC counts every thread on the host running under the same user ID. With that limit in place, whether a normal run survives would depend on how busy the host happened to be.

#The same image may not get isolation, depending on the server's kernel and security modules

The host decides whether isolation holds. These are the conditions we found by actually starting the sandbox in a VM per distribution.

On Ubuntu, turning off the host-wide user namespace restriction also works, but it removes that protection from every process on the server. So we bundle a profile that opens only what bubblewrap needs.

None of these conditions shows up on a development PC with security modules off. In environments such as WSL, AppArmor does not stop user namespace creation and SELinux does not stop mounts. A green test there, after changing isolation options, is no evidence that the change passes on a real server. So whenever isolation or security options change, we start the stack in a VM per distribution, on the real kernel and security modules, and judge two things: the isolation status in the health check is healthy, and at least one real script run succeeds.

On Ubuntu 24.04 with the bundled AppArmor profile loaded, a batch of python and node runs that build docx, pptx, xlsx and hwpx files passed all 8 document-generation and check cases. In a control group where the container had only Docker's default seccomp profile, the same batch failed entirely.

#The self-test starts bubblewrap with the same command as a real run

We cannot repeat the VM verdict on every customer server. So the sidecar checks isolation on the server it runs on. It starts bubblewrap once with the same command a real run uses and checks three things inside.

• It can mount the sandbox volume read-only, as a real run does, and list it
• It can write and delete a file in a dedicated check folder
• /proc is not visible inside the sandbox

The result goes into the health check. When isolation holds, it returns 200; when it does not, it returns 503 with the last line of the bubblewrap error as the reason. A 503 marks the container unhealthy and fails the deployment check, and the sidecar refuses every script run until isolation holds.

A separate command just for the check would miss, by however much it differs from a real run, any condition under which the kernel refuses only the real run. The health check would report healthy while scripts failed, and users would see what looks like broken model code. So the check and real runs share one command for starting bubblewrap.

The install check uses the same self-test. Before the production stack comes up, it starts the delivered sidecar image as a throwaway container without a network and runs the check once. The host conditions in the table above are each judged by their own install-check item (“The preflight check only reads the host and leaves the fixes to a person”).

#Design choices

On a server where isolation does not hold, we chose to refuse every run instead of running without isolation. The cost is that document generation and post-processing stop entirely on a server that is not ready. So the install check judges isolation before installation, and a host setup script puts the needed profile and policy module in place. In exchange, no server ever runs a script without isolation.

We limited support to the combinations a real run has passed in a VM, and the install check warns on combinations we have not judged, such as a kernel that is not the distribution's stock kernel. If an unfamiliar combination blocks isolation in an unfamiliar way, the self-test still uses the same command as a real run, so the deployment check catches it.

Dropping /proc and keeping the address space small has a price too. Without /proc, some runtime information changes; node's CPU count, for example, returns 0. With a 2 GB address space, WebAssembly instantiation fails. So before adding a new document library, we run it in the real sandbox under both conditions.

OwlAI SaaS cannot set a per-container seccomp profile or PID limit in its task definitions. So the host Docker daemon's default seccomp profile follows the same rule, and the task's CPU and memory limits and the run time limit stop runs that never end, in place of the PID limit. The command that starts bubblewrap for each run and the seccomp filter on the script are the same as on-premises.

If you ship isolation that depends on kernel features to servers your customers choose, check first that your health check starts the isolation with the same command as a real run, and that tests of changed isolation options ran on a real kernel with security modules on.