Database backups carry stored passwords, so the key lives elsewhere

OwlAI's metadata database holds customer database passwords, LLM provider API keys, OAuth tokens for connectors (external services such as MCP servers that a company connects to OwlAI), and SSH private keys used for database connections. Store those values as plain columns and anyone who takes a single dump or backup file can read every one of them. Backup files get copied to places other than the production database, and they are kept longer than production data.

So OwlAI stores credentials with envelope encryption. Each credential is encrypted with a data key, and that data key is encrypted again with a key that lives outside the database. Someone holding only the database can open nothing. This post covers how far that protection reaches, and what happens when the key is rotated or lost and when plaintext is left behind. The other half, keeping secrets from flowing to the model and into logs, is covered in “Keeping presigned URLs out of the model with 19-character references”.

#Database encryption or a single app key either misses dumps or makes rotation hard

There are two common approaches. The first is the database's storage encryption (TDE or disk encryption). It stops someone who walks off with the disk or the data files. But anyone who can connect to the database and take a dump gets the values in plaintext, and a backup made from a logical dump contains that plaintext too.

The second is for the application to encrypt secret columns with a single key. That does stop dumps, but three problems remain.

• Rotating the key means decrypting every row and encrypting it again under the new key.
• If the key is derived from the installation secret used for login signing, rotating that secret leaves every stored credential undecryptable.
• Ciphertext copied into another row or another organization's column decrypts with the same key. Someone who can edit the database can move another organization's database password onto their own connection and use it.

With envelope encryption, rotating the key only means re-encrypting the wrapped data keys, and ciphertext can be bound to where it is stored.

#The credential goes under a data key, the data key under a key outside the database

The credential body is encrypted with AES-256-GCM under a data key (DEK). The data key is wrapped with a key encryption key (KEK) and stored next to the ciphertext. The stored value is a single JSON blob holding the key ID of the KEK that wrapped it, the wrapped data key, the nonce, and the ciphertext. The KEK is not in the database.

Values stored this way are customer database passwords, LLM providers' shared and personal API keys, connectors' tokens, API keys, and client secrets, organization SSH private keys, account MFA secrets, and the private key used to encrypt login credentials in transit. Short-lived values such as intermediate OAuth values (10 minutes) and download URLs masked from the model (1 hour, “Keeping presigned URLs out of the model with 19-character references”) are stored the same way.

Writes always use this storage format. Without a KEK configured, credentials are not saved. Reads open only values in this storage format. If a credential column holds plaintext, the value is not used as a credential and is treated as empty. Reading plaintext as-is would let someone who can edit the database swap an encrypted value for plaintext they know and make OwlAI connect with it.

#Ciphertext only opens in the column where it was stored

AES-GCM verifies additional authenticated data (AAD) along with the ciphertext. When the body is encrypted, that slot holds the purpose (which column of which table), the organization, and the row identifier. Ciphertext copied into a different row, organization, or purpose column fails the authentication tag and won't decrypt. That is what stops moving another organization's database password onto your own connection.

The information bound in when wrapping the data key is the purpose and organization only, not the row identifier. Including the row would mean wrapping a separate data key per row, and on OwlAI SaaS, which uses KMS, every write would need one more KMS call. With only purpose and organization bound, writes for the same organization and purpose share one wrapped data key for 5 minutes. The body's AAD already binds each row, so copying is still blocked.

#On-premises keeps the KEK in the installation secrets, OwlAI SaaS in KMS

Where the KEK lives depends on the installation type, and both options follow the same interface.

Each KMS call is recorded in CloudTrail along with its encryption context. The row identifier is not part of that context, so it does not appear in the cloud audit log. An OwlAI SaaS host starts only with KMS. If a misconfiguration tries to start it with a file key, startup is refused.

The on-premises KEK is a different value from the installation secret used for login signing, so rotating the login secret leaves stored credentials decryptable. If the KEK setting is missing or malformed, the backend refuses to start, and the pre-installation check looks at the same conditions.

By threat, the protection reaches this far.

#Key rotation rewraps only the data keys

Reads hold both the current KEK and old KEKs, so rows wrapped with an old KEK stay readable mid-rotation. Rows wrapped under a key ID that is not on the list are not opened. That closes a path in which someone who edited the database swaps in a key ID pointing at another KMS key the app can use.

Rotation takes four steps.

1. Add the new KEK to the read list.
2. Switch the write KEK to the new key and restart the backend and worker. From then on, newly saved values are wrapped with the new key.
3. A periodic worker job runs every hour, unwrapping data keys wrapped with the old KEK and wrapping them again with the new one. The body is not re-encrypted. If you don't want to wait, a command runs the same work once right away. Running it again moves only the rows that remain.
4. Check the count moved at the next run, confirm that the run after that finds zero rows to move, and only then delete the old KEK. Delete the old key while unmoved rows remain and those rows won't decrypt.

Before rewrapping a data key, the periodic job checks the body's authentication tag. Rewrapping ciphertext copied from another row under the new key would make a value that should never open look like a normal stored value.

Rewrapping does not change the data key, and that matters when you rotate because the KEK leaked. Anyone holding both the old KEK and an old database dump already knows the data keys, and rows whose body is unchanged can be opened from a new dump as well. So after a KEK leak, rotate the key and also replace the stored credentials themselves: reconnect connectors, and change API keys and database passwords where they were issued before registering them again. Rotation alone is enough only for routine replacement.

#The key is kept somewhere other than database backups

If the KEK is lost, stored credentials cannot be recovered, and an administrator has to register them all again.

The on-premises backup tool saves the database and file storage as a single point-in-time bundle, and it does not include the installation secrets that hold the KEK. Keep the installation secrets somewhere other than data backups, in a store with separate access or a secrets management tool. If database backups and the KEK sit together, envelope encryption stops protecting anything the moment one backup leaks. After rotating the KEK, keep the old key as well, because rows wrapped with it remain in older data backups.

OwlAI SaaS KMS keys are not deleted. A key that is no longer used is disabled, with the deletion waiting period set to the maximum. The key policy separates key administrators from key users: administrators can't decrypt with the key, and the app's role can't manage it.

#A check command counts credentials that aren't encrypted

Envelope encryption means something only when no plaintext credentials remain in the database. A check command verifies that. It runs inside the backend container without the KEK and prints no values. It reports the following for each column.

• Rows holding values not in this storage format (plaintext)
• Rows holding old-format ciphertext from earlier versions
• MCP server URLs carrying secrets in the query string (stored before the rule that refuses such URLs existed, “Keeping presigned URLs out of the model with 19-character references”)

All zeros is success, any remaining rows is failure, and configuration or database errors exit with their own code. Organization tables have row-level security (RLS), so the command opens a session for each organization and counts there. Before upgrading, confirm zero with this command. This version cannot open old-format ciphertext, so any credential still in that form has to be registered again.

The periodic job does not encrypt plaintext in credential columns. Plaintext in a credential column may have been planted by someone who edited the database, and if the periodic job encrypted it, the plaintext rejection described above would shrink to a one-hour delay. The job counts those rows and logs them, and the check command reports the count. If it really is old plaintext that needs moving, a person who has confirmed where it came from runs the option that also encrypts plaintext, once.

To keep new secret columns from slipping past the check, a test confirms that the list of code that writes values in this storage format matches the list of columns the check scans. Add a column without putting it on the check list and the build fails before merge.

#Design choices

The default on-premises KEK lives in the installation secrets. Making an external KMS the default would take the raw KEK off the host, but it doesn't fit an on-premises product that has to stand up on a single customer host without extra equipment. So database leaks are the default scope of protection, and installations that need host-root-level protection use file injection from a secrets management tool or an external KMS. Both options follow the same interface, so the switch is a configuration change.

With KMS, unwrapped data keys are cached for 5 minutes. Calling KMS on every read would make revocation immediate, but latency and KMS cost would grow with every credential read. The price is that revocation can take up to 5 minutes to apply, and calls made during those 5 minutes are still recorded in the audit log.

Rotation stops at rewrapping data keys, and KEK-leak response is a separate procedure paired with credential reissuance. If rotation also replaced data keys, a leak response would finish in one step, but every routine rotation would have to re-encrypt every body. Routine rotation happens often and leak response rarely, so we kept the frequent one light.

For a lost KEK we chose a re-registration procedure over a recovery path. A recovery path would be one more way to open credentials without the KEK. Instead, the procedure keeps the key somewhere other than database backups, and the encryption scope is limited to credential columns so other data survives a lost key.

If you run an installation on a single customer host, decide at install time where the KEK lives, who holds it, and how it stays apart from your database backups.