Certificate errors are fixed with the right CA and the right name, not by turning off verification

When OwlAI runs on a corporate network and connects over TLS to a cloud LLM or a customer database, certificate errors come from two places. One is an SSL inspection appliance at the network exit, a device that decrypts HTTPS, inspects it, and re-signs it with the company's internal root CA, so the server certificate arrives signed by that CA. The other is a database driver that connects to 127.0.0.1, the local end of an SSH tunnel, while the database certificate is issued to a name like db.corp.example. Both show up as CERTIFICATE_VERIFY_FAILED, and the fastest workaround is the same for both: turn certificate verification off. The traffic is still encrypted, but nothing stops someone in the middle from swapping in their own certificate.

Each error comes from one wrong input to verification. In the first case the CA to trust is missing; in the second, the name compared against the certificate is wrong. So instead of turning verification off, OwlAI fixes the input. The CA to trust is decided in one place and handed to every outbound call as the same TLS verification settings, and the name to check on the certificate travels separately from the address the driver dials. On installs that supply an internal CA bundle, the one check we turn off is a format check that Python 3.13 enables.

#Turning verification off leaves only encryption

TLS verification checks two things: that the certificate chains up to a trusted CA (chain verification), and that the name on the certificate matches the host being contacted (hostname verification). verify=False turns off both, so a certificate signed by an attacker's CA is accepted and the database credentials and query results go to that party.

Database drivers start out in this state. If TLS is not enabled on a connection, PostgreSQL's libpq (sslmode=prefer) uses TLS when the server offers it but does not check the certificate, and MariaDB/MySQL's pymysql (PREFERRED) tries unverified TLS and falls back to plaintext when the server offers none. Oracle (python-oracledb thin) and Tibero (JDBC) connect in plaintext.

So in OwlAI's database connection form, turning on Use TLS verifies both the chain and the hostname, and the connection fails if either check fails. A database with a self-signed certificate or a private CA is handled by pasting the CA certificate (PEM) into the connection. New connections start with TLS on, and a connection with TLS off that also skips the SSH tunnel is labeled Unencrypted connection.

#The trusted CA is decided once and handed to every outbound call

Behind an SSL inspection appliance, the fix is to trust the internal root CA. The hard part is getting every client to trust it. The backend and worker go out through the OpenAI and Anthropic SDKs, connectors, web tools, embedding, reranker and speech-recognition calls, and web push. Leave httpx or urllib3 at verify=True and each library builds its own default TLS context, and one SDK uses a separate package that reads the operating system's trust store. A single setting change then fixes only some paths, and one missed path is an outage on its own.

So there is one trust rule: SSL_CERT_FILE if set, otherwise SSL_CERT_DIR if set, otherwise the default bundle (certifi). The network policy module builds the TLS verification settings from that rule, and every outbound client receives them. The SDKs get an HTTP client built with them; the web push library (requests), which does not accept a context argument, gets them in its connection pool directly. A unit test walks every place in the product code that creates an HTTP client and fails if one of them does not receive these settings.

The same module also decides proxy behavior. Internal calls to the document-rendering and script-execution sidecars must not go through the corporate proxy, so they turn off environment proxies (httpx trust_env=False), and that option also makes the client ignore SSL_CERT_FILE. Decide the two separately and the internal CA drops out right there, so internal calls get the TLS verification settings passed explicitly. Outbound calls through the corporate proxy set up TLS with the server inside the CONNECT tunnel and are verified with the same settings.

Operators need to watch one thing. SSL_CERT_FILE replaces the default bundle rather than adding to it. Supply only the internal CA, and certificates for domains the appliance exempts from inspection start failing instead. So copy the host OS bundle that already contains the internal CA (the file update-ca-certificates or update-ca-trust produces) into the install directory, mount it read-only into the backend and worker, and add one line, SSL_CERT_FILE=<path inside the container>, to .env. To confirm it took effect, check trust_source (file) and trust_location in the net.egress.policy startup log; if the path cannot be read, net.tls.ca_bundle_unreadable is logged at ERROR.

#Only installs with an internal CA bundle turn off the strict format check

Some networks still produce this error after the internal CA is installed correctly:

CERTIFICATE_VERIFY_FAILED: Missing Authority Key Identifier

The cause is VERIFY_X509_STRICT, which Python 3.13's default TLS context turns on. The flag checks that a certificate also meets the format requirements of RFC 5280. Certificates from public CAs meet them, but the certificates an inspection appliance mints on the fly for each connection sometimes lack the Authority Key Identifier (AKI) extension. On a VM behind an open-source intercepting proxy (mitmproxy 8.1.1 on Ubuntu 24.04), every outbound call failed with this error even with the internal CA installed, and turning off that one flag let the requests reach the server.

OwlAI turns this flag off only on installs that supply a bundle through SSL_CERT_FILE. Supplying a bundle already says that this network re-signs traffic with an internal CA. Only the format check goes; chain verification and hostname verification stay. A certificate signed by a CA outside the bundle, or issued to a different name, is still rejected. Installs without a bundle (OwlAI SaaS included) and installs using SSL_CERT_DIR keep the strict check on, and we turn the flag on explicitly on Python 3.12 and earlier too, so the answer is the same whatever the Python version.

A CA pasted into a database connection works the same way. That connection trusts only the pasted CA and turns the strict format check off, because some private CAs and self-signed certificates on customer databases do not meet every format requirement. For the install, check x509_strict in the startup log. If a bundle was supplied and it reads true, SSL_CERT_FILE never reached the backend and worker, and trust_source also reads certifi.

#The dialed address and the name to verify travel separately

On database connections, the address the driver actually connects to and the name on the certificate often differ. Through an SSH tunnel to a bastion host, the driver connects to an ephemeral port on 127.0.0.1. Without a tunnel, OwlAI resolves the name, checks whether the address is allowed, and has the driver connect to that checked IP. A driver that resolved the name again could land on an address that changed after the check (DNS rebinding). Either way, a driver that checks the name against the address it dialed rejects a certificate issued to the name.

So the connection details handed to the driver have two fields: the address to dial (the tunnel's local end or the checked IP) and the name to check on the certificate (the database host written in the connection). TLS runs over the tunnel all the way to the database, and the name checked is always the connection's host, whatever the path.

The hard part is that each driver takes that name differently, or not at all.

PostgreSQL is the simplest, because libpq takes host and hostaddr separately. libpq accepts a CA only as a file, so a CA pasted into a connection is written once to a file named after a hash of its contents.

pymysql passes the host it opened the socket to straight through as server_hostname and has no argument for a separate name, so we put the name on the SSL context and have it drive SNI and certificate verification when it wraps the socket. Without a context, pymysql drops to unverified PREFERRED, so connections with TLS on always get one. python-oracledb thin turns off check_hostname on the context it receives and compares the name against the dialed host itself (127.0.0.1 through a tunnel). So the driver's name check is off, and the context turns verification back on the moment it wraps the socket. Whatever the driver changed in the settings, the context owns chain and name verification.

For Tibero, the driver runs in a sidecar, so the sidecar sets up TLS. The sidecar has no install trust store, so the backend sends the connection's CA, or the contents of the install trust store, as PEM in the request. On all four engines, the CA to trust is the one pasted into the connection or, failing that, the trust rule from the earlier section. Opening Tibero's TLS port through a tunnel is covered in “The bastion gets one key line that only forwards to the database, not an account”.

#Design choices

Database TLS has one mode: verify the chain and the name. A step like libpq's sslmode=require, which encrypts without verifying, would let a database with a self-signed certificate connect without a CA and make registration easier. But administrators tend to read "encrypted" as "safe," and such a connection is open to a man-in-the-middle attack. So instead of that convenience we chose the guarantee that a connection with Use TLS on always stops a man in the middle. The cost is that the administrator has to find and paste the private CA certificate.

For the trust store, we chose the standard variable SSL_CERT_FILE over a package that reads the operating system's trust store (truststore). Inside a container, the operating system store is the distribution bundle baked into the image, not the customer host's store, so adding an internal CA means mounting a file anyway. The cost is that every new outbound client has to be given these settings, and the unit test catches any that are not.

We tied turning off the strict format check to the fact that a bundle was supplied, rather than to a config switch. A switch would let operators decide explicitly when to turn it off, but they only go looking for the switch after hitting the error. The cost is that there is no way to keep the strict check on while supplying a bundle, so we narrowed what is turned off to that one format check. Unit tests hold this behavior with a local TLS server that presents a certificate without an AKI, and through a CONNECT tunnel.

The dial address and the name to verify travel separately on every engine. That took four different methods, one per engine, and a driver upgrade means rechecking how the name is verified. It adds engine-specific code, but neither a tunnel nor a checked IP ever requires turning certificate verification off.

If a database that uses a private CA sits behind a bastion, paste its CA certificate into the connection and keep TLS on while using the tunnel. The certificate is verified against the host name written in the connection, so if the certificate is issued to a name, register the connection by that name rather than by IP.