The bastion gets one key line that only forwards to the database, not an account

When a production database sits in a private network behind a bastion host (an SSH server that leads into that network), OwlAI reaches it through an SSH tunnel over the bastion. The usual way to set this up is for the customer to create a bastion account and hand over its password or private key. That credential opens more than the database port. It also opens a shell login, command execution, and forwarding to any host the bastion can reach. If a credential stored by OwlAI leaked, a shell into the customer's private network would leak with it.

So OwlAI creates its own SSH key for each organization, and the customer puts only the one line shown on screen into the bastion's authorized_keys. That line blocks shells and commands and opens forwarding only to the database addresses it lists. This post covers what the line allows, why each bastion gets exactly one line, how the line becomes two while the key is being replaced, and what opening a tunnel for every query costs and how to configure the bastion for it.

#One line decides what OwlAI can do on the bastion

The line shown on screen looks like this.

restrict,port-forwarding,command="/bin/false",permitopen="<db>:<port>" ssh-ed25519 AAAA… owlai

restrict turns off command execution, terminal allocation, agent and X11 forwarding, port forwarding, and user rc files for any session that logs in with this key. The port-forwarding that follows turns port forwarding back on, and permitopen narrows its target to the listed database host:port. command="/bin/false" is a forced command, so nothing runs even if a session opens. restrict exists from OpenSSH 7.2 onward; older sshd versions reject the line.

The database host field takes the database address as seen from the bastion, so a private address works. That address goes into permitopen as is, so characters that could change the line's syntax, such as whitespace, quotes, and commas, are rejected at registration.

OwlAI creates the key as ed25519 the first time an organization needs it and stores the private key encrypted. The customer sees only the public key and the line. The SSH keys dialog also lets you copy the raw public key, but putting that alone on the bastion applies none of the restrictions above. What goes on the bastion is always the line from the screen.

#Databases behind the same bastion share one line

When several lines carry the same public key, sshd uses only the options on the first one. If each connection got its own line, the customer would append them one after another, and forwarding would be refused from the second database on. So there is one line per bastion per key, and it lists the database addresses of every connection through that bastion as permitopen entries. When you register a new connection, the screen shows the combined line together with the list Databases this line opens, and the customer replaces the existing OwlAI line on the bastion with the new one.

The hard part is recognizing the same bastion. An administrator can register one sshd by name for one connection and by IP for another. Comparing names alone would make that look like two bastions and produce two lines, but the server has one authorized_keys file, so again only the first line counts. So connections are grouped as one bastion when their normalized names (lowercased host, port, and account) match, or when their verified host key fingerprints and accounts match. Connections linked by either criterion end up in the same group.

The line also covers cases where the forwarded port differs from the database port. With TLS on, Tibero connects to its TLS listener at the listener port + 2, so that is the port the tunnel forwards to. The line is built from the port actually forwarded, given the engine and TLS setting. That means a tunneled Tibero connection needs a fresh line on the bastion each time TLS is turned on or off. Which name the certificate is checked against over the tunnel is covered in “Certificate errors are fixed with the right CA and the right name, not by turning off verification”.

#The bastion is accepted only with a verified host key

The line narrows what OwlAI's key can open; the host key check keeps OwlAI from handing database credentials to the wrong server. Pressing Check host key in the registration dialog makes OwlAI connect to the bastion, fetch only the host key fingerprint, and disconnect. The administrator compares it with the fingerprint the bastion's administrator provided, ticks the confirmation, and registers. After that, the connection proceeds only when the host key matches the stored fingerprint exactly.

Accepted host keys are ed25519, ECDSA, and RSA SHA-2; a bastion that offers only SHA-1 ssh-rsa is not connected. The fingerprint lookup and the real login use the same algorithm list in the same order. If the lists differed, an administrator could verify the ed25519 fingerprint and then get an RSA key at login, and the connection would drop as a mismatch. Login uses only the registered connection details; the server's known_hosts, ~/.ssh/config, SSH agent, and default key files are not read. The bastion address is checked as an allowed destination at registration, on edit, and each time an executor is created, and OwlAI dials that checked IP directly.

OwlAI's end of the tunnel is also reserved for the OwlAI process. The driver connects to the tunnel through an ephemeral port on loopback, and another process on the same host that found and connected to that port would get a path to the database behind the bastion. So the tunnel checks that the connecting socket belongs to its own process and drops it otherwise. For Tibero, whose driver runs in a separate container (a sidecar), the tunnel end is a Unix socket on a shared volume, and it forwards only when the peer is the sidecar.

If the bastion refuses forwarding under permitopen, the driver sees only an unexplained disconnect. So right after the SSH login, the tunnel opens one forward as a probe. If that is refused, the bastion step of Test connection fails with "the bastion could not forward to the database; check the database address and permitopen", which tells you right away that the line on the bastion is out of date.

#Both keys work while the key is being replaced

Replacing the organization key, on a schedule or when a leak is suspected, means changing the line on every bastion. With several bastions, the customer needs time to change them all, and connections must not break in the meantime. So key replacement has four steps.

1. Press Start key rotation to create a new key. From then on, the tunnel offers the current key and then the new key.
2. On each bastion, the customer replaces the OwlAI line with the two lines shown on screen (current key and new key).
3. Press Complete rotation. OwlAI tries logging in with the new key alone for each tunnel connection, and if every one succeeds, it destroys the old private key.
4. The customer removes the old key's line from each bastion.

If any connection fails to log in at step 3, the replacement does not finish and that connection is listed. It means a bastion still lacks the new line. Disabled connections are checked too, since they must not be broken when turned back on. If another administrator registers a tunnel connection during the check, the two operations lock the same key rows and run one after the other, and whichever comes second checks again.

#Tunnels open per query, so give the bastion room for concurrent logins

OwlAI's SQL executor opens and closes a fresh driver connection for every query. The tunnel opens and closes with that connection, so no tunnel remains when a query ends. The price is the SSH login, key exchange, and public key authentication on every query. We measured it in a local Docker environment (an OpenSSH bastion on the same Docker network, 40 runs of SELECT 1 including connection setup, 2026-10-08).

Opening and closing the tunnel alone takes about 100ms, about 78ms of it for the SSH login and authentication. Round-trip latency to the bastion adds to that once per handshake round trip. The SQL tool in chat connects twice, once for explain and once for execution, so the cost barely shows in the wait for an answer. It adds up in table analysis, which queries the catalog, statistics, and samples several times per table. At 130–150ms more per query, 1,000 queries take a little over 2 extra minutes.

On the bastion, sshd's MaxStartups comes into play. Every query opens a new SSH session, so as many sessions waiting to authenticate overlap as there are concurrent queries. The OpenSSH default of 10:30:100 starts dropping new connections at random once more than 10 are unauthenticated. Size it above the per-database concurrency limit (dbConnection.perDbConcurrency, default 4) × the number of connections sharing the bastion × the number of backend and worker processes. For example, MaxStartups 50:30:200.

PerSourcePenalties, in OpenSSH 9.8 and later, penalizes connections that close without authenticating. Check host key fetches the key and disconnects, so pressing it several times in quick succession can get the egress IP refused for a while. If that happens, wait a moment or add the egress IP to PerSourcePenaltyExemptList.

#Design choices

Instead of taking the customer's bastion account and private key, we chose to have OwlAI create an organization key and hand out a single line. Taking an account would save the customer one step, since there would be no line to manage. But that account opens a shell and every forward, so a leak would reach the whole network behind the bastion. With the line, OwlAI's key opens only the database ports the line lists. The cost is that the customer must put a new line on the bastion whenever a database is added or a Tibero connection's TLS setting changes.

Instead of a line per connection, we merged them into one line per bastion, because sshd reads only the first line for a given key. The cost is that two HA bastions that deliberately share a host key are also grouped as one. The screen shows a single representative name, so the customer has to put the line on both machines; during a key replacement, the Complete rotation check exposes connections on the machine that was missed.

Instead of keeping tunnels open and reusing them, we open one per query. Reuse would save roughly 100ms per query, but it brings state management for noticing and reopening dropped tunnels, and it leaves sessions standing on the bastion. Per-query tunnels leave no session on the bastion once a query ends; the cost is the time measured above and a MaxStartups adjustment.

For the period when both keys work, we chose a login check over a deadline. A deadline makes the exposure window look shorter, but connections through a bastion whose line was changed late would break when it passed. Only OwlAI holds either private key, so a longer overlap does not widen where the key can reach.

If your databases sit behind a bastion, the customer side comes down to three things: put the line from the screen into the authorized_keys of a dedicated account with shell login disabled, open the bastion's SSH port to OwlAI's egress IP only, and size MaxStartups for concurrent queries. Give the database account read privileges only, and what OwlAI can do on a database behind the bastion stays limited to reads as well.