People set the Docker address pool, and the check compares it with internal ranges

Install on-premises OwlAI on a host inside a corporate network, wait a few weeks, and one day an internal database or inference server that answered yesterday may be unreachable from your containers. A shell on the same host can't reach it either. The firewall rules haven't changed and the server is up. The cause is a bridge network Docker created in the meantime. When Docker takes the range that holds the internal server as that network's subnet and adds a route to the host routing table, every packet for that range leaves through the bridge.

When Docker picks a subnet, it tries to avoid ranges already in the host routing table. Internal ranges behind the default gateway don't appear in that table, so Docker can't avoid them. An installer that judges from the routing table alone misses them for the same reason. So on-premises OwlAI has the person installing it set Docker's address pool, and the preflight check compares the pool against internal ranges that a person writes down. This post covers how the collision happens and those two safeguards.

#Internal traffic that used the default route leaves through a Docker bridge

Say the host is 10.20.0.15 and it reaches the internal server 192.168.5.20 through the default gateway. The host routing table holds only default via 10.20.0.1 dev eth0 and the route for its own subnet. Once Docker takes 192.168.0.0/20 for a new network, you see this:

$ ip route get 192.168.5.20
192.168.5.20 dev br-3f2a… src 192.168.0.1

When several routes match a destination, Linux picks the one with the longest prefix (longest prefix match). The default route is /0 and the Docker bridge route is /20, so packets for the internal server leave through the bridge instead of the physical interface. All 4,096 addresses in that /20 are now unreachable from containers and from the host.

If the outage has already happened, one command tells you whether this is the problem: ip route get <internal server IP>. A result of dev br-… or dev docker0 means a Docker network has taken that range. Use docker network inspect to find which network owns the bridge, fix the address pool, and recreate that network. The bridge may belong to another project running on the same host, so find its owner before deleting anything.

#Docker avoids only the ranges in the routing table

If daemon.json doesn't set default-address-pools, Docker carves subnets out of its default pool. According to Docker's documentation, the default pool is the following, and Docker tries to avoid prefixes already in use on the host.

The 172 range yields 15 networks, counting the default bridge docker0. After that, networks come from 192.168.0.0/16. On-premises OwlAI's production compose file uses one shared external network plus four dedicated networks, one each for the script execution, document rendering, file storage, and database driver sidecars, so that each sidecar is attached only to the networks it needs. On a host with no other networks, that leaves 9 slots in the 172 range once the production stack is up. Start three more compose projects that create four networks each, and the third project's second network comes from the 192.168 range.

That sequence is what makes the cause hard to find. On installation day there is room in the 172 range and nothing happens. Weeks later someone starts another stack on the same host, the collision happens, and the only thing that person changed was their own project's networks. Few people connect an internal server going dark with starting one test stack.

The 172 range isn't safe either. Corporate networks often use 172.16.0.0/12 too, and the AWS default VPC range, 172.31.0.0/16, sits inside 172.28.0.0/14 in Docker's default pool.

#Whatever default range you pick, some customer network uses it

The common approach is for the installer to ship a default range that looks unlikely to collide. The person installing can skip the decision, and on most hosts nothing goes wrong. But an on-premises product meets a different corporate network at every customer, and the installer can't know which of the three private ranges (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16) a customer uses. At a customer whose network overlaps the default, the problem surfaces as an outage a few weeks after installation.

Another approach is for the installer to read the host routing table and pick a free range. It needs no input, but it misses exactly the ranges Docker misses. Internal ranges beyond the default gateway, a VPN, or an upstream router aren't in the routing table.

Only the company's network team knows the whole corporate network. So a person picks the range, and the tools check the value that person picked.

#The installer picks the range, and the host preparation script stops on an overlap

On-premises OwlAI's host preparation scripts (one for Ubuntu, one for the RHEL family) take Docker's address pool as a required argument. Without a value, they don't start. The value must be a range used nowhere in internal routing, VPN routes, or peered VPCs.

{
  "default-address-pools": [ { "base": "10.250.0.0/16", "size": 24 } ]
}

The script writes daemon.json with this value. With this setting, Docker carves /24 networks out of 10.250.0.0/16, room for 256 networks. Before writing, the script compares the range against the host routing table and stops on an overlap. If an existing daemon.json differs from what it wants to write, the script stops instead of overwriting it; only with the force option does it keep the original as .bak and replace it. These checks run before any step that changes the host, such as installing packages, so when the script stops, nothing on the host has changed.

After the address pool changes, dockerd has to restart. A restart takes every container on the host down and back up, so the script restarts only when a configuration file actually changed. Existing networks keep their old subnets, so you have to recreate them for the new pool to apply.

#The check looks at existing networks and future pools separately

The preflight check (“The preflight check only reads the host and leaves the fixes to a person”) runs again on every deployment and update, and it judges network collisions with three items.

A check that looks only at existing networks is simple and raises no false alarms. On installation day, though, the colliding network doesn't exist yet, so that check passes an outage that comes weeks later. network.pools_future looks at the pool instead of the networks. On a host that keeps the default pool, if --corp-cidrs includes 192.168.0.0/16, the check fails before installation, even though Docker hasn't carved a single subnet from the 192.168 range yet.

Internal ranges come in through the --corp-cidrs option. The only way to compare against ranges that aren't in the routing table is for a person to write them down.

tools/ops/preflight.sh --corp-cidrs 10.0.0.0/8,172.16.0.0/12

network.pools also compares against the host routing table, so it catches a VPN that connects or a static route that appears after the networks were created. It leaves out the routes Docker and podman created (docker0, br-*, veth*, and the podman bridges). Without that exclusion, the check would mistake its own networks for the corporate network and fail, and the check that runs on every deployment would stop every deployment. Podman adds its bridge route when the first container attaches, so on a host where even one service is running, the next deployment would stop without this exclusion.

The check reads the address pool from the right place for each engine: /etc/docker/daemon.json for rootful Docker, the deploying user's ~/.config/docker/daemon.json for rootless Docker, and default_subnet_pools under [network] in containers.conf for podman. Without that setting, podman's default pool runs from 10.89.0.0/16 to 10.255.0.0/16, so if the corporate network is 10.0.0.0/8, the check fails outright. If daemon.json exists but can't be read, the item is left as SKIP (cannot judge). Treating an unreadable file as the default pool would produce a wrong PASS or FAIL.

#Design choices

We made the address pool a required argument instead of shipping a default. A default would save a step at installation and would be right on most hosts. In exchange, at a customer whose network overlaps it, the outage would come weeks after installation, right after some change that looks unrelated. The cost of a required argument is that the installer has to ask the network team for a free range, a question asked once, on installation day.

We also chose to take internal ranges through --corp-cidrs instead of inferring them from the routing table. Docker fails to avoid those ranges and the check would miss them for the same reason: both look only at the routing table, and internal ranges usually aren't in it.

For network.pools_future, we chose to fail whenever the pool contains an internal range instead of predicting whether Docker will actually get far enough into the pool to carve a subnet there. That depends on how many more networks end up on the host, which nobody knows before installation. So every possible collision becomes a FAIL on installation day, and the installer settles the address pool that day.

We left recreating existing networks after a pool change to a person. Letting the tools do it would save effort, but containers would go down while services are running. When to take containers down is a decision for the people who share that server, the same reason the check is read-only (“The preflight check only reads the host and leaves the fixes to a person”).

If your product runs several compose stacks on one host, a good place to start is taking Docker's address pool as an installation input and the list of internal ranges as an input to your checks. The more networks pile up on a host, the further the default pool reaches into the private ranges corporate networks use.