On-premises features are turned on by version, not by config flags

Put a boolean field that turns a feature on or off into the settings file of an on-premises product, and every customer server ends up with its own value. A version tested only with the defaults fails at one customer and nowhere else, and the development team can't reproduce it until they get that customer's settings. On an upgraded installation, a new feature starts switched off, with no error, only because nobody wrote the new key.

OwlAI leaves turning features on to the version. Adopted features go into the code, so installing that version means the feature is on. The settings file holds numbers, such as retention periods and timeouts, and model specifications. Whatever has to differ per installation is data that administrators edit in the admin console, and the difference between on-premises and OwlAI SaaS comes down to a single edition value that is fixed at first boot. This post explains why we drew those lines and how tests keep them in place.

#Flag values on customer servers turn every combination into a different product

Feature flags have clear advantages. You ship a feature turned off, turn it on for 10%, then 50%, then 100% of users, and when something goes wrong you flip the switch instead of redeploying. That works when the operations team holds the flag values in one place. In SaaS, even with ten flags, only one combination is on in production right now, and the operations team knows it and tests it.

In on-premises software, flag values live in settings files on customer servers. Customer A keeps the defaults, customer B turns two off, and customer C turns one on after reading documentation from a few versions back. With n boolean fields there can be 2ⁿ different products in the field. Ten fields make 1,024.

The development team tests one combination: the defaults. Feature B, written on the assumption that flag A is off, fails only at the customer who turned A on. The defect first appears at that customer, and there is no way to confirm that the settings file you receive matches what was actually running.

The off branch also stays in the code. A single flag that turns the playbook engine on and off needs, for its off branch alone, guards at 28 endpoints, early returns scattered across services and workers, three separate pieces of code that read the flag, and 26 tests that exercise only the disabled engine. If the flag defaults to on, all of that code serves a branch that almost no installation ever takes. So the playbook engine ships always on.

#On an upgraded installation, the new feature starts switched off

Harder to spot than the number of combinations is a feature that turns off without any signal. Say a new feature sits behind a flag that defaults to off. The new version's sample settings file has the new key, but the key does not appear on its own in the settings file the customer has edited. The customer doesn't know about the key, doesn't add it, and the feature stays off with no error and no warning. Someone asks why a feature from the release notes is missing, and support finds the cause only after opening the settings file.

A feature decided by "on if the value is present" sets the same trap. If outgoing mail were decided by "send when .env has the sender coordinates," an installation missing one line would start, with no error, showing links on screen instead of sending mail. So the edition decides whether an installation sends mail, and an edition that sends mail checks the sender coordinates and public address at startup and refuses to start without them.

Settings key spelling leads to the same result. The settings model drops keys it doesn't know. That choice lets a settings file from a few versions back still start a new version, but a misspelled key leaves its field at the default and still passes startup validation. So the model keeps dropping unknown keys, and at startup it logs a config.unknown_key warning with the path of each key it doesn't recognize. Misspelled keys and old keys that are no longer read both show up there.

#Adopted features go in code, only numbers go in settings

There is one rule. Adopted features go into the code, and the version decides whether a feature exists: install that version and the feature is on. Trying out a feature happens in development and test environments, and only features that are ready get merged.

What stays in the settings file (config.yaml) is operating numbers. Here are some examples:

• playbook.runRetentionMaxPerPlaybook: the number of run records to keep per playbook (default 100; 0 means no cleanup)
• observability.accessLogRetentionDays: retention in days for the access log audit (default 1,095; anything shorter refuses to start)
• timeouts.dbQuerySeconds: the time budget for one query (default 600 seconds)
• deployJudge.regressionFailureLimit: the number of turns failing with a regression after a deployment at which the deployment counts as failed (“Deploy success is judged by failed turns after the deploy, not by health checks”)

Numbers don't multiply into combinations. Whether you keep 100 run records or 1,000, the code path is the same and only the cutoff differs. So tests look at the boundaries where behavior changes, such as 0, and at the minimums.

The only boolean left among the model fields is whether a model accepts image input (imageInput). It isn't a switch an operator flips; it is a specification set by the weights being served. To use the same model at the fast processing tier (speed: fast), you add a second model row instead of a toggle. The two rows appear separately in model management, and setting fast on a model that doesn't support it refuses to start.

#Per-installation differences are data, and the edition is fixed at first boot

Whatever genuinely has to differ per installation is kept as data. A switch decides "whether to do this"; data decides "what to do it for." Which connector tools to block and which to require approval for depends on the servers each customer attaches, so administrators set it as per-installation tool policy (“Customers know their connector tools' risks, so tool policy is per install”). They edit it in the admin console, it is stored in the database, and tools with no matching policy follow default rules defined in code.

The difference between on-premises and OwlAI SaaS comes down to the edition. OWLAI_EDITION=onprem|saas in .env is read at startup and written to the database on the first boot. Every later boot compares it with the stored value and refuses to start if they differ. The edition is also part of the signed license, so editing .env alone won't start a different edition.

Other modules read only the capabilities that the edition profile defines. A capability is a named behavior such as "create multiple organizations," "send mail" or "update from the admin console." The frontend also opens menus based on the list of capabilities the server sends. Because the fork ends at two editions, two test environments, one on-premises and one OwlAI SaaS, are enough.

Infrastructure coordinates, secrets, service-to-service authentication tokens and database connection strings live in compose and .env instead of the settings file, because components that start before the code that reads settings have to read them. These values aren't switches that change behavior; they are the installation's address.

#A new boolean field fails a test

Left to reviewers' memory, this rule erodes. So two tests hold it in place.

The first checks boolean fields in the settings model. The test walks the settings model, including nested models, lists and dict values, collects every boolean field, and checks that the set equals an allowlist where each entry carries a reason. Today the list holds only imageInput. A new boolean field fails because it isn't on the list, and a field that was removed but left on the list fails too. That is why the test compares for set equality rather than a subset.

The second checks where the edition is used. Only the edition module in product code may know edition values, and the capabilities it declares must equal the capabilities the code actually uses. If any module compares against "saas" directly, the test fails.

#Design choices

• We chose fix releases and rollback over kill switches. A kill switch lets you turn off a broken feature at once without redeploying. But the switch is itself an untested branch, and a value turned on and off once stays behind in the customer's settings file. So a feature with a problem is handled by shipping a fixed version or rolling back to the previous one, and we keep the rollback procedure (“When an update fails, the schema change decides between rolling back code and restoring the dump”) and the deployment check (“Deploy success is judged by failed turns after the deploy, not by health checks”) strict.
• We chose testing before release over gradual rollout. Turning a feature on for a few customers first and widening it as you watch their response splits customer sites into experiment groups. New code is deployed automatically to the on-premises and OwlAI SaaS test environments every time it is merged, and only then does it ship in a version. The cost is slower releases, and a problem that only shows up with a customer's data is first seen at that customer.
• We chose a warning over refusing to start for unknown settings keys. Refusing would catch misspellings immediately, but a settings file with an old key left in it would no longer start the new version. We picked the warning log so that an upgrade doesn't wait for someone to clean up the settings file, and reading that log is left to the operator. Values that can't be used (an audit log retention shorter than the minimum, the fast tier on a model that doesn't support it) do refuse to start.
• The edition is one of two values rather than a boolean, fixed for the life of the installation. On-premises data (a single organization, accounts that administrators create directly) doesn't meet OwlAI SaaS's assumptions, so we closed the path to starting as a different edition by changing one value.

If customers edit your product's settings file themselves, it's worth asking who will hold a value before you add a boolean field. When the operations team holds it in one place, a flag stays a release tool. When it is scattered across customer servers, each field doubles the number of products nobody has tested.