LLM answers vary by run, so E2E tests check what the server recorded

In a product with an LLM inside, an E2E test that compares the answer text with an expected string fails even when nothing is broken. The model words its answer a little differently every time, and it decides case by case whether to call a tool. A test that only checks that the answer isn't empty has the opposite problem: it passes when something is broken. "I couldn't process your request" is not an empty answer.

OwlAI's E2E scenarios check only facts that hold whatever sentence the model writes: the content of an answer to a question with one right answer, the server's record that the conversation made a tool call, and the calls and spend added to the usage ledger. This post covers what each scenario checks and where we moved the checks that don't need a model.

#Neither string matching nor an emptiness check gives a verdict

There are four common ways to test LLM responses. Each has its strengths, but none of them is enough to judge a scenario that runs a real model.

Mocking has its own place rather than being a method to throw away. Once you separate what holds only with a real model from what doesn't, the scenarios that run a real model have far less to check.

#Checks that don't need a model run without one

The model's answers vary, but the request we send to the model must be the same every time. Unit tests for the provider adapters compare the entire request body with a frozen value for every combination of provider, model, reasoning effort, tools, and streaming. If one field goes missing or changes, that combination fails. Permission decisions, tool argument validation, and usage calculation are fixed once their input is fixed, so we prove them the same way.

Where a test lives depends on what has to be running to prove it.

Using HTTP isn't by itself a reason to put a test in E2E. What remains in E2E are scenarios that need the screen or hold only when a real model answers. The rest of this post is about what those scenarios check.

#To check an answer, ask a question with one right answer

The basic chat turn scenario asks "Answer in one word: what is the capital of South Korea?" (in Korean) and checks that "서울" (Seoul) appears in the answer body. Whichever model answers, in whatever sentence, that fact is the same. Unlike an emptiness check, this scenario fails when the model returns an error message. A scenario that asks the same question in English from an English-language account checks both that "Seoul" appears and that the answer contains no Korean characters at all. The answer's language is also a fact you can check.

A refusal is judged by what is missing from it, not by its wording. The scenario that asks about a salary table in a database the role isn't allowed to use checks three things.

• The answer contains a human-readable refusal. An answer saying there is no permission and an answer saying no such database exists both count as refusals.
• The answer does not start with the model's raw JSON.
• No large number, such as a salary amount from the test data, appears in the answer.

The last check is the point of the scenario. The refusal sentence differs from model to model, but whether the data stayed out of the answer comes down to whether the number is there.

#Whether a tool was called is checked by matching the server record with the screen

The model decides whether to call a tool, so the scenario never asserts "it should have called a tool." When the turn ends, the scenario reads the conversation's messages through the API to see whether a tool call happened, then checks that the screen agrees with that record. If there was a tool call, the Create Playbook link must be clickable. If not, the link must be disabled with the note "Playbooks can only be built from conversations that include tool calls." Whichever path the model takes, the test passes when screen and server agree.

Some features can be checked only when the model uses a particular tool. The card where the model asks the user back with a set of options is one of them. That scenario's prompt tells the model to use the ask-back tool, and if the model answers directly without it, the scenario sends the prompt once more. It runs at most two real turns, and if the card still doesn't appear, the scenario fails. The verdict is whether the card shows the recommended option already selected, and whether submitting that option ends the same turn with that answer.

#Billing is checked in the usage ledger

Whether a turn was billed is read from the usage ledger, the record that stores one row per LLM call with its model, tokens, and amount (“Reconcile the LLM cost ledger with the bill, and count cache tokens by response format”). The scenario reads the member's LLM usage summary for today before and after the turn and checks that the amount went up. Only calls made with the system key carry an amount, so an increase means a ledger row was written and the turn was billed to the system key.

If the default chat model is a self-hosted model priced at zero, the amount doesn't move. In that case the scenario doesn't guess the price from the model name. It decides from two responses.

1. Find the model cells in the ledger whose call count went up between before and after the turn.
2. Read from the model options response whether each model is one the installation runs itself.
3. Pass if every cell that grew belongs to a self-hosted model, its amount is zero, and its tokens went up.

If a cell for an external API model grew with no amount, either a price is missing or the call was billed to a source other than the system key. The verdict then stays pending and fails when the wait runs out. Ledger rows can be written just after the turn ends, so the scenario keeps rereading the ledger for 30 seconds.

When the run finishes, the runner checks the ledger once more. It connects to the environment's database in a read-only session and totals calls and amounts by the conversation IDs created in this run. Rows where an external model was called with the system key and used tokens but cost zero are counted separately as integrity violations.

#When the model asks back, the turn is stopped and its question is kept

When information is missing, the model can ask the user a question instead of answering. A test that only waits for an answer to render sees this as a 180-second timeout, and the cause is hidden. So the scenario helper waits until either an answer renders or an ask-back card appears. If the card appears, it stops the turn and returns the model's question in the failure message.

There's a second reason to stop the turn. A turn that asked back waits up to 600 seconds for the user's reply while holding a slot in the queue. Left alone, later scenarios can end up waiting for that slot.

Before sending a follow-up question, the helper remembers the last answer on screen. Otherwise the previous answer, already visible, could be mistaken for the new one and the wait would end immediately.

#Verdicts come from the real test environments, after matching the deployed commit

The reference for E2E verdicts is two test environments: on-premises and OwlAI SaaS. Scenarios tagged for nightly runs go every night, and the full set runs every week. Expensive scenarios such as deep research run only weekly.

Before running any scenario, the runner checks that the commit deployed in the target environment is the commit of this run, and doesn't start if they differ. The test environment's deploy job and the E2E run share one lock, so the stack is never recreated mid-scenario. In the real environments a failed scenario is retried once, and a scenario that passes on retry is counted separately from a clean pass.

Some scenarios also run on the isolated stack, but only as a rehearsal. The isolated stack runs on a self-hosted model, so its spend is always zero, and its responses and speed differ from external providers.

#Quality is measured separately, and SQL is compared by its results, not its text

E2E scenarios look only at conditions with a clear right or wrong. Something that needs a score, such as the quality of SQL generated from natural-language questions, is measured separately on a measurement build. Questions from a public text-to-SQL benchmark go through the same intake path as a chat turn, and the results are read from the turn record and the usage ledger.

Grading follows the same principle. Many different SQL statements return the same answer, so instead of the generated SQL text we compare the result data with the reference answer.

• Column names, column order, and extra columns the question didn't ask for are ignored.
• Row order is compared only for questions whose reference SQL has an outermost ORDER BY; the rest are compared as sets of rows.
• Numbers are matched to two decimal places.

We also produce the result under the benchmark's official grading rules and look at the two numbers side by side. The gap between them shows how many right answers were counted wrong because of an alias or an extra helper column.

#Design choices

• Real-model scenarios run on a schedule instead of on every merge request. Running them on every merge would verify every change against a real model, but each scenario costs money and several minutes, and its result moves with the state of external providers. We can't let a provider outage decide whether a change merges. Instead, a change to the LLM call path (tool loop, kernel, provider adapters) merges only after one real turn passes. A green unit test that mocks the whole kernel can't tell you whether the real call path works.
• We check fixed facts instead of LLM grading. A grading model can tell whether the meaning matches, but if its verdict varies from run to run, a failed scenario can't tell you whether the product broke or the grader wobbled. The cost is that scenarios don't see answer quality; the measurement in the previous section covers that.
• Scenarios that fail often are judged against the pre-change state. To separate failures from a transient provider outage or rate limit from real regressions, we rerun the pre-change commit in the same environment and compare the failure lists.

If you're writing E2E tests for features that call an LLM, start by pairing each thing the model decides with the trace that decision leaves. A tool call leaves a trace in the server's records, billing in the ledger, and data exposure in the numbers inside the answer. Scenarios check those traces instead of the wording.