Keeping old answers out of edited questions with a conversation tree

Say a user asks for "last month's top 10 customers by revenue," and the model answers from a table that holds only domestic customers. The user edits the question to "last month's top 10 customers by revenue, including overseas subsidiaries" and sends it again. If the next request still carries the old question, the domestic-only SQL and its result, and the old answer, the model can reuse the numbers already in its context. The edited question gets an answer with the old revenue figures mixed in, and that answer goes straight into a report.

OwlAI stores conversation context as a tree rather than a single list of messages. Each LLM call is stored as one node, each node records its parent, and together they form a conversation tree. The conversation record holds a single pointer to the node that currently ends the conversation. The next request is assembled by walking from that end node through each parent back to the root node. Editing a question therefore means attaching a new node under an earlier node and moving the pointer; there is no separate code that deletes the old answer. This post covers that structure, and how playbook conversion, failed calls, and conversation compaction are decided on the same structure.

#With a message list, every edit means cutting out the old answer

A message list is the natural first choice. The speech bubbles on screen form a list, and provider APIs take an array of messages. But the context sent to the model carries more than the bubbles: tool calls, their results, and reasoning blocks. Call everything from one question to its answer a turn, and one model call plus executing the tool calls in its response a round. A turn's list then gains items like these every round.

To edit a question in this list, you have to cut every item after the edited question. Getting the cut point wrong by one item is costly. If a tool call stays and its result is cut, the provider rejects the whole request with a 400 because a call has no result. From then on, every turn in that conversation fails for the same reason. The old answer that was cut is gone, so letting the user compare two answers means copying the whole list first.

The same problem shows up in three more places.

All four cases need separate code that copies the list or deletes part of it, and each piece of that code has to keep tool calls paired with their results.

#One call is stored as one node

So we made a single LLM call the unit of storage. Each node records these values:

• Parent node ID
• The items newly sent in this call (the user message, tool results from the previous round, and so on)
• The items the model returned (the answer, tool calls, reasoning blocks)
• Node kind (a conversation turn or a playbook conversion)
• Provider and model name, status (pending, succeeded, failed), and token usage

The node is created as pending right before the call and switched to succeeded, with the returned items, when the response arrives. The rounds of a turn become a line of nodes, and when the turn ends, the conversation record's end-node pointer moves to the last one. Deleting a conversation deletes its nodes with it.

Tool results go into the next node. The results of the tool calls that node N returned are in the items node N+1 sent. Because call–result pairs cross node boundaries, which nodes belong on the path decides whether the pairs match. That matters later, for failed calls.

#The next request is assembled by walking from the end node back to the root node

To send a call, OwlAI climbs from the end node through each parent to the root node. One recursive PostgreSQL CTE reads the whole path, and the sent and returned items of each node are concatenated in order from the root node down. Appending this call's new items completes the request.

If a failed or unfinished node sits on the path, its returned items count as empty. Its sent items stay and only the returned items drop, so any tool calls that node may have produced stay out of the context.

Nodes off the path never appear in assembly. However many branches sit under the same parent, only the single line from the end node to the root node becomes context. Creating a branch requires nothing to be copied or deleted.

The last step of assembly filters out tool calls with no result. On a normal path it changes nothing; when it does filter something, it treats that as a defect and logs a warning. “A 400 that sticks: add tool results on write, fix shapes on send” covers this pairing.

#Editing a question starts a new branch from an earlier node

Users can edit a past question and resend it, or use Regenerate below an answer to get a new answer to the same question. Both come down to moving the end-node pointer back to an earlier node. Editing a question returns the pointer to the last node of the turn just before that question, and the edited question creates a new node with that node as its parent. Regenerate goes back to the same spot and resends the original question unchanged.

In the revenue example, the domestic-only SQL, its result, and the old answer stay on the old branch. Requests on the new branch climb only from the end node to the root node, so they never pass through the old branch's nodes. The model picks tables again and writes new SQL as if it were receiving the question with overseas subsidiaries for the first time.

The old branch is not deleted. Where branches exist, the bubble shows a marker like "2/3" with left and right arrows, and pressing an arrow moves the end-node pointer to the last node of that branch. Users switch between branches to compare two answers and continue on whichever they prefer. The branch being continued is the context for the next request.

This behavior has two boundaries.

• The pointer never moves while a turn is running. A running turn attaches nodes below the end node it started from; if the pointer moved to another branch in the meantime, there would be no way to decide where it should go when the turn ends. So editing, regenerating, and switching branches are accepted only when no turn is in progress.
• Questions before a compaction node cannot be edited. Compaction nodes are covered below. Editing an earlier question would mean restarting from the pre-compaction original, and that branch would carry context of pre-compaction size again.

#Playbook conversion, failures, and compaction are also decided by parent and pointer

On the tree, playbook conversion, failures, and compaction all come down to two decisions: where the new node's parent goes, and where the end-node pointer moves.

Playbook conversion reads the full conversation context to write a draft, and when validation fails, it rewrites the draft from the errors up to three more times by default. Conversion nodes are marked with a kind different from a conversation turn, and each retry takes the previous conversion node as its parent, so they form a single line. The end-node pointer doesn't move, so draft JSON and validation errors never enter the context of the next question. When the user first opens the editing conversation for changing the new playbook through chat, the new conversation's end-node pointer is simply set to the original conversation's end node. Not one line of context is copied, yet the first editing turn starts on top of the original conversation's full context. If the original conversation keeps going afterward, its new nodes attach only on its own side, so none of it reaches the editing conversation.

When a call fails, the pointer moves to the failed node. That node's returned items are empty, but its sent items hold the previous round's tool results. Leaving the pointer on the last succeeded node would drop those results off the path, and if the user asked the same question again, the model would rerun web searches that were already paid for. With the failed node as the end node, the next turn sees the results it already has and continues from there.

The one exception is a node that failed from context overflow, which never becomes the end node. The overflow may come from that node's sent items, for example a large tool result over the size limit that went in as is. Making it the end node would assemble the same items on every following turn and block the conversation for good. So the pointer stays on the last succeeded node, and the failed node is left as a branch.

Compaction cuts the path. Compacting the context creates a new parentless node holding the compacted context, and the next call takes this node as its parent. We'll call it the compaction node. From then on, assembly stops at the compaction node and never climbs to the pre-compaction nodes. Those nodes aren't deleted; the compaction node keeps a separate pointer saying "the original before compaction is this node." It goes unused until the conversation's model is switched to a model from another provider that can't read the earlier compaction output, at which point assembly follows the pointer to unfold the original. “Compaction thresholds come from the input limit, not the context window” covers when and how compaction runs, and “A 400 that sticks: add tool results on write, fix shapes on send” covers unfolding the original on a model switch.

#Design choices

We chose to keep old branches. Answers from before an edit, failed rounds, finished conversion calls, and pre-compaction originals all remain, so we can reconstruct what the model saw when it answered. The cost is storage; nodes are deleted along with their conversation.

Assembly costs one recursive query per call. We pay to reread the path every time, and in exchange there is no code that copies or cuts lists. Because compaction nodes cut the path, even a long conversation never gets an endlessly long read path.

Making a failed node the end node is a choice to keep tool results already received. Context overflow is the exception. Blocking the whole conversation to save one round of results costs more, so in that case we accept losing that round's tool results.

Switching branches doesn't undo what already happened. If an email was sent or something was written to an external system on the old branch, that result remains. It's one reason tools that send something out get the user's approval before they run (“Told to confirm first, the model still sends the email, so approval happens right before execution”).

The speech bubbles on screen are stored separately from the tree, because display and model input hold different things. Bubbles don't carry raw tool results or reasoning blocks, and model input doesn't carry chart data used only on screen. Each bubble records the node it belongs to, and the screen draws only the bubbles on the path from the end node to the root node. Branch markers come from the same record. The price of two stores is that clearing a conversation has to delete both. Clearing only the screen while keeping the pointer lets the next turn pick up the old context, which looks to users like "I deleted it, but it still remembers."

If your model calls mix tool results and reasoning blocks, and you need question editing, separate calls like playbook conversion, or compaction on the same context, a tree with one node per call is easier to manage than a list. For a chat that only exchanges messages without tools, and can throw away everything after an edited question, a list is enough.