Tool calls a script makes still get permission and approval checks

Take a request like "For each issue opened last week, look up the customer's contract, and email the owner only on issues whose customer's contract has ended." Each call builds its arguments from the previous result, so the calls form a chain. If the model calls the tools one by one, every link in the chain costs a round, and the bodies of dozens of issues and their contract details all pile up in the context. A round is one model call plus running the tool calls in its response. Hand the same chain to a single Python script and it takes one round, with intermediate results passed only between the script's variables. The industry calls this programmatic tool calling.

The catch is that, from outside, running a script looks like a single tool call. If permission checks, user approval and the audit log see only "ran a script," the twenty emails sent from inside it pass no check at all. So we send each tool call a script makes back to the turn that ran the script, and judge it in the same order as a call the model made directly. This post covers the path those calls take back, the order of checks, the tools a script cannot call, and the cap on call count. Passing large data by file path so the model never retypes it is covered in “Query results go to a file so the model never retypes them”.

#Approving the whole script hides what goes out inside it

There are two easy ways to let a script call tools.

The first is to give the script network access and service credentials and let it call APIs directly. It is the easiest to build. But the allow and approval settings an administrator sets per tool, the blocks that follow from user permissions, and the audit log all sit on the path where the server runs tools. A script that calls an API directly skips that path. That is why OwlAI's sandbox has no network (“Sandbox each script run, then prove isolation on the actual server”).

The second is to make running the script itself the thing that needs approval. The user reads the code and approves it before it runs. But what the user sees is loops and conditions, and how many emails that loop will send, and to whom, is only known by running it. When the approval prompt shows code instead of what will be sent, the user approves without being able to check the content. An administrator's split of "sending email needs approval, looking up issues is allowed" also means nothing in front of a single script run.

Another option is a separate list of tools that scripts may call. That makes two lists, and the moment a new tool lands in only one of them, the permission checks disagree.

So the tools a script can call come from the same list of tools the model can call in that turn, and every single call rides the same execution path as the model's calls.

#A script's calls come back to the turn that ran it

Scripts run in a sandbox inside a sidecar container. Each script has a call function built in that takes a tool name and arguments, and all that function does is pass the request out of the sandbox. One call travels this way.

1. When the script calls the function, the request goes through a pipe to the sidecar process outside the sandbox.
2. The sidecar puts the request on the response stream of the connection the backend opened when it handed over the run. The sidecar does not open a new connection to the backend.
3. The backend process running that turn receives the request, checks it, runs it, and answers the sidecar with the result.
4. The script gets the result and moves on to its next line.

Because whoever opened the connection owns the turn, a request cannot reach a different turn, and the sidecar needs no identity to prove itself to the backend. When the connection drops, the sidecar ends the script. A turn being cancelled, a turn failing and the backend process exiting all end in a dropped connection, so no script keeps running on its own after its turn is over.

#A script's call is checked in the same order as the model's

A call that comes back to the turn goes through these steps in order. A call refused at one step does not reach the next.

Step 3 judges and records in one place. The pre-execution check remembers each call that passes, refuses the same call when it comes again, and counts calls that need approval. If it judged without recording, a script could repeat the same call any number of times and slip past the cap on calls that need approval. So calls from the model and calls from a script go through the same procedure that ties judging and recording together.

The audit log also records which script run a call came from. Display names and error codes match those of direct calls, so the same refusal never shows up under two names depending on its path.

In chat, the bookkeeping after a call runs is the same too. When a turn is cancelled or fails, the user is told which approval-required actions may already have run, and an email sent by a script is on that list. When an approval-required action has already run, we also skip the recovery that calls the model once more after an empty answer, because a model called again could send the same email again.

A policy refusal, a denied approval, a tool failure and a cancellation all come back to the script as the same failure-shaped result rather than as exceptions, with the reason attached. It is the same shape the model would have received calling the tool directly, so the script decides what to do next from that one value.

#A script that calls an approval-required tool waits on that line

Scripts can call tools that need approval. When a call needs approval, the same approval prompt appears as when the model calls the tool directly, showing the arguments that will actually be sent (“Told to confirm first, the model still sends the email, so approval happens right before execution”). The script stops on that line until the user decides. The wait limit is also the same as for a direct call.

While it waits, the script uses no CPU. So for that time the sidecar gives back one of its concurrent execution slots, and takes one again when the result arrives. A script waiting a long time for approval should not hold up another user's document generation. The number of runs waiting on tool results has its own cap, and once it is full, a new call comes straight back refused with "call it directly in your next response."

Time is counted separately as well. A script run's time limit excludes the time spent waiting on tool results, so a normal script that calls a few long queries is not cut off for time its tools spent. A single tool call is cut off by that tool's own time limit, and the script's own computation by its CPU time limit (“Sandbox each script run, then prove isolation on the actual server”).

#Tools that take the user's answer cannot be called from a script

Code, not the model, decides which tools a script cannot call.

• A tool that asks the user a question. The user's answer is meant for the model to read and decide its next step, but a script returns only its final result to the model, so the model never sees an answer received inside the script.
• A tool that runs on the user's device, such as the desktop app's tools that read and write files on the user's PC, where the work makes a round trip to the device.
• The script tool itself, which prevents running a script inside a script.
• The tool that activates tools and the tool that calls other tools on the model's behalf. These calls change the turn's state rather than run anything.

Tools not yet activated cannot be called either. When there are many tools, some are listed by name only and activated when needed (“185 tool definitions cost 99K tokens, so we list names and load picks”). A script that calls one of them gets back a refusal saying to activate it outside the script first. A tool activated partway through the turn, on the other hand, can be called right away, because the tool list a script sees is the same list the turn keeps growing.

#Each script stops at 50 calls

One script can make 50 tool calls. One playbook run in production made 38 calls looking up records one at a time, and we set 50 as a width that holds chains like that. Past the limit the script gets a refusal, and the rest of its calls get the same refusal immediately, without a trip to the turn. The model reads the refusal and narrows the scope or splits the work across scripts.

Calls that will be refused are counted too, and counted first. If only calls that passed the checks were counted, a script that loops over a nonexistent tool name would run without limit and write an audit entry for every refusal. Time spent waiting on tool results does not count against the CPU time limit, so without this cap nothing would stop such a script. After the limit is passed, the refusal is written to the audit log only once.

Where a run has a cap on total tool calls for the turn, calls made by a script count toward it. The count includes calls still running in the same round, so a single script cannot get around the total, and the error code for exceeding it is the same as for direct calls.

#Design choices

Blocking approval-required tools in scripts would be simpler. No script would stop to wait for approval, and nothing would need to give back and retake execution slots. But then a chain that looks things up, picks, and writes breaks just before the write, and the model has to take the lookup results and issue the write calls itself. So we allow approval-required tools and accept, as the cost, that a script stops while waiting for the user's decision. Giving scripts the same wait limit as direct calls follows from the same reasoning. A waiting script gives back its execution slot, so it does not hold up other runs. The more write calls a script makes, though, the more approval prompts appear.

Intermediate results the script saw do not go back to the model. We chose that to save context, but in exchange the model cannot step in and judge partway through the chain. Each call is checked only for whether it respected permissions and policy, and whether the script's conditions were right is judged by the model and the user from the final result the script returns. That same property is why the tool that asks the user a question is excluded from scripts.

The cost of having only the backend open the connection is that run state lives in a single sidecar process. That process's memory holds the runs in progress and the calls waiting for answers, so running several sidecars would first need a new design for how answers get back. In exchange, the sidecar never needs credentials to call the backend, and when a turn ends, its script ends with it.

If your agent lets the model call tools from code, first check whether those calls travel the same execution path as the model's own calls. If permission checks, approval and the audit log see only one script run, send each call inside the script back down that path, and tie judging and recording into one procedure.