Tens of thousands of orders: does AI really read them all?

#Most of the data can vanish without a single error

You ask an AI to look through this year's orders and find anything unusual. It queries 30,000 orders, points to a few, and reports: "After reviewing the full data set, I found 5 unusual transactions." No error appears, and no warning.

Yet the AI may have read only a tiny fraction of what it received. If the rest was dropped without any sign, "5 out of 30,000" is a report written from a few hundred records. In business AI, wrong answers of this shape are hard to catch.

#An AI can read only so much at once

An AI model has a limit on how much it can read in one pass, and a query against a company database easily returns more than that limit in one go. We tested what happens then with a commercial AI service. We sent a 1.29-million-token query result to a model with an input limit of 272,000 tokens (a token is the unit AI uses to count text). The service returned a normal response with no error. A closer look showed it had cut 94.4% of the result and passed only the remainder to the model, which answered as if it had received everything.

A feature that summarizes long conversations does not help here. Summarizing means reading the original first, and this single query result already exceeds the limit.

#The wrong answer looks exactly like a right one

Nothing on the surface gives this answer away. There is no error on screen, so users have no reason to doubt it, and the answer is written in confident sentences. Without safeguards, requests that carry very large query results are actually observed losing most of their data and still going out as normal answers. That is why OwlAI added the safeguards below.

#Reduced before the AI reads it, and labeled as reduced

OwlAI sets a size limit on each tool result (the data the AI gets back when it runs a task such as a database query). A result over the limit is reduced in a way that keeps its beginning and its structure, and the AI is told plainly that it was reduced. The AI is also instructed not to talk as if it saw everything, to state how many records it actually received, and to say that the rest is missing.

The original is not thrown away; it is kept as a file. Instead of running the same query again, the AI reads that file in as many parts as it needs, or searches it for specific values, and carries on with the answer. If the user asked for the query result itself as a file, the original file is attached for download unchanged. On screen, Step details also shows a notice that only the beginning is displayed because the result is long.

#Three questions to ask before you adopt

If your work involves large data sets, ask these three questions before you bring in an AI:

• When the data it receives is too large, does the AI know?
• Does it tell users it saw only part of the data?
• Is there a way to get back to the part that was cut?

OwlAI answers yes to all three. If an AI's answer to even one of them is vague, double-check any big number it gives you.