Model-drawn bars can miss their numbers, so the screen draws charts

Ask a model to draw a bar chart as SVG and it converts each value to a height on its own scale and writes the coordinates. The answer can say Q3 revenue was 1,200 while the bar is drawn at the height of 1,020, and nothing compares the two. Users look at the picture before the numbers, so a wrong bar is a wrong answer.

So we decide who draws a picture by asking what breaks when that picture is wrong. Charts carry numbers, and the model does not draw them: it passes rows and axis names, and the screen draws the bars from those rows. Diagrams carry no numbers, so the model draws them as SVG and the server checks for overlapping boxes and overflowing text. Document pages, which the model looks at to make decisions, are drawn by the render sidecar (a dedicated rendering container that runs next to the backend) to match the real document, and anything it could not draw faithfully is reported in words. This post covers those three paths and how we contained the render sidecar that opens users' documents.

#The screen draws charts from the rows the model passes

The chart tool takes a chart type (bar, line, or pie), a list of rows, the field names to use for the x and y axes, a series field for line charts, and axis labels. It does not take coordinates, ticks, or colors. The server checks the input against the same rules as the screen's chart schema: the fields chosen as axes must exist in the rows, and cell values can only be strings, numbers, or empty. If a field name differs only in case or surrounding spaces, the server maps it to the real field and tells the model what it corrected. If the model picks a field that doesn't exist, the server returns which spot is wrong.

Once the input passes, the server issues a widget ID. The model places that ID as a marker wherever it wants in the answer, and the screen draws the chart from the same rows. Bar heights come straight from the row values, so the picture and the numbers cannot disagree. The rows go only to the screen; the tool result the model receives keeps just the ID. That keeps a 2,000-row chart from riding along in the next round's input.

The model could also skip the tool and write a plausible-looking ID. Before the answer goes out, the server compares the IDs in the body with the IDs it issued in this turn; if one is unknown, it sends the model back to actually draw the chart and use the ID it gets. Any marker still unknown at the end is removed from the answer. When the data has a shape no chart can draw, two consecutive failures switch the output to a table, which “Tools copy the form and load the guide; models fill the arguments” covers.

#The model draws diagrams as SVG, and the server checks their shape

A picture of a process flow or a system layout has no numbers to match. When it is wrong, boxes overlap or text spills out of its box, and the user sees it immediately. Redrawing fixes it. So the model draws new diagrams directly as SVG. Instead of colors it uses names such as box, arrow, and highlight, and the screen applies light- and dark-theme colors to those names. Colors don't vary from model to model, and text doesn't vanish in the dark theme. The cost is output tokens. One SVG runs 1,000 to 3,000 tokens, so the tool description tells the model to draw only when the user asks for a picture or when a flow of four or more steps is hard to follow in text.

The server does not accept the model's SVG as is. It copies only the elements and attributes it knows into a new document and drops scripts, elements that pull in outside content, links, and external addresses. Because it drops whatever it doesn't know, risky elements added in the future don't survive either. The screen shows the SVG only as an image, so no script runs there, but a user who downloads the SVG file and opens it in a browser gets it interpreted as a document. Sanitization is for that case.

The model must also say whether the picture is a flow or structure diagram. If it is, the server checks whether boxes overlap and whether text spills past its box. The server has no browser to measure text width, so it estimates width from the character count. When it finds problems, it returns up to a few of their locations and the model fixes the coordinates and redraws. We tilted this check toward missing problems, because a false alarm pushes a good picture on to the next step.

That next step is Mermaid. If the same diagram fails as SVG twice in a row, the server changes its instruction from "redraw" to "draw the same picture in Mermaid." In Mermaid the model writes only the nodes and links as text and Mermaid decides the layout, so overlaps and overflow can't come from the model's coordinate mistakes. A picture the check missed goes the same way when the user clicks Redraw with Mermaid. The screen draws Mermaid, at the strict security level that blocks HTML in labels and click actions; directives inside the code cannot change that level, and the result is again shown only as an image. If Mermaid fails twice as well, the model writes the steps as a numbered list instead of a picture.

#The render sidecar draws document pages to match the real document

The render sidecar draws page images of the Word, PowerPoint, Excel, and Hangul files the model creates and of documents users upload. Those images become the preview users see, and they are also attached to the model's next input (“Block Office files that won't open, then show the model its pages”). The model looks at them to decide what to fix. So what must not break here is that the picture matches the real document. If the picture shows a defect that isn't there, the model sets out to fix a defect that isn't there.

Conversion has two stages. A per-format converter produces a PDF, and poppler turns that PDF into PNGs.

HTML gets its own engine for the same reason. When soffice drew the same HTML, flexbox layouts collapsed vertically and CSS grid and table borders disappeared. WeasyPrint kept the horizontal card layout, the three-column grid, and the table borders, and the backend also uses WeasyPrint when it turns HTML into a PDF file. The preview and the actual output come from the same engine.

Sometimes the sidecar can't draw a document faithfully. An hwpx file the Hangul filter can't open is drawn with an approximate layout written in pure Python instead. Fonts, letter spacing, and page breaks then differ from the real document, and the page count is off; in one measurement a 13-page document came out as 18 PDF pages. Photos in a Hangul document are sometimes drawn as blank boxes. The render sidecar puts facts like these in the response's diagnostics list, and the backend adds one line per diagnostic to the model's input. Without diagnostics, the model reads the approximate layout as the real one and judges length from its page count. When only some pages are drawn, the response also uses actual page numbers rather than sequence numbers. Sequence numbers would make the model call page 7 page 2.

#The render sidecar opens documents on a network with no way out

Keeping the renderer as a library inside the backend process would save a container and a service-to-service call. We split it out anyway because soffice is hard to handle and because the renderer opens documents we can't trust.

soffice can wait forever on broken input. It has only one default user profile, so concurrent calls fight over the lock, and the loser quietly exits leaving an empty PDF. So each request gets a fresh profile, and when time runs out the whole process tree is killed. WeasyPrint is an in-process library whose threads can't be interrupted, so it runs in a child process under the same time limit. Memory is the other problem. The Hangul filter starts Java for each conversion, and one 11 MB hwpx full of images used up to 763 MiB. Without a cap, the process that dies of OOM isn't the renderer but the database or backend on the same host. So the container gets 3 GB of memory and 2 CPUs, and conversions are limited to two at a time.

A document can point to outside addresses, such as linked images or external data. If the renderer follows them, requests go out to addresses the document chose (SSRF), and those can include internal network addresses and cloud instance metadata addresses. So we stacked two defenses. Before handing an Office file to soffice, the render sidecar makes a copy with the relationships that point to external targets removed. A preview is a picture, so losing the links costs nothing, and a linked image is drawn as a blank box. And the render sidecar sits on an internal-only network with no gateway. The only other things attached are its callers, the backend and the worker; the database is not there. It is also on a different network from the execution sidecar, where code written by the model runs.

Permissions are narrow too. Requests carry a sandbox path instead of file bytes, and the render sidecar sees the sandbox volume read-only. The container's root filesystem is read-only as well. The port is not exposed outside, and callers are verified with a service-to-service token. Each render draws 30 pages by default and up to 100 with configuration. Even a 100-slide deck takes about 6 seconds to convert and 9 seconds to turn into PNGs, so the renderer is not the bottleneck. The cap is set by the model's image tokens: 100 pages fill the context with pictures.

#Documents it can't process get a 422 so nothing retries them

The response code tells render failures apart.

A broken document must not come back as 500. The backend's HTTP client retries 5xx responses, calling up to three times in all, so one broken file would run soffice three times and tie up both conversion slots, stalling other users' renders. Timeouts go under 422 for the same reason. Their known cause is broken input, and even when load is to blame, running the same document again right away only adds load. Unexpected exceptions stay 500, because retrying those may be worthwhile.

A render failure doesn't stop the turn. Only that file skips the picture check, and a warning is logged. The file itself has already passed validation and been saved, so the user still gets it.

#Design choices

What the chart prevents is a mismatch between bars and row values. The model still copies the row values from query results, so whether the rows are right is decided by the SQL and the tool result that produced them. The chart is a display inside the chat screen; it does not persist as a file and is not used in playbook steps.

Having the model draw diagrams as SVG first costs output tokens and relies on an estimated check. Because text width is estimated from the character count, overflow can slip through, and a missed picture is redrawn by Mermaid when the user clicks the redraw button. Drawing with Mermaid means giving up control over layout, and a Mermaid syntax error only surfaces when the screen draws it, at which point the code is shown instead of a picture.

Document rendering runs on LibreOffice alone, because in an on-premises install it lets one sidecar draw Word, Excel, PowerPoint, and Hangul documents. The cost is small differences from the real apps in font substitution and some features; large differences such as the approximate layout are reported to the model as diagnostics.

If your agent produces pictures, start by asking what breaks when a picture is wrong. If the numbers can be wrong, take the coordinates away from the model and accept only data. If only the shape can be wrong, let the model draw and have the server check it. If the model uses the picture as evidence, have the renderer return in words whatever it drew differently from the real thing.