What changed when 163 employees asked AI 3,541 questions a month

#From a seven-person pilot to 163 people company-wide

On an average weekday, 152 questions reach the company's internal OwlAI. The company that builds OwlAI adopted it as its own work tool too, and this post records what it did over the four months that followed. Every number comes from the activity records of its internal production environment, with test accounts excluded.

Through the three spring months, no more than seven people used it as a pilot. Accounts opened to every employee in July, and monthly active users went from 89 in July to 135 in August and 177 in September. In October, 164 people used it.

In the 30 days up to the end of October, 163 employees asked 3,541 questions. In between, the company did two things. It connected business services one at a time, and it decided, and kept revising, who could use each service and how far.

#One or two services a month

The first business database, connected in June during the pilot, was the issue-tracking database. When OwlAI opened company-wide in July, the company connected email and calendar, issue tracking and the wiki, code repositories, and the product manual all at once. From late July it also registered eight shortcut link rules, so an issue or build number in an answer opens that record's detail page in the original system with one click.

After that, services were added one or two a month: the customer support ticket system in August, the CRM and an internal issue intake tool in September, and the test management system in October. Services such as email, where each person signs in with their own account, are connected by employees themselves, and 107 people have connected their own accounts so far. As departments gathered their material into document collections, 21 were added in September and October alone, bringing the total to 25, and 13 business databases are connected.

#Open tools selectively, widen as usage shows

Connecting a service did not mean opening all of its tools. For the internal wiki, only eight tools were opened in August; for the customer support system, only three. After a month of watching how they were used, both were widened to the whole service in September. The 69 tools that leave a mark outside OwlAI, such as sending email, creating calendar events, changing an issue's status, or deleting documents, are set to Approval required, so the user reviews each action in an approval prompt before it runs. When accounts for demonstrating OwlAI to outside visitors appeared in September, the company created a separate demo role that blocks eight external services and opens only the databases and document collections meant to be shown.

Each adjustment starts from user feedback. Forty-four employees left 172 answer ratings, and administrators read the rated conversations along with employees' requests and fix tool settings. Recurring requests become playbook work types, screens people keep looking for become shortcut links, and services people need are added as connectors. When someone asked to use the to-do list in the email and calendar service from playbooks, it became a work type in September, and the internal issue intake tool became a work type on the day it was connected.

#More people asking the company's databases

The number of people who queried business databases through the AI grew from 61 in July to 94 in August and 115 in September. In the last 30 days it was 103, or 63% of everyone who used OwlAI. The table below breaks down the 3,428 questions processed in the last 30 days by the tools the AI used. One question can use several tools, so the shares add up to more than 100%.

Each team leans on different systems. Development and engineering teams look up a feature in the product manual, then query a test database in the same conversation to see how it actually behaves. Technical support summarizes a ticket's conversation thread, finds supporting evidence in product documents, and leaves it as an internal note. Sales pulls customer and deal status from the CRM and drafts messages to share with the team. Recurring work goes into playbooks that can be run again: the 236 playbooks employees have built ran 283 times in the last 30 days.

#An operating playbook to borrow

• Connect one business database while a few people pilot it, and open company-wide accounts after that.
• Add one or two services a month, open tools selectively, and widen access as usage shows.
• Set tools that leave a mark outside OwlAI to Approval required.
• Give accounts used by outsiders their own role that opens only the data meant to be shown.
• Read answer ratings and requests regularly, and feed them into tool settings, work types, shortcut links and connectors.