Why Can't Anyone Answer Questions About the Business? — Garrett Galow, WorkOS

Key Concepts

• Studio: An internal workspace/agentic platform developed by WorkOS that allows non-technical team members to query business data and build reusable dashboards (widgets) using natural language.
• Agentic Workflow: The use of LLMs (specifically Claude 3 Opus) to orchestrate data retrieval, query generation, and UI construction.
• Widgets: Self-contained, declarative code blocks (UI, APIs, and queries) generated by the LLM that function as persistent, reusable tools.
• Context Injection: The practice of providing the LLM with specific schema information and business logic (e.g., join rules) at runtime to ensure accurate query generation without needing a full RAG (Retrieval-Augmented Generation) database.
• Pre-flight Validation: A mandatory step where the agent executes and validates queries against live data sources before finalizing them into a widget.

---

1. The Problem: Bottlenecks in Data Access

WorkOS identified a common organizational friction point: non-technical teams (support, marketing) rely on engineers to answer business questions. This creates a "request-wait-refine" loop that is inefficient and does not scale. Traditional dashboards are often too rigid to answer ad-hoc questions, leading to one-off Slack conversations that lack long-term utility.

2. The Solution: Studio Architecture

Studio acts as an internal interface where users can ask questions in natural language. The system processes these requests through the following stack:

• Orchestration: Uses LangGraph to manage the agentic flow.
• Intelligence: Powered by Claude 3 Opus, chosen for its high performance and reasoning capabilities.
• Data Sources: Connects directly to internal systems like Snowflake (database), Linear (project management), and Notion (documentation).
• State Management: Uses Convex to persist session state and tool configurations.

3. Step-by-Step Methodology

1. Prompt Parsing: The user submits a question via the Studio dashboard or Slack bot.
2. Pre-flight Checks: The agent verifies tool connectivity and determines if it has sufficient context. If not, it asks clarifying questions.
3. Context Injection: At the moment of tool invocation, the system injects specific schema details and "quirks" (e.g., complex join requirements) into the prompt to guide the LLM.
4. Query Execution & Validation: The agent writes and executes the SQL/API call. It validates that the query returns actual data (not just a valid syntax that returns zero results).
5. Widget Generation: If the user requests a persistent tool, the agent generates a "widget"—a piece of declarative JavaScript code that encapsulates the UI and the logic.
6. Deployment: Once generated, the widget is a standalone tool. Subsequent refreshes execute the code directly, bypassing the LLM to ensure reliability and speed.

4. Key Arguments and Perspectives

• Trust but Verify: While LLMs are highly effective at generating queries, they can miss business-specific filters (e.g., "only include active users"). The speaker argues that providing these rules in the context block is more effective than relying on RAG or model training.
• Declarative Code over LLM-in-the-loop: By converting LLM output into declarative code (widgets), the system becomes deterministic. The LLM is only involved during the creation or modification phase, not during daily usage.
• Prioritizing Quality over Cost: The team prefers using high-end models like Opus because the cost of "hallucinated" or incorrect business data is higher than the API costs of the model.

5. Notable Quotes

• "We don't need to have some sort of platform team or data team building these dashboards that are going to be used and need to be constantly modified." — Garrett, on the value of self-serve tooling.
• "There's no RAG database for us in all of this. We're just invoking tools directly with just context on top." — On the efficiency of direct context injection versus complex retrieval systems.

6. Real-World Application: Radar

The team uses Studio to manage Radar, their security product. Support staff can input a user's email to see why they were blocked. The agent queries the database, joins the relevant tables, and presents the result. If the UI needs a tweak (e.g., fixing a column display), the user simply asks the agent to "fix the visual bug," and the agent updates the underlying code.

7. Future Roadmap: Governance and Access

• Org Connectors: Moving away from per-user authentication to "Org Connectors" (leveraging WorkOS's own "Pipes" product) to allow centralized management of data access permissions.
• Access Control: Implementing role-based access (e.g., read-only vs. admin) for different team members within the Studio environment.

Synthesis

Studio represents a shift from "building dashboards" to "building agents that build tools." By focusing on context injection and declarative widget generation, WorkOS has successfully empowered non-technical staff to perform complex data analysis without constant engineering intervention. The system’s reliability is maintained by treating the LLM as a developer that writes code, rather than a black box that interprets data in real-time.