Building the platform for agent coordination — Tom Moor, Linear

Key Concepts

AI in product development, issue tracking, vector search, embeddings, hybrid search, query rewriting, product intelligence, customer feedback analysis, AI agents, agentic systems, Linear platform, GraphQL API, webhooks, SDK, best practices for AI agent integration.

Linear's AI Journey: From Pragmatism to Agentic Systems

Early Explorations (Early 2023)

• Linear started exploring AI with a skunk works team focused on summarization and similarity using GPT-3.
• The team lacked prior AI experience and learned by experimenting.
• A key realization was the need for a solid search foundation for most AI features.
• Elasticsearch lacked a good vector offering at the time.
• Instead of using the many new vector database startups, Linear pragmatically chose PG Vector on GCP with OpenAI embeddings.
• Shipped v1 of similar issues using cosine embedding comparisons (considered naive in retrospect).
• Implemented natural language filters (e.g., "bugs assigned to me in the last two weeks that are closed"), which proved useful.
• Automated issue creation from Slack threads, seamlessly parsing text to create appropriate issues.
• Decided against shipping a co-pilot feature due to insufficient quality.
• Focused on small, pragmatic value adds rather than aggressively pushing AI.

Turning Point (End of 2024)

• GPT-4's release, improved planning and reasoning models, multimodal capabilities, and larger context windows (up to a million tokens) marked a turning point.
• Deepseek's emergence further fueled the sense that AI could be deeply integrated.
• Experiments became less brittle, and the team gained a clearer understanding of AI's potential.

Rebuilding Search and Product Intelligence

• Rebuilt the search index using a hybrid search approach.
• Moved to Turppuffer for search indexing.
• Switched embeddings to Cohere after a comparison, finding them better suited for their domain than OpenAI's.
• Developed "Product Intelligence," an enhanced version of similar issues.
• Product Intelligence pipeline: query rewriting, hybrid search, reranking, deterministic rules.
• The output is a map of relationships between issues, including suggested labels, assignees, and possible duplicates.
• For projects, it suggests the right person or project for an issue.
• Aimed at helping companies like OpenAI manage thousands of incoming tickets.

Customer Feedback Analysis and Summarization

• Leveraging LLMs to analyze customer feedback from various channels to inform product development decisions.
• AI analysis was reported to outperform 90% of candidates in product manager interviews.
• The system can process thousands of customer requests to identify features for a given project.
• Implemented a daily or weekly "pulse" that synthesizes workspace updates into a summary and an audio podcast version.

Issue Creation from Video

• Automates issue creation from customer video recordings by analyzing the video and identifying reproduction steps.

Agents as Infinitely Scalable Teammates

• Linear is positioning AI agents as infinitely scalable, cloud-based teammates.
• Launched a platform for agents, integrating them into the existing communication workflow.
• Agents are designed to live within Linear, alongside human team members.
• Codegen Example:
  • Codegen can be assigned and mentioned in Linear like any other user.
  • It produces plans and pull requests (PRs).
  • Users can review the PRs as they would from any other team member.
  • It can be interacted with from Slack or other communication tools.
• Bucket Example:
  • Bucket is a feature flagging platform.
  • The Bucket agent can create feature flags and roll them out.
  • Users can interact with it using natural language commands (e.g., "create a new flag roll it out to 30% of users").
• Charlie Example:
  • Charlie is a coding agent that performs root cause analysis of bugs.
  • It can analyze Sentry issues, recent commits, and the codebase to identify the cause of an issue.
• Linear is building additional surfaces in the product for agents, allowing users to see their observations and tool calls.
• Integration with Intercom's Finn agent to automatically reply to customers who reported a bug after it's fixed.
• Aims to reduce the number of bugs in company backlogs by enabling agents to tackle them.

Agent Architecture and Integration

• Agents are first-class users with identity, history, and a full audit trail.
• They are installed via OAuth.
• Admins can manage agents and their access.
• Agents operate transparently.
• Linear provides a mature GraphQL API that allows agents to perform any action a human could.
• Granular scopes control agent access.
• New webhooks provide notifications of agent-specific events (e.g., when an agent is mentioned).
• A new SDK is being developed to simplify agent integration.

Best Practices for AI Agent Integration

• Respond Quickly and Precisely: Agents should respond rapidly to triggers, using emoji reactions to acknowledge requests.
• Inhabit the Platform: Agents should use the language and conventions of the platform they are integrated with.
• Natural Behavior: Agents should respond naturally to replies in a thread without requiring repeated mentions.
• Don't Be Clever: Agents should clarify their intent before acting and avoid one-shot attempts.
• Form a Plan: Coding agents should form a plan and communicate it upfront for clarification.
• Add Value: Agents should be concise, useful, and avoid simply splatting raw LLM output into comments or issues.
• Emulate Human Behavior: Agents should strive to behave like a good teammate would.

Conclusion

Linear's journey with AI has evolved from pragmatic, small-scale integrations to a vision of AI agents as integral members of product development teams. By focusing on seamless integration, providing a robust platform, and emphasizing best practices for agent behavior, Linear aims to empower teams to build more, build better, and build faster. The key is to treat agents as first-class users within the existing workflow, enabling them to augment and enhance human capabilities.