Self Driving Products: Product Signals to Pull Requests — Joshua Snyder, PostHog

Key Concepts

• Observability-to-PR Pipeline: An automated system that converts product signals (errors, logs, session replays) into actionable GitHub pull requests.
• Signal Normalization: The process of converting disparate data types (JSON logs, stack traces, chart results) into a unified structure for analysis.
• Agentic Research: Using LLM-based agents (Claude SDK) in sandboxed environments to investigate and diagnose product issues.
• MCP (Model Context Protocol): A standard used to connect AI agents to external data sources (e.g., Linear, Notion, logs) to provide context for decision-making.
• Embedding Space Clustering: A technique for grouping related signals; the speaker highlights the importance of using LLM-generated queries rather than raw signal embeddings to avoid grouping by structural similarity instead of semantic meaning.
• Actionability Thresholds: A filtering mechanism to determine if a problem is ready for an automated fix or requires human intervention.

---

1. The Problem: The "Slow" Observability Loop

The current standard for software observability is reactive and manual:

1. Signal: An event occurs in the product.
2. Dashboard: A metric changes; the engineer notices it hours or days later.
3. Investigation: The engineer manually investigates the issue.
4. Resolution: The engineer creates a ticket, writes a PR, reviews it, and ships it.

PostHog aims to collapse this timeline by having a background agent perform the investigation and PR creation automatically, allowing engineers to focus on high-level feature development rather than bug maintenance.

2. The Pipeline Architecture

The pipeline consists of five distinct stages:

• Ingestion & Safety: Trillions of events are ingested monthly. An LLM classifier acts as a safety filter to drop malicious signals (e.g., attempts to trigger data leaks).
• Normalization: Signals are assigned a source, type, content, weight (importance), and embedding.
• Grouping: Signals are clustered into "reports." The team discovered that raw embedding models fail here because they group by structure (e.g., all errors together) rather than intent. They now use an LLM to generate queries based on the signal, which are then matched in embedding space.
• Research Agent: A sandbox (using Modal) runs the Claude agent SDK. It uses MCP servers to pull in logs, code context, and external project management data (Linear/Notion) to diagnose the root cause.
• Actionability & Execution:
  • Not Actionable: Returned to the pool for more data.
  • Human Input: Sent to an inbox for review.
  • Actionable: The agent clones the repo, writes a fix, and pushes a PR. If CI fails, the agent rehydrates the sandbox snapshot to iterate until the PR is "green."

3. Key Lessons Learned

• Evals are Mandatory: "Vibe checks" on local data are insufficient. Testing must be done on representative production data to ensure the pipeline is actually solving real problems.
• Avoid "Blind" Agents: If a problem is too generic (e.g., "Onboarding is broken"), an agent will try to fix something just to satisfy the prompt, leading to noisy, useless PRs. The system must verify if a problem is specific enough to be solved.
• Tokens are an Investment: While agents are expensive, running them repeatedly reveals patterns. Once these patterns are identified, expensive agentic steps can be replaced with cheaper, one-shot LLM calls or specialized models.

4. Notable Quotes

• "We're trying to turn observability data... into something that submits pull requests for you."
• "If you don't think about the structural similarity of your different sources when you're grouping them, then the grouping works really badly."
• "When you're waking up in the morning... you ideally wake up to just green PRs."

5. Future Roadmap

PostHog’s ultimate goal is a "product that builds itself." This includes:

• Automated Experimentation: Automatically shipping and measuring pricing or onboarding experiments.
• Feature Flag Integration: Deploying low-risk fixes behind feature flags automatically, with the ability to roll back if metrics decline.
• Feedback Loops: Learning from every outcome—if a PR is rejected or a deployment fails, the system uses that data to improve the quality of future generated code.

Conclusion

The transition from "observability as a dashboard" to "observability as an automated agent" represents a shift in developer productivity. By leveraging LLMs to bridge the gap between raw telemetry and code changes, PostHog is moving toward a model where the system handles the "boring" maintenance work, allowing engineers to focus on creative, high-impact development. The core takeaway is that successful automation requires rigorous data normalization, semantic (not structural) grouping, and a robust evaluation framework.