Sentry Founder: AI Patch Generation Is 'Awful' Right Now

Key Concepts

• Agentic AI: AI systems capable of autonomous action and decision-making.
• Fair Source: A software licensing model that allows access to source code but restricts commercial redistribution.
• LLMs (Large Language Models): AI models trained on vast amounts of text data, capable of generating human-like text, translation, and other tasks.
• Trace ID: A universal request ID used to connect different pieces of data (errors, logs, etc.) for debugging purposes.
• Embeddings: Numerical representations of data (e.g., text) used to measure similarity and perform tasks like aggregation.
• Root Cause Analysis: Identifying the underlying cause of a problem or error.
• Codegen (Code Generation): Automatically generating code from a higher-level description or specification.
• Context Window: The amount of information an LLM can consider when processing a prompt.
• Recall: The ability of an LLM to accurately remember and use information from its context window.
• Eval (Evaluation): Assessing the performance and quality of AI models and systems.
• Detectors: Rules or algorithms used to identify potential problems or anomalies in data.
• GRR (Gross Revenue Retention): A measure of how much recurring revenue is retained from existing customers.
• TAM (Total Addressable Market): The total market demand for a product or service.
• PMF (Product Market Fit): The degree to which a product satisfies market demand.

Sentry's Origin and Evolution (2010-2015)

• Sentry was born out of the need to solve problems in a less mature tech landscape where databases and distributed systems were still challenging.
• It started as a side project by David Kramer, aimed at creating dev infrastructure and libraries to simplify his work and share with peers.
• The project gained traction organically, leading to a bootstrapped cloud service and paying customers.
• Sentry's early success was accidental, driven by its inherent value and the need to support paying users.
• By 2015, when Sentry received funding, it already had thousands of customers and significant revenue.

Open Source vs. Fair Source

• David Kramer initially embraced open source due to its collaborative nature and the desire to share code.
• Sentry later transitioned to a "Fair Source" license to prevent commercial exploitation by entities not contributing to the community.
• The Fair Source license allows free use of Sentry's software but prohibits selling a competing cloud service based on it.
• After two years, code released under the Fair Source license becomes Apache licensed, allowing unrestricted use.
• The goal was to balance open access with the need to fund Sentry's development and prevent free rides for commercial entities.
• "We want everybody to use our software. We don't want people getting like like a free ride like especially other ventureback companies to just like take our software and commercialize it because like we have to fund our own development."

Sentry's AI Journey

• Sentry's foray into AI was driven by the potential to enhance debugging capabilities and go beyond traditional error monitoring.
• The initial focus was on leveraging LLMs to improve error aggregation and root cause analysis.
• LLMs excel at large-scale pattern matching, making them well-suited for identifying similar errors and summarizing interconnected data.
• One early success was using embeddings to significantly improve error aggregation, augmenting existing techniques.
• Another breakthrough was the ability to summarize data and root cause problems in a way that mimics experienced engineers.
• "Century has this big network of data right like we have errors and then we have a bunch of other data like logs and some other stuff and we connect all that together via what's called a trace ID... Turns out it's even better with LLM because they're phenomenal at large scale pattern matching."
• A prototype system was able to accurately identify the root cause of a cryptic JavaScript error, validating the potential of AI in debugging.

Challenges and Limitations of AI in Software Development

• Current AI models, particularly for codegen, are not reliable enough for production use.
• Reliability is a major concern, with models often failing to produce correct or useful code even with specific context.
• LLMs struggle with context management and recall, often ignoring instructions or fabricating information.
• "If it can't see sort of prior art, if you will, to scope a problem, how is it going to possibly region through it?"
• The industry is still in the early stages of applying AI to software development, with a need for human-in-the-loop approaches.
• Despite the hype, AI is more likely to augment existing workflows than to replace engineers entirely.

Sentry's AI-Enabled Products

• Sentry is launching AI-enabled products focused on augmenting debugging workflows rather than full autonomy.
• One product aims to root cause bugs and generate patches, but the patch generation is acknowledged to be unreliable.
• The focus is on providing developers with better insights and potential solutions to accelerate debugging.
• Sentry is also exploring the use of LLMs as "judges" to evaluate traces and identify potential problems in traditional applications.
• "Everything we're doing is not designed for full autonomy... The patch generation is awful... The root cause is really good, somewhat reliable."

Monitoring Agents

• Monitoring agents is similar to traditional software monitoring, with errors, logs, and metrics remaining fundamental.
• Key differences include the need to track token consumption and understand the qualitative aspects of non-deterministic systems.
• Sentry is using trace semantics and logs to monitor agents, with a focus on curated data like token consumption.
• The company is exploring the use of LLMs to evaluate traces and identify potential problems in agent-based systems.
• "Agents are just services, right? ... How would you debug or monitor or like build technology? And it's the same thing."

The Future of Engineering Roles

• AI is likely to lead to some new engineering roles, but not a drastic shift in the overall landscape.
• Developer infrastructure teams will need to focus on enabling and supporting AI-powered tools.
• There will be a greater need for ML expertise within engineering organizations to evaluate and validate non-deterministic systems.
• "You need some degree of ML on staff because it you need some degree of qualitative expertise whereas like software is generally deterministic."

Investor Pressure and Market Dynamics

• Investors are generally pushing for faster progress in AI, but it's important to balance growth with product value.
• Many companies have achieved rapid growth with "shaky product value," raising concerns about long-term sustainability.
• Sentry is focused on being bold and exploring new opportunities, but also on building products with real value and retention.
• "You need to capture the opportunity and then catch up to the opportunity."

Shaky Product Value Examples

• Low-code/no-code platforms are cited as an example of "shaky product value," as they often fail to produce high-quality, maintainable products.
• The value proposition of these platforms is often based on the potential for future improvements rather than current capabilities.
• Sentry's own AI-powered patch generation is acknowledged to have questionable value, but it represents a bet on future potential.
• ChatGPT is presented as a counter-example, offering clear value extraction and widespread adoption.

Conclusion

David Kramer provides a nuanced perspective on the current state of AI in software development, balancing enthusiasm with skepticism. While acknowledging the potential of AI to augment debugging workflows and improve software quality, he emphasizes the limitations of current models and the need for human oversight. Sentry's approach is focused on building AI-enabled products that provide tangible value to developers, while remaining cautious about the hype and potential for obsolescence. The key takeaways are the importance of reliability, context management, and a focus on solving real-world problems rather than chasing unrealistic promises. The future of engineering roles will likely involve a greater emphasis on ML expertise and the ability to evaluate non-deterministic systems.