Design like Karpathy is watching — Zeke Sikelianos, Replicate

Key Concepts

• Andre Karpathy: AI researcher and educator known for his work at Google, OpenAI, Tesla, and Eureka Labs.
• Vibe Coding: A term coined by Karpathy, referring to an intuitive and experimental approach to programming.
• Software 2.0: A concept predicting that machine learning models will increasingly write code.
• MenuGen: An app created by Karpathy that generates image representations of menu items from text-based menus.
• Replicate: A cloud platform for running AI models via API.
• LLM.txt: A method of formatting documentation in a text-based or markdown format that is easily consumable by language models.
• Curl: A command-line tool for making API calls.
• Cog: An open-source tool for packaging machine learning models in production-ready Docker containers.
• OpenAPI: A specification for defining and documenting APIs using a JSON schema.
• MCP (Model-as-a-Composable-Procedure): A way of packaging an OpenAPI schema for use by language models, allowing them to interact with and control APIs.

Andre Karpathy and His Influence

The speaker introduces Andre Karpathy as a prominent AI researcher and educator. Karpathy's concept of "vibe coding" and his "Software 2.0" manifesto are highlighted as influential ideas in the field. The speaker emphasizes Karpathy's ability to explain complex AI concepts in an accessible manner.

MenuGen: A Case Study in Deployment Challenges

The talk centers around Karpathy's experience building MenuGen, an application that generates images of menu items from text descriptions. While Karpathy found the initial development "an exhilarating and fun escapade," he encountered significant challenges when deploying it as a real-world application. This experience serves as a case study for the difficulties developers face when moving from local development to production environments.

Replicate's Response to Karpathy's Feedback

Karpathy's blog post criticizing the developer experience of various platforms, including Replicate, is discussed. The speaker acknowledges the validity of Karpathy's criticisms and outlines steps Replicate is taking to improve its platform. The issues Karpathy faced included:

• Outdated LLM knowledge of Replicate.
• Out-of-date documentation.
• API changes.
• Rate limiting.
• Difficulties with new paid accounts.

Improving LLM Integration with LLM.txt

Replicate is addressing the issue of outdated LLM knowledge by embracing LLM.txt. This involves providing text-based or markdown versions of documentation that are easier for language models to parse than HTML. The speaker emphasizes the importance of simple, text-based documentation with a copy-to-clipboard button. Replicate has implemented this by adding a feature to model pages that allows users to copy the page content as markdown or send the page directly to Claude for interaction.

Leveraging Curl for API Interaction

The speaker highlights the importance of curl as a standardized way for language models to interact with APIs. A curl command contains all the necessary information for an LLM to make an API request, including the HTTP method, JSON payload, credentials, and API endpoint.

Cog and LLM.txt Integration

Replicate's open-source tool, Cog, is used to package machine learning models. The documentation for Cog has been converted into an llm.txt file, allowing developers to easily integrate Cog models into their projects using language models. By dropping a reference to the llm.txt file into their editor, developers can leverage the language model to understand and modify the code.

The Primary Audience is Now an LLM

The speaker argues that the primary audience for products, services, and libraries is now an LLM, not just a human. This shift requires developers to optimize their documentation and APIs for machine consumption.

MCP (Model-as-a-Composable-Procedure) Explained

MCP is explained as a way of packaging an OpenAPI schema for use by language models. It allows language models to interact with and control APIs. Replicate has implemented an MCP server that can be easily installed and used with tools like Claude, GitHub Copilot, and Visual Studio Code. The speaker emphasizes that having a well-written and well-documented OpenAPI schema is crucial for enabling MCP functionality.

Addressing Payment and Abuse Prevention

The speaker acknowledges that Replicate's abuse prevention mechanisms inadvertently blocked Karpathy's account after he signed up and started making a large number of API requests. To address this, Replicate is working on implementing a system that allows users to pre-pay for credits, giving them more flexibility and preventing accidental blocking.

Key Takeaways: Feeding the Machines

The speaker concludes with a set of recommendations for improving the developer experience for language models:

• Accept Payments: Allow users to pre-pay for credits to avoid accidental blocking.
• Document Everything: Ensure that all features are well-documented and that the documentation is easily consumable by LLMs.
• Feed the Machines: Produce content in formats that language models can understand and consume.
• Use Boring Technology: Favor well-established technologies that language models are more likely to understand.
• Practice Good API Hygiene: Design APIs with language models in mind, ensuring that responses are concise and information-dense.

Q&A Highlights

• Generating Docs: The speaker recommends starting with an OpenAPI schema and using tools like Docsaur, Read the Docs, or Readme.com to generate documentation and SDKs.
• Discovery and Purchasing Decisions by LLMs: The speaker emphasizes the importance of making pricing and other key information available via API so that language models can make informed decisions.