MCPs are Boring (or: Why we are losing the Sparkle of LLMs) - Manuel Odendahl

Key Concepts

• MCP (Managed Code Program): A system allowing LLM applications to interact with external tools.
• Tool Calling: The process where an LLM identifies and uses external tools (APIs, functions) to fulfill a user request.
• Eval: A function or tool that executes code, often used as a powerful alternative to complex tool-calling setups.
• Code Generation: LLMs' ability to produce code (e.g., SQL, JavaScript) to perform tasks.
• Recursive Thinking: Applying LLMs to generate code that generates more code, creating a self-extending system.

MCPs and Their Limitations

• The Promise of MCPs: MCPs enable LLMs to interact with various tools (GitHub, Blender, custom APIs), offering vast possibilities.
• The Problem of Too Many Tools: Giving an LLM too many tools (e.g., 100) leads to confusion, incorrect tool selection, and schema misuse.
• Token Inefficiency: Tool calls often require repeating information already in the context and return large JSON responses with irrelevant data, wasting tokens and increasing latency.
• Example: CRM Query: Asking for OpenAI's contact information from a CRM tool can result in a massive response with data for all 36 companies, even though only one email is needed.
• Engineering Around Limitations: Optimizing for specific cases (e.g., misspelling of "OpenAI") leads to complex solutions resembling GraphQL, which the LLM may generate incorrectly.

Leveraging LLMs Beyond Tool Calling

• Passing Chat History and Memories: MCPs are restricted because they don't leverage the full potential of LLMs. Passing chat history and memories to tool calls can enable context-aware tool usage and reduce redundancy.
• User Interface for Tool Interaction: Instead of simple "allow/reject" prompts, provide users with a UI to edit tool arguments and results before execution.
• Example: Overdue Invoices: Allow users to edit the SQL query generated by the LLM to find customers with overdue invoices, correcting errors and refining the results.

Code as a Tool: The Power of "Eval"

• LLMs as Code Generators: LLMs are excellent at generating code, making code itself a powerful tool.
• The "Eval" Alternative: Instead of relying on predefined tools, use an "eval" function to execute code generated by the LLM.
• Example: SQL Query with "Eval": Asking "How many orders did customer John Smith place last month?" can be solved by having the LLM generate SQL code using "eval," which is more efficient and deterministic than traditional tool calling.
• Dynamic Tool Creation: LLMs can create tools on the fly based on the specific needs of the task.
• Example: Creating a View: After a successful SQL query, the LLM can create a view to store the query for future reuse, simplifying subsequent calls.

Building Richer Tools with Code and UI

• Libraries of Functions: Instead of exposing a generic GitHub tool, create a preconfigured GitHub library with specific functions that the LLM can use.
• UI Generation: Use LLMs to generate UI code (e.g., using a UI DSL) that allows users to interact with tools in a more intuitive way.
• Example: JavaScript Sandbox: A simple MCP with a JavaScript sandbox allows LLMs to create REST APIs and even entire CRM systems with a single "eval" call.

Recursive Thinking and Infinite Loops of Creation

• LLMs as Magical Genies: LLMs are capable of creating anything on demand, leveraging their vast training data.
• Recursive Code Generation: Use LLMs to generate code that generates more code, creating self-extending systems.
• Focus on Engineering: Approach LLMs as engineers, writing code to solve problems instead of relying on predefined agents and tools.

Conclusion

The speaker argues that the current focus on tool calling limits the potential of LLMs. By leveraging their code generation capabilities and embracing recursive thinking, developers can create more powerful and flexible systems that unlock the true magic of LLMs. The "eval" function, combined with UI generation, offers a promising path towards building richer and more interactive tools.