LongCat (560B): Bye Sonnet! Why is NO ONE Talking about this!? This OPEN Model is AWESOME!

Key Concepts:

• LongCat: A 560 billion parameter large language model (LLM) developed by Meituan, a Chinese food delivery company.
• Mixture of Experts (MoE): An architecture where only a subset of the model's parameters are activated for each input, improving efficiency.
• Tool Calling: The ability of an LLM to use external tools or APIs to perform tasks.
• KingBench: A benchmark suite for evaluating the performance of LLMs.
• Microsass Fast: A Next.js boilerplate for building Micro-SaaS and AI side projects.
• FlashInfer: A library for fast inference.
• SGLang: A framework for building and deploying LLM-powered applications.
• Quantization: A technique for reducing the size and computational cost of a model by reducing the precision of its weights.

1. Introduction to LongCat

• LongCat is a 560 billion parameter large language model created by Meituan, a Chinese food delivery and local services platform.
• The core version, LongCat Flash, is based on a Mixture of Experts (MoE) architecture.
• LongCat outperforms models like Sonnet in various benchmarks and excels at tool calling.
• The model is open source.

2. Mixture of Experts (MoE) Architecture

• LongCat utilizes a Mixture of Experts (MoE) architecture for efficiency.
• Instead of activating all 560 billion parameters for every prompt, the model dynamically activates only the necessary experts.
• Typically, 18 to 31 billion parameters are activated per token, averaging around 27 billion.
• This dynamic activation of experts of different sizes allows for faster performance compared to traditional MoE models with static activated parameters.

3. Accessibility and Deployment

• LongCat can be used for free on the LongCat website without requiring an account.
• A "thinking variant" of the model is under development.
• Deployment requires significant resources, approximately 8 H200 clusters.
• Shu offers LongCat for inference at a cost of 19 cents per 1 million input tokens and 80 cents per 1 million output tokens.
• The model was tested on Lightning AI using 8 H100s, with deployment being relatively straightforward.
• SGLang officially supports LongCat in its latest version, while VLM support is pending.
• FlashInfer is used for fast inference.

4. KingBench Performance

• The KingBench tests were completed in under a minute.
• The floor plan generation was functional but not exceptional.
• SVG creation was a strong point, with the panda SVG example being highlighted as "awesome."
• The Pokeball in 3JS failed to render.
• The chessboard with autoplay worked well, following the rules of chess, although the moves were not always optimal.
• The Kandinsky Minecraft clone was functional but visually glitchy due to the Kandinsky style.
• The butterfly image generation resulted in a blank screen.
• The CLI tool for image conversion worked without issues.
• The Blender script for creating a Pokeball was unsuccessful.
• LongCat achieved fourth place on the KingBench leaderboard, performing similarly to Deepseek GLM.

5. Tool Calling Capabilities

• LongCat is noted for its strong tool calling capabilities.
• The model was tested for AI coding tasks.
• A movie tracker mobile app was created using Expo, demonstrating successful tool calling and code generation.
• The app allowed users to search for and view movies.
• The process was largely a one-shot generation with minimal error correction.
• A similar movie tracker app was created using Next.js, also with positive results.

6. Critique and Recommendations

• The speaker expresses surprise that a food delivery company (Meituan) developed such a capable LLM, contrasting it with the perceived underperformance of GPTOSS from OpenAI.
• The lack of support from inference providers is criticized, with the speaker hoping for wider adoption.
• OpenRouter is encouraged to add LongCat to their platform.
• Users are encouraged to try LongCat via Shu or by deploying it on GPU cloud services like Lightning AI.
• The speaker hopes for quantization and Olama support to make the model more accessible.
• An official API from LongCat is desired to facilitate broader usage.

7. Sponsor: Microsass Fast

• Microsass Fast is a Next.js boilerplate designed to accelerate the development of Micro-SaaS and AI side projects.
• It includes Clerk, Stripe, Resend, PostgreSQL, and AI instructions.
• It claims to reduce hallucinations by 90% for vibe coding.
• It offers easy back-end integration with Python, Node, and Go.
• It is designed to save developers 50+ hours of setup time.

8. Conclusion

• LongCat is a powerful and impressive LLM developed by Meituan.
• Its MoE architecture, strong tool calling abilities, and open-source nature make it a significant development in the field.
• Despite its capabilities, wider adoption is hindered by limited availability on inference platforms and the lack of quantization and official API support.
• The speaker encourages viewers to explore LongCat and hopes for increased support and accessibility in the future.