Mistral Small 3.1: New Powerful MINI Opensource LLM Beats Gemma 3, Claude, & GPT-4o!

Mistral Small 3.1 Model Summary

Key Concepts:

• Open-source Large Language Model (LLM)
• Multimodal and multilingual capabilities
• Parameter size and computational requirements
• Performance benchmarks against proprietary models (GPT-4 Omni, Gemini, Claude 3.5 Haiku)
• Context window size and token processing speed
• Local chatbot deployment and cloud-based access
• Code generation, image understanding, mathematical reasoning, and logical problem-solving
• SVG generation limitations

Introduction of Mistral Small 3.1

Mistral AI has released Mistral Small 3.1, a new open-source model under the Apache 2.0 license. This model outperforms Google's Gemma 3 (27B parameters) and competes with or surpasses larger proprietary models like GPT-4 Omni, Gemini Mini, and Claude 3.5 Haiku. Mistral Small 3.1 is a 24 billion parameter model, making it suitable for running on a single RTX 4090 or a macOS device with approximately 32 GB of RAM. It features a 128k context window and a processing speed of 150 tokens per second.

Model Capabilities and Performance

Mistral Small 3.1 excels in various tasks:

• General Knowledge: Demonstrates strong general knowledge.
• Reasoning and Problem Solving: Exhibits strong reasoning and problem-solving capabilities.
• Multimodal Understanding: Shows surprisingly better multimodal capabilities than GPT-4 Omni.
• Multilingual Support: Supports over 21 languages.
• Long Context Mastery: Pre-trained for long context understanding.
• Benchmark Performance: Performs well on benchmarks like MMLU and GPQA.

The model is versatile for programming, math reasoning, dialogue, long document understanding, visual understanding, summarization, and low-latency applications.

Accessing and Deploying Mistral Small 3.1

• Le Chat: Mistral's chatbot platform allows easy interaction with the Small 3.1 model.
• Hugging Face: The model can be installed locally via the Hugging Face model card.
• Google Cloud Vertex AI: Access and fine-tune the model within a Google Cloud notebook.
• OpenRouter: Access the model through platforms like OpenRouter.
• LM Studio (AMA): While not immediately available, the model will be accessible through LM Studio once the model card is released.

Installation via Command Line (Example):

1. Copy the Mistral Small 3.1 model card command.
2. Ensure LM Studio is running in the background.
3. Open a command prompt and paste the model card command to install.
4. Open the model within LM Studio's web UI.

Performance Evaluation with Prompts

The video assesses Mistral Small 3.1's performance across different categories using various prompts.

1. Web Page Generation (HTML):

• Prompt: Build a web page for users to log monthly expenses and income.
• Result: Successfully generated a basic web app for a monthly budget tracker, more sleek than Gemma 3's output. The app allows adding transactions, income/expense types, salary, amount, and description. A chart generation failed initially.
• Verdict: Pass

2. Image Detection (Multimodal):

• Prompt: Detect the red car in an image containing one red car.
• Result: Correctly detected one red car quickly.
• Verdict: Pass

3. Image Description (Multimodal):

• Prompt: Describe an image of a dog in a forested area during winter.
• Result: Accurately described the scene, including the dog's breed, coat, and surroundings.
• Verdict: Pass

4. SVG Generation:

• Prompt: Create an SVG representation of a butterfly.
• Result: Initially generated a box instead of a butterfly. A second attempt produced a nonsensical output.
• Verdict: Fail (Worst SVG generation ever seen)

5. Mathematical Equation Solving:

• Prompt: Solve a quadratic equation for x.
• Equation: Not specified in the summary, but the task was to solve for x.
• Result: Correctly solved the equation, providing the answers x=3 and x=1.
• Verdict: Pass

6. Logical Reasoning Word Problem:

• Prompt: A farmer has 10 cows, 5 goats, and 2 chickens. Each cow gives 10 liters of milk daily, and each goat gives 3 liters. How much milk does the farmer collect in a week?
• Result: Correctly calculated the total milk production as 805 liters per week, breaking down the calculation into daily milk production from cows and goats, total daily production, and weekly production.
• Verdict: Pass

7. Debugging Python Code:

• Prompt: Identify and fix a bug in a Python function that is supposed to return a new list containing only the even numbers from the original list.
• Result: Correctly identified the bug (unnecessary else condition), provided the fixed code, and ensured the entire list is processed before returning.
• Verdict: Pass

8. Physics/Science Question:

• Prompt: If you put a bowl of water in freezing temperatures, what will happen?
• Result: Correctly explained the phase change of water, including cooling, expansion, and ice formation.
• Verdict: Pass

9. Reading Comprehension:

• Passage: Alice went to the market, bought three apples, two bananas, and five oranges. She met her friend Sarah, who bought a loaf of bread and a bottle of milk. They went to the park and enjoyed their purchases.
• Question: How many oranges did Alice buy?
• Result: Correctly answered five oranges.
• Verdict: Pass

Conclusion

Mistral Small 3.1 is a lightweight, open-source model that performs impressively across various tasks, often outperforming larger proprietary models. While it shares some limitations with models like Gemma 3 (e.g., SVG generation), it provides better answers in many other cases. Its speed, multimodal capabilities, and multilingual support make it a valuable option for various applications. The model's accessibility through different platforms like Le Chat, Hugging Face, Google Cloud Vertex AI, and OpenRouter further enhances its usability. The presenter recommends trying Mistral Small 3.1 and staying updated with AI news through their newsletter and other social media channels. The model is faster than Gemma 3 and GPT-4 Omni Mini.