Why Your AI Is Underperforming (And How To Fix It)

Key Concepts

• Inference Setup: The process of configuring and running large language models (LLMs) for generating outputs.
• OpenWeight Models: LLMs with publicly available weights, allowing for local hosting and customization.
• GPT-OSS: OpenAI's open-source GPT model.
• API Providers: Companies offering access to LLMs through APIs (e.g., Azure, Amazon Bedrock, Fireworks).
• Benchmarking: Evaluating the performance of LLMs using standardized tests (e.g., GPQA, AME 2025, IFBench).
• Quantization: Reducing the precision of model weights to decrease memory usage and improve inference speed (e.g., 4-bit, 8-bit).
• Prompt Template: The structure and format of input prompts used to guide LLM generation.
• Harmony Format: OpenAI's recommended response format for GPT-OSS.
• Inference Settings: Parameters controlling the generation process (e.g., temperature, sampling strategy, reasoning effort).
• GGML: A tensor library and quantization method used in llama.cpp.
• Llama.cpp: A library for running LLMs on various hardware.

Performance Variance in API Providers

• Main Point: Different API providers hosting the same open-weight model (GPT-OSS) exhibit significant variations in performance (speed, price, and intelligence).
• Benchmarking Results: Artificial Analysis team benchmarked various API providers using GPQA, AME 2025, and IFBench.
  • GPQA: Navita and Parasel performed best, while Azure and Amazon Bedrock performed worst (up to 8% difference).
  • AME 2025: Amazon Bedrock showed almost a 10% reduction in performance, and Azure almost 13%.
  • IFBench: Deep Infra, Fireworks, and Nova performed best, while Azure was the least performant.
• Price and Speed: Grock and Cerebras are the most expensive but offer the highest throughput (Cerebras reaching almost 3,000 tokens per second). Cerebras also has the most variance in speed.
• Vendor Lock-in: Enterprises often use Amazon Bedrock or Microsoft Azure due to vendor lock-in, despite their lower performance.
• Peter Gstv: His analysis of the benchmark results is highlighted, recommending following him on X for generative AI insights.

Reasons for Performance Variations

• Quantization:
  • Reducing model precision (e.g., from 16-bit to 4-bit) impacts performance, especially in mixture-of-experts models.
  • OpenAI releases GPT-OSS in 4-bit floating-point precision.
• Prompt Template:
  • Inconsistent prompt formats used to be a major issue.
  • OpenAI's new "Harmony" response format is crucial for GPT-OSS; improper setup degrades performance.
  • OpenAI blog post emphasizes the importance of using the Harmony format.
• Inference Settings:
  • Parameters like temperature and token sampling influence output.
  • "Reasoning effort" is a critical parameter for reasoning models; incorrect settings cause performance variations.
  • Azure Issue: Lucas Bear pointed out that Azure initially defaulted all requests to "medium" reasoning effort, leading to poor performance.
  • Azure Fix: Lucas Pickup from Microsoft Azure AI team confirmed the issue and announced a fix implemented across all instances.

Locally Hosted Solutions

• Performance Differences: Even in local setups (e.g., LM Studio vs. Olama), performance varies significantly.
• Olama vs. LM Studio:
  • User reported slower performance with Olama's GPT-OSS 20B compared to LM Studio.
  • Llama.cpp Creator's Analysis: LM Studio uses the optimized upstream GGML implementation from llama.cpp.
  • Olama's Implementation: Olama's modifications to GGML (branching, MXFP4, attention sync) were inefficient.
  • Olama's Transition: Olama team is moving away from Llama.cpp and using their own implementation.
  • Context Length Issue: Initial Olama plots showed increased tokens per second with longer context, which was later attributed to fewer tokens generated in the responses.

Inference Complexity and Model Capabilities

• Inference is Hard: Setting up proper inference for open-weight models is complex due to numerous moving parts.
• Model Potential: The model's inherent capabilities might be masked by suboptimal inference configurations.
• Hugging Face's Perspective: Clement Delangue (Hugging Face CEO) acknowledges the difficulty of inference for new open models, especially with new formats like Harmony.
• Hugging Face Inference Providers: Hugging Face powers the official OpenAI demo using inference providers like Fireworks, Systems Croc, and Together AI.
• Benchmark Comparison: Peter Gtov's benchmark suggests GPT-OSS 120B is comparable to or better than GPT-4 on math problems and outperforms Grok-4.

Conclusion

The video emphasizes the challenges of achieving optimal performance with open-weight models like GPT-OSS. It highlights the significant performance variations across different API providers and even in locally hosted solutions due to factors like quantization, prompt templates, and inference settings. The key takeaway is that proper inference setup is crucial to unlock the full potential of these models, and users should carefully evaluate different providers and configurations to achieve the desired results. The video also suggests that GPT-OSS, when properly configured, can achieve performance levels comparable to or even exceeding that of more established models like GPT-4 in specific tasks.