Same LLaMA 4 Models, 5 Providers—Very Different Results!

Key Concepts

Llama 4 Scout, Llama 4 Maverick, API providers (Meta AI, Open Router, Grock, Together AI, Fireworks AI), inference providers, temperature, output length, tokens, context window, benchmarks (Live Codebench, Scoding, Human Eval), 8-bit floating point precision, hyperparameters, code generation, model performance, repetition penalty, independent benchmarking.

Llama 4 Scout Testing Across Different API Providers

The video details a test of the Llama 4 Scout model across five different API providers (Meta AI, Open Router, Grock, Together AI, and Fireworks AI) using the same complex HTML code generation prompt. The goal was to determine if the same model hosted by different providers yields the same performance, considering variations in price points and hosting configurations.

Prompt Details

The prompt requested an HTML program displaying 20 balls bouncing inside a spinning heptagon, with specific instructions regarding ball radius, numbering, initial drop point, colors, gravity, friction, realistic bouncing off rotating walls, bounce height limitations, rotation with friction indicated by numbers, and heptagon spinning speed (60° per 5 seconds). The output was to be a single HTML file.

Testing Parameters

• Model: Llama 4 Scout (due to limited Maverick availability across all providers)
• Temperature: Set to 0 where possible (to minimize randomness)
• Output Length: Minimum 4,000 tokens (Grock limited to 8,000)
• Hardware: Assumed 8-bit floating point precision across all providers

Results by Provider

• Meta AI: Failed to generate the complete code due to word limits. After prompting to continue, the generated code resulted in balls stopping upon touching the heptagon's sides, halting the rotation.
• Open Router: Generated code at 66 tokens per second. Interestingly, the model provided an "improved version" of the code. The improved version failed to draw the heptagon correctly, and the balls remained stationary. The original version also failed. Open Router doesn't allow setting hyperparameters like temperature.
• Grock: Extremely fast generation (500 tokens per second). The initial output had the same issue as Meta AI: balls stopping upon touching the sides. An updated code produced the same result.
• Together AI: Generated code followed by repetitive, nonsensical text ("Have fun with your bouncing balls best regards hope you like it let me know cheers thanks bye see you have a great day enjoy right and it goes on and on and on and just start saying best cheers thanks"). The generated code produced a distorted heptagon and multiple versions of the balls. Adjusting the temperature to zero didn't resolve the issue.
• Fireworks AI: The initial code had the same issue as Meta AI and Grock. An updated code also failed, and the simulation disappeared entirely.

Interim Conclusion

The results indicated performance differences even when using the same model (Llama 4 Scout) across different inference providers.

Llama 4 Maverick Testing

Due to the unsatisfactory results with Llama 4 Scout, the video then tested the larger Llama 4 Maverick model on Open Router, Together AI, and Fireworks AI, using the same prompt. Meta AI was excluded due to previous limitations, and Grock didn't host Maverick.

Comparison with Cloud Sonnet 3.7

Before testing Maverick, the presenter showed the output of Cloud Sonnet 3.7, which, while not perfect (no bouncing due to friction), correctly implemented the initial conditions and ball rotation upon collision.

Results with Llama 4 Maverick

• Open Router: Generated code at 60 tokens per second. The output was significantly better than Llama 4 Scout. The balls exploded from the center initially but then followed the prompt instructions reasonably well.
• Together AI: The model generated code at a faster rate. However, the output was poor; the balls dropped from the middle and fell off the heptagon. Adjusting the repetition penalty was not explored to maintain a fair comparison.
• Fireworks AI: Generated code at 123 tokens per second, the fastest among the Maverick providers tested. However, the balls fell very slowly and stopped upon touching the sides.

Further Testing with Together AI

The presenter attempted to replicate Fireworks AI's hyperparameters on Together AI, but the results were similar to the previous Together AI test: the balls dropped and fell out of the frame.

Analysis and Benchmarking Discussion

The video highlights the importance of independent benchmarking and not relying solely on aggregate scores. Different benchmarks (Live Codebench, Scoding, Human Eval) can yield varying rankings for the same model. For example, Llama 4 Maverick ranks higher on some coding benchmarks than others. The presenter emphasizes the need to understand what each benchmark measures and to choose a model based on the specific application.

Context Window and Output Token Considerations

The video discusses the context window limitations imposed by different API providers, even though the Llama 4 models theoretically support large context windows (Maverick: 1 million tokens, Scout: 10 million tokens). Open Router's provider list shows varying maximum output lengths and context windows. For coding tasks, a larger context window is crucial.

Conclusion

The video concludes that when selecting an API inference provider for the Llama 4 series, it's essential to:

1. Test multiple providers with your own benchmark dataset.
2. Consider the maximum context window and output tokens offered.
3. Evaluate different price points and speeds.

The presenter also notes that Open Router offers a free version with a decent context window, although it's rate-limited and hosted in 8-bit precision. The video emphasizes the need for users to conduct their own testing and benchmarking to determine the best provider for their specific needs.