Horizon: OpenAI’s Secret Open-Weight Model?

Key Concepts:

• Horizon Beta: A new model on Open Router, potentially OpenAI's open-weight model.
• Open-weight model: A language model with publicly available weights.
• Context window: The amount of text a model can consider at once (Horizon Beta: 256,000 tokens).
• Tokens per second: A measure of model throughput (Horizon Beta: 37-140 tokens/second).
• Mixture of Experts (MoE): An architecture using multiple specialized sub-models (experts).
• Quantization: A technique to reduce the size of a model by using lower-precision numbers.
• FP4: A specific low-precision floating-point format.

Horizon Beta: Capabilities and Performance

• Impressive Output: Horizon Beta demonstrates impressive output quality, comparable to Google's Gemini 2.5 Deep Think, particularly in generating detailed and interactive outputs from complex prompts.
  • Example: Generating an image based on a detailed prompt, allowing zooming, time-of-day adjustments, and style regeneration.
• Coding Prowess: The model excels at coding tasks, generating solutions based on detailed instructions.
  • Example: Creating a Rubik's Cube solver that works flawlessly for smaller sizes (3x3) using Kociemba's algorithm. It also attempts to solve larger cubes, though with some limitations in scrambling.
  • Example: Generating a functional SAS website landing page with animations, dark/light theme switching, and working buttons (though with mock functionality).
• Recursion and Problem Solving: Horizon Beta can implement recursion, as demonstrated by its attempt to solve the Tower of Hanoi.
  • Limitation: The model exhibited an orientation problem with the blocks, solving the puzzle upside down.
• World Generation: The model can generate different planets with biomes, oceans, forests, deserts, and mountains, with adjustable quality settings.
• Failure Cases: The model struggles with certain complex prompts involving physics simulations.
  • Example: Failing to implement a heptagon with 20 bouncing balls colliding with the sides.
  • Example: Failing to simulate letters falling under the influence of gravity.
• Structured Output: Horizon Beta excels at generating structured output from unstructured text, which is crucial for building agentic systems.
  • Example: Extracting data from unstructured text based on a defined schema.

Alleged Open-Weight Model Leaks and Sizes

• Jimmy Apples' Discovery: Jimmy Apples claimed to have found OpenAI's open-source model shortly after its upload and saved configurations before it was removed.
• Two Model Sizes: Allegedly, there are two versions of the model: one with 120 billion parameters and another with 20 billion parameters.
• Mixture of Experts Architecture: The architecture is rumored to be a Mixture of Experts (MoE) with 128 experts.
• Quantized Weights: Yanjen, CTO of Hyperbolic, stated that the leaked weights were quantized and not pre-trained in FP4.

Open-Weight Model Landscape

• Dominance of Chinese Models: The current landscape of open-weight models is dominated by models originating from China.
• Artificial Analysis Intelligence Index: According to this index, Grok-1 is the best model, but most open-weight models listed are Chinese.
• Limited Non-Chinese Models: Out of the top 22 open-weight models, only a few are not Chinese, including Mr. Small and Meta Maverick.
• Potential OpenAI Release: There is anticipation for OpenAI to release an open-weight model, but the licensing terms remain uncertain (research or commercial).

Conclusion:

Horizon Beta on Open Router shows promising capabilities, particularly in coding and structured output generation, potentially indicating a significant open-weight model from OpenAI. Leaks suggest the existence of two model sizes (120B and 20B parameters) with a Mixture of Experts architecture, though the leaked weights are reportedly quantized. The release of an OpenAI open-weight model could diversify the current landscape dominated by Chinese models, but the licensing terms will be crucial.