OpenAI's Opensource OSS 120B & 20B (Fully Tested)

Summary of YouTube Video: OpenAI OSS Models

Key Concepts:

• OpenAI OSS (Open Source Series): OpenAI's family of open-source models.
• GBT OSS 120B: A large-scale model (120 billion parameters) for data centers and high-end devices.
• GBT OSS 20B: A medium-sized model (20 billion parameters) optimized for desktops and laptops.
• Apache 2.0 License: Permissive open-source license allowing free use, modification, and commercial deployment.
• Tool Use: The ability of the model to utilize external tools to accomplish tasks.
• Chain of Thought Reasoning: A technique where the model explains its reasoning process step-by-step.
• Mixture of Experts (MoE): An architecture where different parts of the model specialize in different tasks.
• Context Length: The amount of text the model can consider at once (128k tokens for both OSS models).
• Ollama & LM Studio: Platforms for running open-source models locally.
• Open Router: A platform for accessing models via API.

OpenAI's Open-Source Shift

OpenAI has released its first open-source model family, the Open AI OSS (Open Source Series), marking a significant shift in their approach. The OSS models are designed for customization and can run in various environments.

Model Variants and Specifications

Two variants are available:

• GBT OSS 120B (120 billion parameters): Designed for data centers and high-end desktops/laptops. It has 117 billion total parameters with 5.1 billion active parameters per token.
• GBT OSS 20B (20 billion parameters): Optimized for most desktops and laptops. It has 21 billion total parameters with 3.6 billion active parameters per token.

Both models have a 128k context length and are trained using techniques from OpenAI's internal models (like the GPT-3). They utilize a mixture of experts architecture.

Licensing and Accessibility

The models are released under the Apache 2.0 license, allowing for experimentation, customization, and commercial deployment. They can be accessed via:

• Ollama and LM Studio for local installation.
• OpenAI platform via API.
• Open Router (priced at 15 cents/1M input tokens and 60 cents/1M output tokens for the 120B model, and 5 cents/1M input tokens and 20 cents/1M output tokens for the 20B model).

Benchmarking and Performance

The models were evaluated across academic benchmarks, including coding, math, science, and agentic tool use. They perform competitively against proprietary models like GPT-3 with tools and GPT-4 Mini. While they lag behind GPT-3 in the "Humanities Last Exam" benchmark, they perform well compared to GPT-4 Mini and GPT-3 Mini. They also show strong performance in health, math, and GPQA benchmarks.

Real-World Reasoning and Safety

The models are designed for real-world reasoning tasks and feature strong tool use and chain-of-thought reasoning. They are also fine-tuned to prevent the generation of malicious content, adhering to OpenAI's content policies.

Testing and Examples

The 120B parameter model was tested on various tasks:

• Basic Reasoning: Successfully identified the number of "R"s in "strawberry" quickly.
• Front-End Coding (AI SaaS Landing Page): Generated a functional but outdated and lackluster design.
• SVG Code Generation (Butterfly): Failed to generate functional SVG code.
• Financial Advice (Retirement Planning): Provided a well-reasoned portfolio management proposal for a truck driver aiming to retire by 30.
• Financial App Design: Generated a better design when prompted to reason, highlighting the importance of reasoning for quality output.
• Unscrambling Words: Quickly unscrambled a word, demonstrating fast reasoning capabilities.

Analysis and Conclusion

While the OSS models are a welcome addition to the open-source community, the presenter expresses disappointment, as the performance doesn't match the leaked "Horizon Alpha" model. The presenter speculates that the leaked models might be variants for GPT-5, which could be priced very high. The presenter recommends installing the OSS models locally for offline access to intelligent AI. Despite the safety measures, the presenter encourages viewers to share their thoughts on the models' helpfulness.