OpenAI o3 & o4-mini shocking abilities

Summary of YouTube Video: OpenAI Models 03 and 04 Mini Review

Key Concepts:

• 03 and 04 Mini: New OpenAI models, claimed to be smarter and more capable with full tool access.
• Agentic Tool Use: The ability of the models to autonomously pick and use various tools and agents to solve tasks.
• Image Analysis Tool: A tool that allows the models to analyze images and extract information.
• Hallucination Rate: The rate at which the models provide incorrect or nonsensical answers.
• Benchmarks: Standardized tests used to evaluate the performance of AI models.
• Gemini 2.5 Pro: Google's AI model, used as a comparison point for 03 and 04 Mini.
• GPT-4o: OpenAI's multimodal model, used for image generation.
• Tiff File: A layered image file format.

Model Differences and Capabilities

• 03: Described as OpenAI's most powerful reasoning model, excelling in coding, math, science, and visual perception.
• 04 Mini: A smaller, more cost-efficient model optimized for fast reasoning, also proficient in math, coding, and visual tasks.
• Both models are trained to use tools through reinforcement learning, granting them agentic tool use capabilities.
• They can autonomously select and utilize various tools and agents, such as web search, data scraping, and coding agents.

Image Analysis Examples

• Restaurant Menu: 03 successfully identified a restaurant's name and location from a blurry photo of its menu, despite the absence of explicit restaurant information in the image. It used multiple agents to analyze the image, crop it, search the web for menu items, and cross-reference information on Tripadvisor.
• Maze Solving: 03 solved a maze by using Python code to load the image, identify the entrance and exit points, and apply a breadth-first search (BFS) algorithm to find the shortest path. In one example, it cleverly found the shortest path by going around the maze instead of through it. In another example, it successfully traced a path through a more complex maze.
• Ship Identification and Location: 03 identified the model and owner of a ship from a blurry photo and provided its last known location by searching the web for relevant information, including AIS data and news reports. It identified the yacht as "Yord," owned by Russian billionaire Mortis, and determined its last known location was off the coast of the Seychelles.
• Geo-Location Guessing: 03 accurately guessed the location of a landscape photo taken on a hike by analyzing the image and searching the web for similar photos and viewpoints. It identified the location as a lookout point north of Porto Cove Provincial Park, providing GPS coordinates. However, the video notes that other models like Google's Gemini 2.5 Pro and OpenAI's 01 are better at geo-guessing based on the Deep Guesser leaderboard.

Image Generation and Layered Designs

• Children's Storybook: 03 was prompted to create a children's storybook with five pages, each with short text and cute illustrations. It used GPT-4o's image generator to generate the images, maintaining a consistent style and character consistency across the pages.
• Layered Cyberpunk Cityscape: 03 was instructed to create a layered design of a futuristic cyberpunk city skyline at sunset, with separate layers for the background sky, distant cityscape, foreground buildings, and people walking in the foreground. It successfully generated the individual layers and provided them in a TIFF file, allowing for individual editing of each layer.

Coding Examples

• OpenSCAD 3D Model: 03 was asked to create an OpenSCAD 3D model of a house based on a rough sketch. However, the generated model was not accurate and did not resemble the sketch. Gemini 2.5 Pro performed better in this task.
• Interactive Night Sky Viewer: 03 failed to generate a working interactive night sky viewer with the top 20 constellations and labels. The generated code resulted in a blank page with errors. Gemini 2.5 Pro was able to generate this successfully.
• Bee Colony Simulation: 03 successfully generated a bee colony simulation with adjustable settings, stunning visuals, and interactive elements using p5.js. The simulation included features such as adjustable bee and flower counts, bee speed, pollen capacity, and flower regeneration. Both 03 and Gemini 2.5 Pro performed well in this task.

Performance and Benchmarks

• Self-Reported Benchmarks: OpenAI's self-reported benchmarks show that 03 and 04 Mini have significant improvements over previous models in areas such as competitive coding, visual reasoning, and instruction following.
• Independent Leaderboards: Independent leaderboards, such as the one by Artificial Analysis, rank 03 and 04 Mini at the top in terms of intelligence index, but only by a small margin over Gemini 2.5 Pro.
• Creative Writing: 03 excels in creative writing, ranking number one on the creative writing benchmark.
• Long-Context Recall: 03 demonstrates excellent ability to analyze and remember large amounts of information, scoring 100% accuracy on the finction live bench, which tests the model's ability to analyze stories over 120 words in length.
• Math Arena: 04 Mini High scores the highest on the Math Arena leaderboard, which ranks models based on their ability to do competitive math.
• Hallucination Rates: According to the veera benchmark, 03 has a relatively high hallucination rate of 6.8%, while 04 Mini has a hallucination rate of 4.6%. This is worse than Google's Gemini 2.0 o Flash (0.7%) and Gemini 2.5 Pro (1.1%).

Availability

• 03, 04 Mini, and 04 Mini High are available to Plus, Pro, and Team users in the model selector.
• Enterprise and education users will gain access in one week.
• Free users can try 04 Mini by selecting the "Reason" feature.
• 03 Pro with full tool support is expected to be released in a few weeks.
• Both 03 and 04 Mini models are available to developers via the API.

Uber Eats Promo Code Test

• 03 was prompted to find recent working promo codes for Uber Eats in Seattle, Washington.
• It successfully searched the web and provided a table of codes with discounts for new and existing users.
• However, none of the codes tested were valid, indicating potential inaccuracies in the information provided.

Conclusion

03 and 04 Mini are powerful AI models with impressive capabilities, including agentic tool use, image analysis, and coding. They can leverage various tools and agents to perform complex tasks, such as web searching, image generation, and code writing. However, they are not significantly better than Gemini 2.5 Pro in all areas, and they have relatively high hallucination rates. Users should be aware of these limitations and fact-check the information provided by these models.