O1 Model First Impressions: Faster, Smarter, is it? And is unlimited access worth 200$? Sometimes?

Key Concepts

• O1 Model: OpenAI's new, supposedly smartest and fastest model with visual capabilities.
• Visual Capabilities: The ability of a model to accept and work with images.
• Prompt Mode: A $200/month subscription for unlimited access to OpenAI models, including advanced voice.
• 3JS: A JavaScript library for 3D rendering.
• WebSim: An environment for creating web-based simulations.
• Tools: External functionalities like internet access, code interpreter, and API usage.

O1 Model: First Impressions and Tests

Introduction of O1 Model

OpenAI has released the O1 model as part of their "12 days of shipping stuff" advent calendar. The O1 model is advertised as their smartest and fastest model to date, with the key addition of visual capabilities, which were absent in previous models like O1 Preview and O1 Mini. This means it should be able to analyze and generate images, unlike its predecessors which were limited to text-based interactions.

Initial Skepticism and YouTuber Confirmation

The speaker expresses initial skepticism about the "smartest" claim, referencing tests performed by the YouTuber "AI Explained," who is even more skeptical. While the O1 model is acknowledged as an improvement, its practical utility is questioned.

Geometry Problem Test

The speaker presents a geometry problem involving shadows and bar heights, previously used to test O1 Mini and Claude. The problem requires understanding spatial relationships and shadows to determine the height of a red bar. GPT-4, O1, and other models failed to correctly interpret the problem, often misinterpreting shadows as separate objects.

• Problem Description: A 2m green bar casts a 3m shadow. A red bar casts a 6m shadow plus an additional 4m shadow. Find the height of the red bar. The correct answer is 8m.
• Model Performance: O1, like GPT-4, incorrectly identifies the red bar's height as 4m, failing to understand the concept of shadows.
• Previous Success: The speaker notes that O1 Preview could solve the problem if the problem was described correctly, highlighting the bottleneck in visual understanding.

Image-to-3D Conversion Test

The speaker tested O1's ability to generate a 3D scene from an image using HTML, CSS, and JavaScript.

• Prompt: "Can you please generate a 3D scene out of this picture in HTML CSS and JS JavaScript".
• O1's Description: The model inaccurately describes the scene's elements and their spatial relationships.
• Code Generation Issues: The generated code initially failed due to issues with importing the 3JS library.
• Manual Fix and Result: After manually fixing the code, the resulting 3D scene poorly represents the original image.
• Comparison with WebSim (Sonnet 35): A similar test using WebSim with Sonnet 35 also produced an inaccurate 3D representation, but the code worked without manual fixes.

Game Recreation Test

The speaker attempted to have O1 recreate a 3D parking game from an image.

• Prompt: Recreate this game as just HTML.
• Initial Result: O1 produced a functional but basic game with a car leaving a trail.
• Comparison with WebSim (Sonnet 35): WebSim produced a worse result.
• 3JS Issues: The speaker notes recurring issues with 3JS imports across different models.

Overall Assessment of O1

The speaker concludes that O1 is marginally better than previous models and Sonnet 35, and it is faster. However, it doesn't consistently demonstrate superior intelligence, especially in tasks requiring visual understanding.

Prompt Mode: Is It Worth $200/Month?

Limited Utility Without Tools

The speaker argues that Prompt Mode's $200/month price is not justified for most users due to the lack of tool integration. While O1 can now accept images, its limitations in understanding them hinder its usefulness.

Comparison with GPT-4

GPT-4 is favored due to its ability to use tools like internet search, code interpreter, and custom APIs. The speaker finds that the inability of O1 and advanced voice to use tools makes them less efficient for complex tasks requiring external data or functionalities.

Potential Value with Tool Integration

The speaker suggests that Prompt Mode could become worthwhile if OpenAI integrates tools, potentially boosting productivity enough to justify the cost.

AI Explained's Perspective on Pro Mode

The speaker references AI Explained's video, which indicates that Pro Mode's performance is nuanced, sometimes performing worse than other models.

Conclusion on Prompt Mode

Prompt Mode is deemed a niche tool that may benefit specific users, but it is not recommended for most people unless they are already hitting limits on the models in plus.

Conclusion

The O1 model represents a step forward with its visual capabilities, but its limitations in understanding images and the lack of tool integration make it difficult to recommend Prompt Mode for most users. The speaker emphasizes the importance of tools for practical AI applications and suggests waiting for tool integration before considering the $200/month subscription.