Gemini 2.5 Computer Use: BEATS Claude SONNET 4.5 & OpenAI!

Key Concepts

• Gemini 2.5 computer use model: An advanced AI model designed for browser automation and interaction.
• Browser Automation: The process of controlling a web browser programmatically to perform tasks.
• API (Application Programming Interface): A set of rules and protocols for building and interacting with software applications.
• Playwright: A Node.js library to automate Chromium, Firefox, and WebKit with a single API.
• Chromium: The open-source base for Google Chrome and other web browsers.
• Latency: The delay before a transfer of data begins following an instruction for its transfer.
• Accuracy: The degree to which the result of a measurement, calculation, or specification conforms to the correct value or a standard.
• Prompt Engineering: The art of crafting effective input instructions (prompts) for AI models to achieve desired outputs.
• Web Voyager: A competing AI agent model for web automation.
• OpenAI computer use agent model: A competing AI agent model for web automation developed by OpenAI.
• Claude: Another family of large language models, with Claude OneT 4.5 mentioned as a competitor.

---

Introduction to Gemini 2.5 Computer Use for Browser Automation

The video introduces Google's Gemini 2.5 computer use model, highlighting its enhanced capability to control web browsers effectively. This AI model can automate repetitive tasks such as form filling and internet research by interacting with web elements. The speaker demonstrates an automated task where Gemini automatically moves labels to appropriate columns on a webpage, emphasizing that this functionality is executed via an API at the backend, allowing integration with various AI applications.

How Gemini 2.5 Computer Use Operates

The operational framework of Gemini 2.5 computer use involves a cyclical process:

1. Task Provision: A user provides a specific task to the model.
2. Model Processing: The task is sent to the Gemini 2.5 model.
3. Response Generation: The model processes the task and generates a response.
4. Environment Interaction: The response is sent to the computer environment or client application.
5. Action Execution: The received action (e.g., clicking a button, typing text) is executed within the browser.
6. State Capture: The new environment state, similar to a screenshot, is captured.
7. Iteration: This captured state is sent back to the first step, and the process repeats until the task is completed.

Gemini 2.5 possesses native capabilities to fill out forms, manipulate interactive elements like drop-downs and filters, and operate behind login screens, making it highly versatile for complex web interactions.

Performance Benchmarks and Competitive Analysis

Gemini 2.5 computer use demonstrates strong performance compared to other leading models. On official leaderboards, Gemini 2.5 scores 69%, surpassing OpenAI's computer use agent model, which scores 61%. While other models like Web Voyager (88%) and OpenAI (overall 87%) show higher general scores, Gemini 2.5 distinguishes itself with lower latency and higher accuracy when compared to other models, making it particularly efficient for real-world browser automation tasks. The model is currently available in preview via its API.

API Documentation and Setup Process

Interacting with Gemini 2.5 via its API involves four distinct steps:

1. Send Request: Submit a request to the model.
2. Receive Response: Get the response back from the model.
3. Execute Action: Perform the action specified in the model's response.
4. Capture State: Record the new state of the environment.

To set up and run the demonstrations, the following basic requirements and steps are necessary:

• Packages: Install Google Genai and Playwright.
• Browser Engine: Install Chromium using Playwright.
• Installation Commands:
  • pip install Google-Genai playwright
  • playwright install chromium
• API Key Configuration: Generate a Gemini API key from aistudio.google.com and set it as an environment variable: export GEMINI_API_KEY=<your_key>.
• Execution: Run the Python script (e.g., python app.py).

Practical Demonstrations

The video showcases two practical demonstrations of Gemini 2.5's capabilities:

1. Extracting Pet Details from a URL

• Task: Given a specific URL, extract all details for any pet with California residency and output the information in a clear, readable format.
• Process: The speaker provides a Python script (app.py) containing the prompt and execution logic. Upon running the script, Gemini 2.5 automatically opens a web browser, navigates to the specified URL, scrolls through the document, identifies the relevant information (pet details with California residency), and prints it to the terminal. The terminal logs show the steps taken, such as "open web browser," "scroll through the document," and "final turn got the details and printing it out."

2. Dragging Sticky Notes to Relevant Columns

• Task: Navigate to the sticky-note-jam website, which features three columns (promotion, setup, volunteers). The AI is instructed to drag various sticky notes to their appropriate columns, ensuring correct placement, and then output the initial and final states.
• Process: A separate Python script (app_sticky_notes.py) is executed. The AI opens the browser, displays the initial arrangement of sticky notes, and then proceeds to drag and drop each note into its correct column. The speaker observes that the AI moves items approximately every 5 seconds, noting that this speed is "much more faster" than other computer use models previously tested. The terminal confirms "drag and drop execution is happening," and the final state shows all sticky notes correctly categorized.

Conclusion

The speaker expresses strong approval for the Gemini 2.5 computer use model, particularly commending its significantly faster performance compared to other computer use models. The model's ability to automate complex browser interactions with high accuracy and lower latency positions it as a promising tool for various automation needs. The video concludes by encouraging viewers to explore and compare Gemini 2.5 with other AI automation solutions, such as OpenAI's computer use model.