Gemini’s Native Web Scraper: 100% "Free" & Multimodal

Gemini API Context URL: A Deep Dive

Key Concepts:

• Context URL: A Gemini API tool enabling grounding of responses in content from web URLs (text, PDF, images).
• Gemini API: Google’s API for accessing its Gemini large language models.
• Grounding: The process of anchoring LLM responses in external data sources for accuracy and relevance.
• Token Limit: The maximum amount of text (input + output) a model can process in a single request.
• Rate Limit: The maximum number of requests that can be made within a specific timeframe.
• SDK vs. REST API: Software Development Kit (SDK) provides pre-built functions, while REST API offers more direct control but requires more coding.
• Tool Configuration: The method of specifying which tools (like Google Search or Context URL) the Gemini model should utilize.

1. Introduction & Overview

The video focuses on the Context URL feature within the Gemini API, highlighting it as a powerful yet underutilized tool for grounding LLM responses. Unlike dedicated web scraping tools like Reader API or Crawl for AI, the Context URL offers a faster and more reliable approach to providing Gemini with external information. The feature has evolved from initially supporting only text to now including image and, crucially, PDF parsing capabilities. The speaker emphasizes the ability to visually parse PDFs, rather than simply converting them to markdown, as a significant advantage.

2. How it Works: The Two-Step Process

The Context URL operates on a two-step process:

• Cache Check: Gemini first checks if the provided URL is already indexed in Google’s search cache.
• Live Fetching: If the URL isn’t cached, Gemini fetches the data live from the web.

This approach balances speed (using cached data when available) with access to fresh information (through live fetching). The speaker notes that the cost is effectively the token usage associated with processing the scraped content, eliminating the need for separate scraping service fees.

3. API Implementation & Model Support

The Context URL can be enabled in AI Studio via a toggle. However, the video primarily focuses on API usage. The speaker recommends using the REST API over SDKs for greater control and reduced maintenance due to fewer dependency updates. The following models support this feature (specific models were not explicitly named beyond "Gemini 3 flash preview" used in the demo).

4. Limitations & Constraints

Several limitations are outlined:

• No Function Calling: The Context URL is not compatible with traditional function calling mechanisms. It functions as a standalone tool.
• Rate Limit: A rate limit of 20 URLs per request is in place.
• Data Limit: Each URL can contain up to 34 megabytes of data.
• Public Accessibility: Only publicly accessible URLs can be used.

5. Demonstration & Examples (Notebook Walkthrough)

The video demonstrates the Context URL using a Python notebook. The core concept is configuring the Gemini API with the desired tools.

• Grounding with Google Search: The example shows how providing the Google Search tool allows the model to generate inputs for the search query, retrieving up-to-date information. Without the tool, the model relies on its training data, which may be outdated.
• Grounding with URL Context (HTML): The speaker provides a URL to a Google DeepMind Gemini models page. Gemini successfully extracts information from expandable elements on the page, presenting it in a table format, including key differences between model families. The example highlights the model’s ability to access and process dynamically loaded content.
• Grounding with URL Context (PDF): A PDF file URL is provided, and Gemini accurately answers questions based on the PDF’s content.
• Grounding with URL Context (Image): An image URL is used, and Gemini attempts to identify and name the numbered parts of a musical instrument in French. The speaker requests feedback on the accuracy of the French translations.
• Combining Tools: The final example demonstrates combining the Google Search and Context URL tools, allowing the model to leverage both sources of information.

6. Technical Details & Best Practices

• Tool Configuration Format: The API requires a specific format for configuring tools, listing them within the configuration and tools sections.
• Formatted Output: The Context URL doesn’t simply scrape raw HTML; it provides well-formatted output like Markdown or direct PDF scraping.
• Caching Behavior: Gemini prioritizes using cached versions of URLs when available, improving speed.

7. Sponsor Segment: Outskll

The video includes a sponsored segment for Outskll, an AI-focused educational platform. Outskll is hosting a 2-day Generative AI Engineering Mastermind workshop, covering topics like building AI agents and agentic workflows. Early access sign-ups receive free bonuses, including an AI engineering prompt bible and access to GitHub repositories.

8. Notable Quotes

• “It helps Gemini ground its answers in text, PDF and image files that are available on the internet.” – Describes the core function of the Context URL.
• “It’s actually looking at every page visually which I have found to be extremely useful for my own applications.” – Highlights the advantage of PDF parsing.
• “You don't really need dedicated scraping services anymore.” – Emphasizes the cost-effectiveness of the Context URL.
• “If you are building an application, I highly recommend to use the rest API instead of any of the SDKs because that will reduce the complexity that you are dealing with.” – Provides a practical development tip.

9. Data & Statistics

• Rate Limit: 20 URLs per request.
• Data Limit per URL: 34 megabytes.

10. Conclusion & Takeaways

The Context URL feature in the Gemini API is a valuable tool for grounding LLM responses in external data. It offers a fast, reliable, and cost-effective alternative to dedicated web scraping services, particularly with its support for PDF and image parsing. The speaker encourages viewers to explore this feature for their own applications, emphasizing its flexibility and ease of use. The recommendation to use the REST API over SDKs for long-term maintainability is a key actionable insight. The ability to combine the Context URL with other tools like Google Search further enhances its utility.