EASIEST Way to Scrape Any Website using DeepSeek, Gemini & Crawl4AI

Key Concepts

Web scraping, LLMs (Large Language Models), Crawl for AI, Light LLM Proxy, Deepseek, Gemini, Token cost, Markdown generation, Structured output, JSON schema, System prompt, Model configuration, API keys, Prompt engineering.

Web Scraping with LLMs: An Overview

The video discusses using LLMs for web scraping, highlighting the advantages of open-source tools like Crawl for AI, which simplifies the process compared to traditional methods like Beautiful Soup. However, it emphasizes the often-overlooked cost factor associated with using LLMs for web scraping at scale. The video then provides a step-by-step guide on setting up Crawl for AI and using LLMs like Deepseek and Gemini to extract information from web pages.

Setting Up Crawl for AI

1. Virtual Environment: Create a virtual environment using conda create -n <env_name> python=<version>.
2. Package Installation: Install necessary packages: crawlforai, openai, lightllm. Light LLM Proxy allows interaction with various LLMs using a unified OpenAI-compatible API.
3. Playwright Browser: Install the Playwright browser extension if needed, using the command playwright install.

Cost Considerations

The video emphasizes the significant cost associated with using LLMs for web scraping. The speaker shares an example of spending 8 cents for 25 requests using Deepseek, consuming 150,000 tokens. This cost can quickly escalate when processing complex websites or making millions of API calls. The video highlights that Crawl for AI also supports web scraping without LLMs, which can be a more cost-effective option.

Information Extraction with LLMs

1. LLM Strategy: The video uses an LLM strategy for information extraction, leveraging Crawl for AI's ability to directly scrape data and use an LLM to extract information on top of that.
2. URL Input: Provide the URL(s) to be scraped. Crawl for AI can handle single URLs or lists of URLs.
3. Structured Output (JSON Schema): Define a JSON schema to generate structured outputs. This allows for consistent data extraction and easy integration with databases. Example schema provided in the video includes fields like "rank," "model_name," "score," "confidence_interval," "words," "organization," and "license."
4. LLM Configuration: Configure the LLM by specifying the provider name (e.g., "deepseek"), API token (obtained from the LLM provider), and optionally the base URL (if using a custom endpoint via Light LLM Proxy).
5. Strategy Definition: Define the scraping strategy, including specifying the desired output schema, input format (markdown), and chunking options. Hyperparameters may need adjustment for optimal performance.
6. Crawl Execution: Pass the LLM instructions, configurations, and browser settings to Crawl for AI to scrape and extract data from the web page.

Code Example and Execution

The video provides a Python script example demonstrating how to use Crawl for AI to scrape a leaderboard page and extract specific information. The script includes:

• Importing necessary libraries.
• Setting up the LLM configuration (Deepseek V3 in the initial example).
• Defining the scraping strategy with the desired output schema and input format.
• Configuring browser settings.
• Executing the crawl using crawlforai.crawl().

The script is executed using python web_scraping.py.

Model Comparison: Deepseek vs. Gemini

The video compares the performance of Deepseek V3 and Gemini 2.5 Flash for web scraping.

• Deepseek V3: Initially used, but noted to be slower. Successfully extracted most of the required information, but initially failed to extract the full model names. This was corrected by updating the system prompt.
• Gemini 2.5 Flash: Used as a faster alternative. However, it exhibited unexpected behavior, reverting to extracting abbreviated model names despite explicit instructions in the system prompt. This highlights the importance of prompt engineering and model-specific adjustments.

Prompt Engineering

The video emphasizes the importance of prompt engineering when using LLMs for web scraping. The speaker demonstrates how modifying the system prompt can improve the accuracy of the extracted information. However, it also notes that prompts are not universally transferable between different models, even from the same provider.

Key Takeaways and Conclusion

• LLMs can significantly simplify web scraping, but cost is a major consideration.
• Crawl for AI is a powerful open-source tool for web scraping with LLMs.
• Structured output using JSON schema enables efficient data processing and integration.
• Prompt engineering is crucial for achieving accurate and desired results.
• Model selection and configuration impact performance and cost.
• Thorough validation of extracted data is essential.

The video concludes by encouraging viewers to subscribe for more technical content and expressing interest in creating follow-up videos on more advanced features of Crawl for AI.