Scrape Any Website for FREE Using DeepSeek & Crawl4AI

Key Concepts

• Web Scraping: Extracting data from websites.
• AI Web Scraper: A web scraper that uses AI, specifically LLMs, to process and extract data.
• Crawl 4 AI: An open-source library for web scraping, designed to integrate with LLMs.
• DeepSeek: An AI model, specifically DeepSeek R1, known for its reasoning capabilities, speed, and cost-effectiveness.
• Grok: A platform providing AI chips and infrastructure for running AI models, including a free tier.
• LLM Extraction Strategy: Configuring an LLM to extract specific data points from scraped content based on a defined schema.
• CSS Selectors: Patterns used to select specific HTML elements on a webpage for scraping.
• Headless Browser: A browser that runs in the background without a graphical user interface.
• Tokens: Units of text used by LLMs to process and generate language.

AI Web Scraper Construction: A Step-by-Step Guide

1. Introduction

The video demonstrates how to build a free AI web scraper using Crawl 4 AI, DeepSeek, and Grok. Web scraping is highlighted as a highly sought-after skill in the AI development landscape. The tutorial focuses on building a scraper to extract leads from wedding venue websites for a photographer seeking new clients. The complete source code is provided for free.

2. Core Tools Overview

• Crawl 4 AI: This open-source library facilitates web scraping and tagging content for LLM processing. It allows for extracting specific data and passing it to an LLM for further analysis or transformation.
• DeepSeek: The DeepSeek R1 reasoning model is presented as a fast and cost-effective alternative to OpenAI's models. It's known for its human-like thought process in generating responses. It is claimed to be 20 times cheaper to run than other models.
• Grok: Grok provides AI chips optimized for running AI models like DeepSeek. It offers a free tier, enabling users to run DeepSeek without cost. Grok is highlighted for its speed, processing at 275 tokens per second in an example.

3. Use Case Scenario: Wedding Venue Lead Generation

A wedding photographer needs to gather information on wedding venues in a specific area to contact them for potential business opportunities. The goal is to build a scraper that extracts venue names, locations, prices, and generates a one-sentence description for each venue. The scraped data will be saved to a CSV file and then uploaded to a Google Sheet for easy access and sharing.

4. Code Setup and Installation

1. Environment Creation: Use Conda to create a new environment with the necessary dependencies.
2. Dependency Installation: Install Crawl 4 AI using conda install -c conda-forge crawl4ai.
3. API Key Configuration: Obtain an API key from Grok and add it to the environment file.
4. Running the Scraper: Execute the scraper using python main.py.

5. Code Walkthrough: Core Components

• main.py: The main script orchestrating the web scraping process.
• crawl_all_venues() Function: The primary function responsible for scraping wedding venue data.
  • Browser Configuration (browser_config): Sets up the browser (e.g., Chrome) with specified window size and headless mode (whether to display the browser window or run it in the background).
  • LLM Strategy (llm_extraction_strategy): Defines how the LLM will extract information from the scraped content.
    • Venue Model (models/venue.py): Defines the data structure (schema) for a wedding venue, including fields like name, location, and price.
    • LLM Instructions: Provides instructions to the LLM on how to extract the required information from the raw scraped data and generate a one-sentence description.
    • Model Selection: Specifies the LLM to use (DeepSeek via Grok in this case). Alternatives like Olama (for local Llama 3) or GPT-4 are mentioned.
  • Crawler Creation: Creates a crawler instance using the configured browser settings.
  • fetch_and_process_page() Function: Scrapes a single page and extracts venue data.
    • URL Construction: Constructs the URL for the page to be scraped, including the base URL and page number.
    • No Results Check: Scrapes the page to check for a "no results" message. If found, it indicates the end of the listing, and the function returns True.
    • Detailed Scraping: If results are found, the page is scraped again, this time using the LLM extraction strategy and CSS selectors.
      • CSS Selectors: Specifies the HTML elements to be scraped (e.g., elements with the class "info container").
      • LLM-Powered Extraction: The LLM extracts data from the selected elements based on the defined schema and instructions.
    • Data Extraction: The extracted data is loaded from a JSON string into a list of venue objects.

6. Code Execution and Demonstration

The video demonstrates running the main.py script. The scraper opens a Chrome browser (if headless=False) and starts scraping the wedding venue website page by page. The terminal displays logs showing the scraping progress, including the number of leads extracted per page and the LLM calls to Grok/DeepSeek. The scraper continues until it reaches a page with no results.

7. Results and Output

Once the scraping is complete, the script saves the extracted data to a CSV file. The video then shows how to import this CSV file into Google Sheets to create a formatted table. The table includes the scraped venue information (name, location, price) and the AI-generated one-sentence descriptions.

8. Google Sheets Integration

The video demonstrates importing the generated CSV file into Google Sheets. This allows for easy viewing, filtering, and sharing of the scraped data with the client. The data is presented in a table format, enabling the photographer to easily analyze and contact potential wedding venues.

9. Conclusion

The video provides a comprehensive guide to building a free AI web scraper using Crawl 4 AI, DeepSeek, and Grok. It demonstrates how to set up the environment, configure the scraper, define the data extraction strategy, and run the scraper to extract valuable leads from a website. The use case of wedding venue lead generation highlights the practical applications of this technology. The availability of the complete source code makes it easy for viewers to replicate and adapt the scraper for their own needs.