Scrape Anything with Claude MCP's in 15 Min

Key Concepts

• Cloud MCPS (Managed Connector Platforms): Platforms that allow users to connect various tools and services to Claude (and other AI platforms) without coding.
• Scraping: Extracting data from websites and social media platforms.
• Apify: A marketplace of scrapers for various websites and social media platforms.
• Air Table: A database platform that can be integrated with Claude.
• Perplexity: An AI-powered search engine used for research and information retrieval.
• Claude Projects: A feature in Claude that allows users to create specific workflows with pre-defined instructions.
• No-Code Automation Platforms: Platforms like NAN (mentioned but not detailed) that allow users to automate tasks without writing code.
• Composio: A third-party provider that simplifies connecting MCP servers to popular software like Air Table.
• API (Application Programming Interface): A set of rules and specifications that software programs can follow to communicate with each other.
• JSON (JavaScript Object Notation): A lightweight data-interchange format that is easy for humans to read and write and easy for machines to parse and generate.

Competitor Analytics Use Case

• Objective: To analyze a competitor's social media activity and generate a slideshow report.
• Process:
  1. Claude is pre-instructed with a system prompt to perform competitor analysis and create a slideshow.
  2. The user provides the name of the company to investigate (e.g., Canva).
  3. Claude uses Perplexity to identify the company's social media platforms (TikTok, Instagram, X, etc.).
  4. Claude scrapes data from each platform using Appify scrapers.
  5. Claude scrapes the company's website and review platforms for sentiment analysis.
  6. Claude generates a slideshow report with data visualizations (bar charts, pie charts) and key insights.
• Example: The video demonstrates the process for Canva, generating a slideshow with sections on content strategy, top-performing content, and channel-specific analysis.
• Data Points: The Canva example includes pie charts showing the types of posts on LinkedIn (product updates, social media tips, company news, design inspiration).

YouTube Content Ideation Workflow

• Objective: To generate new YouTube content ideas by scraping comments from past videos and using Perplexity for trend research.
• Process:
  1. The user provides their YouTube channel name or URL.
  2. Claude uses a YouTube scraper to identify the user's past videos.
  3. Claude scrapes comments from each video.
  4. Claude uses Perplexity to research current trends.
  5. Claude analyzes the scraped data and generates video ideas, title variations, and identifies key themes and topics of interest from the comments.
  6. Claude updates an Air Table database with the new ideas.
• Example: The video demonstrates the process for the creator's channel, generating ideas such as "MCP: The Ultimate Cloud MCP Setup Guide for Beginners."
• Data Points: The analysis includes identifying the most engaged videos, key themes (cloud MCP integration, AI agent implementation), and common questions/requests from viewers.

Lead Generation and Personalized Outreach Workflow

• Objective: To generate leads, personalize outreach emails, and add them to a database.
• Process:
  1. The user provides criteria for lead generation (e.g., "three dental practice leads in Miami").
  2. Claude uses Google Maps scraper to find dental practices in Miami.
  3. Claude creates a spreadsheet with potential leads and email addresses.
  4. Claude generates an email outreach template.
  5. Claude uses web scraping to find more details about each lead from their website.
  6. Claude personalizes the email templates based on the scraped information.
  7. Claude adds the leads and personalized templates to Air Table.
• Example: The video demonstrates the process for dental practices in Miami, scraping data from Google Maps and company websites.
• Note: The video mentions that the initial email outreach template was not well-instructed and could be improved through iteration with Claude.

Connecting MCPs to Claude

• Three Main Methods:
  1. No-Code Automation Platforms (e.g., NAN): Mentioned as a previous method but not detailed in this video.
  2. Third-Party Providers (e.g., Composio): Simplifies the connection process for popular software like Air Table.
  3. Manual Setup: Involves using API keys and JSON configurations to connect MCPs directly to Claude.
• Apify Setup (Manual):
  1. Prerequisites: Mac or Windows, Cloud desktop app, Node.js, Appify API token.
  2. Install Node.js if not already installed.
  3. Obtain Appify API token from the Appify website (Settings > API Integrations).
  4. Copy the MCP server code from the Appify documentation.
  5. Add the Appify API token to the code.
  6. Add the desired scrapers (actors) to the code.
  7. Open the Cloud desktop app, go to Settings > Developer > Edit Config.
  8. Paste the modified JSON code into the configuration file.
  9. Save the configuration file and restart Claude.
• Air Table Setup (Composio):
  1. Go to the Composio website and select the Air Table MCP.
  2. Generate the command for Claude.
  3. Run the command in the terminal.
  4. Restart Claude.
  5. In Claude, use the Air Table MCP tools to check the connection.
  6. Claude will provide a URL to authenticate and connect your Air Table account.
  7. Grant access to a specific base or your entire workspace.
• Perplexity Setup (Manual):
  1. Obtain a Perplexity API key (requires a paid account).
  2. Use the npx command to set up the Perplexity MCP.
  3. Add the Perplexity API key to the configuration file.
  4. Add the Perplexity MCP to the existing MCP configuration in Claude.
  5. Save the configuration file and restart Claude.

Key Arguments and Perspectives

• Claude as a Superior Automation Tool: The presenter argues that for certain use cases, Claude can be a superior automation tool compared to traditional no-code platforms like NAN, especially when iteration and collaboration are important.
• Benefits of MCPs: MCPs provide instant access to hundreds of scrapers and tools, saving time and effort compared to setting up individual API calls.
• Accessibility of Automation: The presenter believes that Claude workflows are making automation more accessible to everyday users.

Notable Quotes

• "With these amounts of scrapers right here in Claude, the use cases are really endless."
• "I really do believe that for some of these use cases sometimes cloud can really be a superior setup."
• "Especially with these MCPs like the the Ampify one for example where we want to implement many API calls inside of one MCP or Air Table even where we have 13 different actions then it can be a lot faster using the manual setup."

Technical Terms and Concepts

• MCP (Managed Connector Platform): A platform that simplifies the process of connecting various tools and services to Claude.
• API (Application Programming Interface): A set of rules and specifications that software programs can follow to communicate with each other.
• JSON (JavaScript Object Notation): A lightweight data-interchange format that is easy for humans to read and write and easy for machines to parse and generate.
• Node.js: A JavaScript runtime environment that allows you to run JavaScript code on the server-side.
• Terminal: A command-line interface used to interact with the operating system.
• System Prompt: Instructions given to Claude to define the specific use case and desired outcome.
• Knowledge Base: A collection of information used to provide context and guidance to Claude.

Logical Connections

• The video begins by introducing the concept of using Claude with MCPs for web scraping and automation.
• It then presents three use cases (competitor analytics, content ideation, lead generation) to demonstrate the potential of this setup.
• The video transitions to a step-by-step guide on how to connect MCPs to Claude, covering Apify, Air Table, and Perplexity.
• The presenter compares the manual/third-party setup with using no-code automation platforms like NAN, highlighting the advantages of MCPs for certain scenarios.
• The video concludes by emphasizing the accessibility and potential of Claude workflows for automation.

Synthesis/Conclusion

The video provides a comprehensive guide on how to leverage Claude with MCPs to scrape data from the web and automate various tasks. It demonstrates the power of this setup through practical use cases and offers a detailed walkthrough of the connection process. The key takeaway is that Claude, combined with MCPs like Apify, Air Table, and Perplexity, can be a powerful tool for lead generation, marketing analytics, content ideation, and other applications, making automation more accessible to a wider audience. The presenter emphasizes the ease of use and the potential for iteration and collaboration within the Claude environment.