Scrape ANY Website Automatically with n8n + Apify (Full System Breakdown)

Key Concepts

Web scraping, automation, Appify, NADN, Google Maps scraping, email extraction, cold email outreach, HTTP requests, JSON data, API keys, data filtering, looping, array aggregation, Google Sheets integration.

Web Scraping and Automation Tutorial: Appify and NADN

Introduction

The tutorial focuses on web scraping and automation using Appify and NADN for non-coders, aiming for a quick and cost-effective setup. Appify handles web scraping from various platforms like Instagram, Facebook, TikTok, Twitter, YouTube, Apollo, and Google Maps. The example focuses on scraping Google Maps listings and extracting email addresses for cold email campaigns.

Appify Overview

Appify allows scraping data from numerous websites across categories like real estate, news (Reddit, Axe, Google News, Google Trends), and e-commerce (Amazon, Facebook ads). The tutorial emphasizes that learning to scrape one platform (e.g., Google Maps) provides a foundation for scraping others. Free blueprints are available for download.

NADN and Automation Setup

NADN is used for automation. The tutorial guides users through creating a scenario to automate Google Maps scraping.

Step-by-step process:

1. Trigger: A scheduler is set up to trigger the web scraping process weekly (e.g., every Monday).
2. HTTP Request: Since NADN lacks a direct Appify module, an HTTP request module is used to connect to Appify.
3. API Endpoint: The Appify API is accessed to run an actor (web scraper) and retrieve data. The "Run an actor and get data set items" action is selected.
4. Importing Curl Command: The curl command from Appify's API reference is imported into the NADN HTTP request module.
5. Configuration:
   • Method: Set to "POST" for creating a web scrape.
   • URL: The actor ID from the Appify URL is extracted and placed in the URL of the HTTP request.
   • Headers: Content-Type is set to "application/json," and authorization is handled using an API key.
   • JSON Data (Body): The search term (e.g., "landscapers") and location (e.g., "Toronto, Canada") are specified in JSON format. The maxResults parameter is initially set to 5 to conserve free credits.

Obtaining the API Key

The API key is found in Appify's settings under the "API and Integrations" tab. This key is essential for authenticating requests from NADN to Appify.

Initial Testing and Troubleshooting

The initial setup is tested. An error occurs due to an incorrect authorization code (API key). After correcting the API key, another error arises due to scraping too many results (548 instead of the intended 5). This highlights the importance of setting the maxCrawlPlacesPerSearch parameter correctly in the JSON data.

Email Extraction

Google Maps listings typically lack email addresses. To address this, the tutorial demonstrates how to scrape the websites associated with the listings to find email addresses.

Step-by-step process:

1. Filtering: A filter is added to exclude listings without websites to prevent errors.
2. Second Appify Actor: A second Appify actor (web scraper) is used to extract contact details, including email addresses, from the websites. The "Google Maps Email Extraction" actor is selected.
3. Configuration:
   • The actor ID in the HTTP request URL is updated to the ID of the contact detail scraper.
   • JSON data is configured with parameters like maximumCostPerRun (set to 1), maximumPagesPerURL (set to 20), and maximumLinkDepth (set to 1).
   • The URL to scrape is dynamically set to the website URL obtained from the first web scraper using data mapping.
4. Looping: A loop is implemented to process each website URL individually to avoid exceeding memory limits.

Google Sheets Integration

The extracted data (business name, city, niche, email, phone number, etc.) is automatically added to a Google Sheet.

Step-by-step process:

1. Google Sheets Module: A Google Sheets module is added to the NADN scenario.
2. Append a Row: The "Append a Row" action is selected to add new rows to the Google Sheet.
3. Mapping Data: Data from the web scrapers (business name, city, niche, email, phone number, etc.) is mapped to the corresponding columns in the Google Sheet.
4. Array Aggregator: An array aggregator is used to merge all the results into one array, ensuring only one result per website is added to the Google Sheet.
5. Update vs. Append: The tutorial explains the difference between updating and appending rows. For updating, a unique column (e.g., business URL) must be specified for matching.

Final Steps and Cold Email Outreach

After testing and verifying the workflow, the limits on the number of scraped results are adjusted. The tutorial suggests integrating the scraped data with cold email platforms like Instantly.ai or Lemlist for automated outreach.

Conclusion

The tutorial provides a comprehensive guide to web scraping and automation using Appify and NADN, enabling users to extract data from Google Maps and other platforms, find email addresses, and integrate the data with Google Sheets and cold email outreach tools. The importance of careful configuration, error handling, and resource management is emphasized. The speaker promotes his school community for further assistance and resources.

Notable Quotes:

• "Once you learn how to do it with Google Maps, you're going to know how to do it with every single one of these."
• "JSON data is very finicky. So even if you were to like delete that one comma, it's just not going to work."

Technical Terms and Concepts:

• Web Scraping: Extracting data from websites.
• Automation: Automating repetitive tasks using software.
• Appify: A platform for web scraping.
• NADN: An automation platform.
• Actor: A web scraper in Appify.
• HTTP Request: A method for sending data over the internet.
• JSON Data: A data format used for transmitting data between a server and a web application.
• API Key: A code used to authenticate requests to an API.
• Data Mapping: Connecting data from one source to another.
• Array Aggregator: A tool for merging multiple arrays into one.

Synthesis/Conclusion:

The tutorial successfully demonstrates how to combine Appify and NADN to automate the process of scraping Google Maps listings, extracting email addresses, and integrating the data into a Google Sheet. By using HTTP requests, JSON data, and API keys, the tutorial provides a practical solution for non-coders to leverage web scraping and automation for lead generation and cold email outreach. The tutorial also highlights the importance of careful configuration and error handling to ensure efficient and accurate data extraction.