I built an automated newsletter with n8n (free template)

Automated Newsletter Workflow Summary

Key Concepts: Automated newsletter creation, web scraping, N8N, Substack, job board monetization, HTML parsing, API integration, AI hallucination, Docker customization.

Workflow Overview

The video details a workflow for automating newsletter creation by scraping content from existing newsletters, specifically job postings. The creator uses N8N, an open-source workflow automation tool, to achieve this. The process involves fetching emails, parsing HTML, storing data in Notion, manually selecting content, converting it to HTML, and pasting it into Substack.

Step-by-Step Process

1. Manual Trigger: The N8N workflow is initiated manually.
2. Fetch Emails: N8N retrieves the latest emails from a specified sender in Gmail. This is achieved using N8N's Gmail integration.
3. Parse HTML: A custom code node in N8N parses the HTML content of the newsletter emails using the cheerio library.
4. Transform to JSON: The parsed HTML is transformed into JSON format for easier manipulation.
5. Merge JSON Objects: JSON objects from multiple newsletters are merged into a single array.
6. Save to Notion: The merged data is saved to a Notion database, with each job posting represented as a separate entry.
7. Manual Selection: The user manually selects the desired job postings from the Notion database. This step is not automated due to the need for human judgment.
8. Convert to HTML: A custom Python script converts the selected job postings into an HTML list. This script is executed locally because N8N, running on a server, cannot directly access the user's clipboard.
9. Paste into Substack: The generated HTML is pasted into a Substack newsletter draft. Due to the lack of a Substack API, this step is manual. The creator duplicates the latest newsletter and replaces the old jobs with the new ones.

Technical Details

• N8N: An open-source workflow automation tool used to orchestrate the entire process.
• Cheerio: A library used for parsing HTML. It's similar to jQuery for the server-side.
• Node Fetch: A library used for making HTTP requests.
• Custom Code Nodes: N8N nodes that allow users to execute custom JavaScript code.
• Notion: A workspace application used to store and manage the scraped job postings.
• Substack: A platform for publishing newsletters.
• Docker: Used to customize the N8N environment to include necessary libraries.
• API: Application Programming Interface. Substack's lack of a public API necessitates a manual step in the workflow.
• RSS Feed: A data format (XML) used by some job boards to syndicate content. The creator initially attempted to use RSS feeds with ChatGPT but encountered issues.
• XML: Extensible Markup Language, a structured data format.

Scraping Implementation

The scraping process is implemented within a custom code node in N8N. The code uses cheerio to parse the HTML and extract relevant information.

• Targeting Specific Sections: The code targets specific sections of the newsletter, such as "Featured Jobs," to locate the job postings.
• Extracting Data: The code extracts data such as the job title, company name, and link from each job posting.
• Handling Substack Link Redirection: Substack modifies links to track clicks. The code uses node-fetch with the redirect: 'manual' option to follow the redirects and obtain the original destination URL.

Docker Customization

To use cheerio and node-fetch within N8N, the N8N Docker image needs to be customized.

1. Create a Custom Dockerfile: The Dockerfile inherits from the official N8N image.
2. Install Missing Libraries: The Dockerfile installs cheerio using npm install cheerio.
3. Enable External Libraries: The NODE_FUNCTION_ALLOW_EXTERNAL environment variable is set to cheerio,node-fetch to allow the use of these libraries in code nodes.
4. Build and Run the Image: The custom Docker image is built and run with the specified environment variable.

AI Experimentation and Limitations

The creator initially attempted to use ChatGPT to scrape job postings but encountered issues with AI hallucination, where ChatGPT generated fake or non-existent job postings. This led to the decision to use a deterministic workflow with N8N instead.

Monetization Strategies

The creator outlines three potential monetization strategies for the automated newsletter workflow:

1. Lead Generation: Use the newsletter to generate leads for a separate product or service. This is the creator's primary plan, with the newsletter serving as a precursor to a full-fledged job board website.
2. Ad Spots: Sell ad spots directly within the newsletter. This could be a way to test demand before investing in a full job board.
3. Workflow as a Service: Sell the automated newsletter workflow as a template to other users.

Notable Quotes

• "So, this is the newsletter that I want to automate."
• "So, I needed to find a trick. So, the trick I found was just to copy into HTML and I was able to after quickly paste this into Substack."
• "AI wasn't the right tool for this job because this is a process that has to be deterministic. We don't want any creativity here."
• "Chad GPT can do a good job, but you have to provide some guidance."

Conclusion

The video provides a detailed walkthrough of an automated newsletter workflow built with N8N. It highlights the importance of deterministic processes for data scraping and demonstrates how to overcome limitations in API availability through creative solutions. The creator also shares insights into potential monetization strategies for the workflow. The key takeaway is that N8N, combined with custom code and Docker customization, can be a powerful tool for automating content curation and newsletter creation, even in the absence of complete API support.