How I Built a 7,000-Listing Directory Website in 50 Minutes (Using Only AI Tools)

Key Concepts

• Directory website creation
• SEO optimization
• Data scraping (Google Maps)
• AI coding assistance (Cursor AI)
• Website building (Bolt. new)
• LLM (Large Language Model) for content generation
• API integration (Google Maps API, Venice AI)
• Data cleaning and organization
• JSON conversion
• GitHub integration

Website Overview (monorfind.com Example)

• Functionality: Directory website listing businesses (initially monor schools, then shoe repair shops).
• Features:
  • Listings by state and city.
  • Business name, address, reviews (scraped from Google Maps).
  • Map embed with directions.
  • Link to Google Maps.
  • Phone number.
  • AI-generated descriptions.
  • Blog and FAQ sections.

1. Finding a Lucrative Niche

• The video doesn't cover niche selection but references Frey's video for guidance.
• Focus on search terms with low competition.

2. Scraping Listings from Google Maps

• Outscar: Recommended by Frey, allows scraping Google Maps data (paid service, accepts Bitcoin).
  • User defines search term (e.g., "shoe repair shop").
  • Specifies location (e.g., "Florida").
  • Outscar scrapes data and provides it for a fee.
• Google Maps API (Free Option):
  • Requires a Google Cloud account (offers $300 free credit at the time of recording).
  • Uses a Python script to interact with the Google Maps API.

2.1. Using Cursor AI for Script Generation

• Cursor AI: An AI-powered coding app similar to Visual Studio Code.
  • Features an AI agent that assists with code generation and problem-solving.
  • Can generate code based on natural language instructions.
• Example Prompt: "Write a script that uses Google Maps API to scrape Google Maps of all listings in the State of Florida for the term shoe repair shop. Export the results as a CSV file which includes one positive rating and one negative rating."
• Cursor AI's Workflow:
  1. Generates Python code based on the prompt.
  2. Creates a requirements.txt file for dependencies.
  3. Provides a README.md file with instructions.
• Vetting the Script:
  • Important to understand the generated code to avoid malicious code.
  • Use Cursor AI's "Ask" feature to explain specific code snippets.
• Setting up API Key:
  1. Create a .env file to store the Google Maps API key.
  2. Obtain the API key from the Google Cloud Console (APIs & Services -> Credentials).
• Installing Dependencies:
  • Use pip install -r requirements.txt in the terminal to install required Python packages.
  • Troubleshooting: If errors occur, copy the error message and use Cursor AI or Google to find solutions.
• Running the Script:
  • Execute the Python script using python script_name.py.
  • The script scrapes data and exports it to a CSV file.

2.2. Enhancing the Script

• Iteratively improve the script by adding more features:
  • Adding columns for website and description.
  • Separating full address into street address, city, state, and ZIP code columns.
• Example Prompt: "Add a column for website and description into the scrape shoe repair script."
• Importance of Iteration: Start with a basic script and gradually add features to avoid wasting API credits.

3. Organizing and Cleaning the Data

• Purpose: Remove invalid or incomplete listings from the CSV file.
• Script: Create a Python script to:
  • Remove listings with no address or state.
  • Add "no website" to the website column if no website is found.
• Example Prompt: "Write a script that removes listings from our CSV file that has no address or no state. If there is no website, add 'no website' to the website column."
• Command Line Arguments: Modify the script to accept the CSV file name as a command-line argument for reusability.
• Success Message: Add a message to the script that displays the number of rows updated or deleted.
• Manual Verification: Check listings with "no URL" to verify their validity.

4. Generating Descriptions with an LLM (Crawl for AI)

• Goal: Scrape website content and generate descriptions for each listing.
• Crawl for AI: An open-source web crawler and scraper designed for use with LLMs.
  • Uses LLM extraction method to structure unstructured data.
• Steps:
  1. Connect to the LLM AI (Venice AI).
  2. Scrape the website for 2-3 paragraphs of relevant information.
  3. Generate a description of the website.
  4. Output the description to the description column of the CSV file.
  5. Repeat the process for all listings.
  6. Save the updated CSV file.

4.1. Connecting to Venice AI

• Venice AI: An LLM AI service.
• API Documentation: Consult Venice AI's documentation for API specifications (Swagger/YAML).
• Swagger Definition: Copy the Swagger definition into a YAML file in the project.
• Python Script: Use Cursor AI to generate a Python script that connects to the Venice AI API.
• Testing the Connection: Add a command-line flag to test the API connection.

4.2. Scraping Website Content with Crawl for AI

• Installation: Install Crawl for AI using pip install crawlforai.
• Modules:
  • async_web_crawler: Basic web crawling functionality.
  • crawler_result: Handles the results of the crawling process.
  • llm_config: Configuration for LLM integration.
  • markdown_generation: Cleans up and formats the scraped content.
  • pruning_content_filter: Filters out irrelevant content.
• Example Prompt: "Next, we want to scrape the website URL from the listing Row in data updated. CSV and we're just going to say find two to three paragraphs of relevant information to summarize the listing we can even say summarize the shoe repair store format it to markdown let's just say prune filter it and then format it to markdown."
• Directory per Listing: Create a unique directory for each listing to store the raw data and markdown file.

4.3. Generating Descriptions

• Prompt Engineering: Refine the prompt used to generate descriptions to improve their quality.
  • Example: "You are a helpful assistant that creates concise informative descriptions of shoe repair businesses. Focus on key services, specialties, and unique features. Do not output anything else with the description. Thank you. Do not make anything up. If there's not enough data, say 'not enough data'."
• Iterative Refinement: Continuously adjust the prompt and script based on the generated descriptions.

4.4. Saving the Updated CSV File

• Real-time Saving: Modify the script to save the updated CSV file after processing each listing, rather than waiting until the end.

5. Splitting the Address

• Goal: Separate the street address, city, ZIP code, state, and country into individual columns.
• Script: Create a Python script to split the address column into new columns.
• Example Prompt: "Write a script as split address that separates the street address from the city Z code State and Country into new columns."

6. Converting to JSON

• Purpose: Convert the CSV data to JSON format for use in populating the directory website.
• Script: Create a Python script to convert the CSV file to a JSON file.
• Example Prompt: "Write a script that converts data updated addresses that's our latest one to Json for use in populating a directory website to a Json file."

7. Building the Website with Bolt. new

• Bolt. new: An AI-powered website builder.
• Functionality: Build a directory website for shoe repair shops in Florida.
• Features:
  • Listings from the attached JSON file.
  • Blog.
  • FAQ.
  • Photos of shoes from Unsplash.
• Example Prompt: "Build a directory for shoe repair shops in Florida. The listings are in the attached JSON file. Include a Blog fact the directory Eng opiz include a Blog and fact and photos of shoes from unsplash."
• Iterative Development: Continuously refine the website by providing feedback to Bolt. new.
• Exporting the Project: Export the project as a ZIP file.

8. Editing the Website Code

• StackBlitz: An online code editor that can be used to edit the exported website code.
• GitHub Integration: Connect the StackBlitz project to a GitHub repository.
• Cursor AI Integration: Clone the GitHub repository into Cursor AI for further editing.
• Local Development: Run the website locally on your computer using npm install and npm run dev.

9. Customization and Enhancements

• Code Modifications: Modify the website code to customize its appearance and functionality.
• SEO Optimization: Optimize the website for search engines.
• Interactive Features: Add interactive features such as user reviews and contact forms.

Conclusion

The video demonstrates how to build a directory website using AI-powered tools and techniques. It covers data scraping, cleaning, and organization, as well as website building and customization. The key takeaway is that AI can significantly accelerate the website development process, allowing individuals with limited coding experience to create functional and visually appealing websites. The process involves iterative refinement, prompt engineering, and a combination of AI-generated code and manual adjustments.