Python SaaS Project That Scales: Job Finder

Key Concepts

• Scalable SaaS Application: A software-as-a-service application designed to handle a growing number of users and requests without performance degradation.
• Django: A high-level Python web framework that encourages rapid development and clean, pragmatic design.
• Langchain: A framework for developing applications powered by language models, enabling agents to interact with tools and data.
• Docker Compose: A tool for defining and running multi-container Docker applications, simplifying the management of complex application stacks.
• Celery: A distributed task queue system that allows for asynchronous execution of tasks, crucial for scaling web applications.
• Redis: An in-memory data structure store, used here as a message broker and results backend for Celery.
• Bright Data: A web data platform that provides tools and infrastructure for large-scale web scraping and data collection.
• API Key: A unique identifier used to authenticate requests to an API.
• Environment Variables (.env): A mechanism for storing configuration settings, such as API keys, separately from the application code.
• Pyantic: A data validation and settings management library for Python, used here for defining structured output models.
• Agent (Langchain): An entity that can use tools to interact with its environment and achieve a goal.
• Tools (Langchain): Functions that an agent can call to perform specific actions, such as scraping data.
• Database Models: Representations of database tables in Django, used to store application data.
• Asynchronous Processing: Executing tasks in the background without blocking the main application thread, improving responsiveness and scalability.
• Task Queues: Systems that manage the execution of background tasks, allowing for distribution and scheduling.
• Message Broker: A component that facilitates communication between different parts of a distributed system, such as Celery workers and the Django application.
• Results Backend: A storage mechanism for Celery task results.
• Containerization (Docker): Packaging applications and their dependencies into isolated containers for consistent deployment.

Building a Scalable AI-Powered Job Listing Scraper

This video details the process of building a professional, scalable Python SaaS application using Django, Langchain, Docker Compose, Celery, Redis, and Bright Data. The application's core functionality is to scrape job listings based on user-provided free-text descriptions. The tutorial emphasizes principles and techniques for scaling SaaS applications, starting with a basic prototype and progressively adding more robust features for concurrency and asynchronous processing.

1. Project Setup and Initial Prototype

The project begins with setting up a local Django development environment. The presenter recommends using uv as a Python package manager for its speed and efficiency, though pip can also be used.

• Required Packages: django, langchain, requests, pyantic. celery, redis, and docker are introduced later.
• Django Project Initialization:
  • Create a project directory (e.g., job-sas).
  • Use uv run django-admin startproject core job-sas to create the Django project structure.
  • Create two Django apps: accounts for authentication and jobs for the core functionality.
• Authentication (accounts app):
  • Leverages Django's built-in user model and forms.
  • A custom signup view is implemented to handle user registration. This view uses UserCreationForm for form handling and django.contrib.auth.login to log in the user upon successful signup.
  • urls.py in the accounts app is configured to include Django's built-in login and logout views, as well as the custom signup view.
  • The accounts URLs are included in the main core/urls.py under the /off/ path.
  • HTML templates (login.html, signup.html) are created for the authentication forms.
  • Key Point: Django's "batteries included" philosophy simplifies authentication setup.
• Initial Job Functionality (Synchronous Prototype):
  • A services.py file is created within the jobs app to house the core scraping logic.
  • The initial approach focuses on a synchronous, single-user experience to understand the core mechanics before scaling.

2. Integrating Data Scraping with Bright Data

To handle large-scale data access, the project integrates with Bright Data, a platform for web scraping.

• Bright Data Setup:
  • Users need to create a free account on Bright Data.
  • Navigate to "Web Scrapers" -> "Web Scrapers Library" to find pre-built scrapers.
  • The "Discover by Keyword" scraper for LinkedIn and Glassdoor are identified as crucial for job listing retrieval.
  • An API key is generated from Bright Data's "Account Settings" -> "User Management" -> "API Key".
• Environment Variables:
  • A .env file is created at the project root to store sensitive information like BRIGHTDATA_API_KEY and OPENAI_API_KEY.
  • The python-dotenv package is used to load these variables into the application's environment.
• Bright Data API Interaction:
  • The requests library is used to interact with the Bright Data API.
  • The API request involves specifying the dataset_id (scraper to use), parameters (keywords, location, experience level, etc.), and the data payload for filtering.
  • Asynchronous Nature: Bright Data's API returns a snapshot_id for asynchronous job processing. The data is not immediately available.
  • Retrieving Results: Two endpoints are used: one to check the progress of a snapshot and another to retrieve the snapshot data once it's ready.
• Service Functions:
  • search_jobs_on_linkedin and search_jobs_on_glassdoor functions are created in services.py.
  • These functions are designed to accept parameters like location, keyword, country, experience_level, etc.
  • They construct the Bright Data API request, submit it, and retrieve the snapshot_id.
  • Type Hinting: Type hints are used for function parameters to guide the Langchain agent.
  • Limiting Results: A limit_per_input parameter is used to control the number of results returned, balancing data retrieval with token usage.

3. Langchain Integration for AI-Powered Job Search

Langchain is used to enable an AI agent to interact with the scraping tools and process user prompts.

• Langchain Tools:
  • The @tool decorator from langchain.tools is used to convert the service functions (search_jobs_on_linkedin, search_jobs_on_glassdoor) into Langchain tools.
  • Tool Descriptions: Crucial for the agent to understand when and how to use each tool. These descriptions detail the tool's purpose, parameters, and expected output.
• Agent Creation:
  • create_agent from langchain.agents is used to instantiate an agent.
  • A language model (e.g., ChatOpenAI with gpt-4o-mini for cost-effectiveness) is selected.
  • The defined tools are passed to the agent.
• Agent Invocation:
  • The agent.invoke method is used to send a prompt to the agent.
  • A message_history is provided, including a system_message to define the agent's role and a human_message containing the user's prompt.
  • The agent uses its tools to find relevant job listings and then summarizes the findings.
• Structured Output with Pyantic:
  • Pyantic models (JobListing, JobListingResult) are defined in lm_schemas.py to enforce a structured output format for the job listings.
  • These models include fields like title, job_url, job_type, level, summary, salary, posted, and applicants, with descriptions to guide the LLM.
• Django View for Job Search:
  • A search_job_view is created in jobs/views.py.
  • It handles POST requests, extracts the prompt from the request, and calls the search_jobs_with_agent function.
  • The results are rendered in a jobs/results.html template.
  • Initial Error Handling: Basic error handling is implemented, and the search.html template is updated to use a textarea for the prompt.
  • URL Configuration: A urls.py is created in the jobs app for the search functionality, and it's included in the main core/urls.py.

4. Implementing Asynchronous Processing with Celery and Redis

To achieve scalability, the application transitions to asynchronous task processing using Celery and Redis.

• Docker Compose Setup:
  • A Dockerfile is created for the Django application, defining the build process and dependencies.
  • A docker-compose.yml file is created to define three services:
    • web: The Django application (backend and frontend).
    • redis: The message broker and results backend.
    • celery_worker: The Celery worker process.
  • Commands: Specific commands are defined in docker-compose.yml for the web service (running the Django development server) and the celery_worker service (running the Celery worker).
• Celery Configuration:
  • celery and redis packages are installed using uv.
  • Celery settings are added to core/settings.py, configuring the broker URL and result backend to use Redis.
  • A celery.py file is created in the core app to initialize the Celery app and enable task auto-discovery.
• Database Models for Asynchronous Workflow:
  • New Django models are defined in jobs/models.py to manage the asynchronous workflow:
    • Snapshot: Stores Bright Data snapshot IDs, readiness status, and raw data.
    • JobListingResult: Stores individual job listing details extracted from the LLM.
    • LLMResult: Represents a user's job search request, including its status (pending, processing, ready), prompt, owner, and associated snapshots and job listings.
  • Relationships: Foreign keys are established to link Snapshot and JobListingResult to LLMResult, and LLMResult to the Django User model.
• Celery Tasks:
  • A tasks.py file is created in the jobs app.
  • A @shared_task decorator is used to define asynchronous tasks.
  • process_snapshots_and_summarize: This task takes an LLMResult ID, retrieves associated snapshots, fetches data from Bright Data using helper functions (get_data), and then uses a Langchain model (e.g., ChatOpenAI with gpt-4o-mini) and Pyantic models to summarize the job listings.
  • The summarized job listings are then saved as JobListingResult objects in the database.
  • The LLMResult status is updated to processing and then ready.
• Updating Langchain Tools for Asynchronous Operations:
  • The search_jobs_on_linkedin and search_jobs_on_glassdoor tools are modified.
  • They now accept an llm_result_id and are responsible for creating Snapshot objects in the database and returning a success message, rather than waiting for the data.
  • Helper functions is_ready and get_data are created in services.py to interact with the Bright Data API for checking snapshot status and retrieving data.
• Modified Django Views:
  • The search_job_view is updated to:
    • Create an LLMResult object with a pending status.
    • Pass the llm_result_id to the agent.
    • Instruct the agent to first set the title of the LLMResult using a new set_results_title tool.
    • Then, invoke the agent to trigger the scraping tools.
    • The view now renders a jobs/job_search.html template, which displays the prompt, results, and potential errors.
  • A new results_list_view is created:
    • This view iterates through pending LLMResult objects for the logged-in user.
    • It checks if all associated snapshots for a result are ready using the is_ready helper function.
    • If all snapshots are ready, it schedules the process_snapshots_and_summarize Celery task using .delay().
    • It also retrieves and displays all LLMResult objects for the user, along with snapshot status information, in a jobs/results_list.html template.

5. Deployment and Styling

The final steps involve deploying the application using Docker Compose and styling the user interface.

• Docker Compose Build and Run:
  • docker compose build to build the Docker images.
  • docker compose up to start the services (web, redis, celery worker).
• Database Migrations:
  • Run docker compose exec web bash to access the Django container.
  • Execute uv python manage.py makemigrations and uv python manage.py migrate to create the database tables.
• Styling with Cursor:
  • The presenter uses Cursor (an AI-powered code editor) to generate HTML and CSS for styling the application's views, including the login, signup, search, and results pages.
  • The focus is on creating a functional and presentable UI without dwelling on manual CSS coding.
• Testing and Debugging:
  • The application is tested by submitting job search queries.
  • Initial issues, such as incorrect URL paths, missing dependencies, and incorrect model field references, are identified and resolved.
  • The integration of multiple data sources (LinkedIn and Glassdoor) is tested, and the process of combining results is demonstrated.
  • The asynchronous nature of Celery is observed, with tasks being processed in the background.

Conclusion and Key Takeaways

The video successfully demonstrates how to build a scalable AI-powered SaaS application by integrating various technologies:

• Scalability through Asynchronous Processing: Celery and Redis are essential for handling concurrent requests and background tasks without blocking the main application.
• Efficient Data Acquisition: Bright Data provides a robust platform for large-scale web scraping, enabling access to real-time job listing data.
• AI Integration: Langchain and LLMs (like GPT-4o-mini) enable intelligent processing of user prompts and structured data extraction.
• Structured Data Handling: Pyantic models ensure consistent and predictable data formats for LLM interactions and database storage.
• Containerization for Deployment: Docker Compose simplifies the management and deployment of the multi-service application stack.
• Iterative Development: The approach of starting with a basic prototype and progressively adding complexity (authentication, scraping, AI, async processing) is effective for managing large projects.

The final application allows users to input job descriptions, and the system asynchronously scrapes relevant listings from multiple platforms, summarizes them using AI, and presents the results in a structured format. The code for the project is made available on GitHub.