Build and Deploy an AI Automation Platform | Next.js 15, React, Better Auth | Zapier & N8N Clone

Here's a comprehensive summary of the provided YouTube video transcript, focusing on depth and specificity:

Key Concepts

• Workflow Automation Platform Development
• Nodebase (N8N/Zapier alternative)
• Execution Engine
• Triggers (Manual, Google Forms, Stripe Webhooks)
• Integrations (OpenAI, Anthropic, Gemini, Discord, Slack)
• Credential Management (Encryption)
• Execution History & Error Tracking
• Real-time Updates
• Templating Language (Handlebars)
• Topological Sort
• Executor Registry
• Background Jobs (Ingest)
• Server Actions
• Authentication (GitHub, Google OAuth)
• Deployment (Vercel)
• Security Best Practices (Credential Encryption, Webhook Protection)

Summary of YouTube Video Transcript

This tutorial series focuses on building Nodebase, a workflow automation platform similar to N8N or Zapier. The transcript covers the development of the platform in two parts, with this summary focusing on the second part, which aims to complete the production-ready product.

Part 2: Core Functionality and Execution

The second part of the tutorial builds upon the editor, authentication, and payments established in Part 1. It covers the execution engine, triggers, integrations, credential management, execution history, and deployment.

1. Workflow Execution Engine

1.1. Node Data and Props Refinement

• Problem: Initial props for the HTTP request node were passed in three separate props, making them harder to manage.
• Solution: Modified httpRequestNodeData to consolidate these into a single prop, aligning with the form's input structure (endpoint, method, optional body). Renamed the form type to httpRequestFormValues for clarity. Simplified prop handling in the dialogue component by using defaultValues and optional partial of httpRequestFormValues.

1.2. Execute Workflow Button

• Requirement: Implement a button to trigger workflow execution.
• Implementation:
  • Created a new component executeWorkflowButton.tsx.
  • Added a workflowId prop to the button.
  • Rendered the button with "Execute Workflow" text and a flask icon.
  • In editor.tsx, created a hasManualTrigger memoized function to check if any node in the workflow is a manual trigger.
  • Conditionally rendered the executeWorkflowButton in the editor panel if hasManualTrigger is true, passing the workflowId.
  • Disabled the button based on the isPending state of the useExecuteWorkflow hook.

1.3. Ingest Functions and Background Jobs

• Refactoring: Removed old ingest function logic and renamed execute to executeWorkflow.
• Event Structure: Defined the event structure as workflows/execute.workflow.
• Initial Job: The initial job simply sleeps for 5 seconds as a placeholder.
• TRPC Procedure:
  • Created a protected TRPC mutation workflows.execute that accepts a workflow ID.
  • Fetched the workflow using prisma.workflow.findUnique with include: { nodes: true, connections: true }.
  • Used await ingest.send to trigger the background job with the event name workflows/execute.workflow.
  • The TRPC procedure returns the fetched workflow.
• Custom Hook: Created a useExecuteWorkflow hook that wraps the TRPC mutation, handling pending and error states. The onSuccess callback receives the workflow data.

1.4. Workflow Execution Logic (Ingest Job)

• Input Validation: Added a check for event.data.workflowId and throws a nonRetryableError if missing.
• Workflow Fetching: Fetched the workflow with its nodes and connections using prisma.workflow.findUnique or throws.
• Topological Sort:
  • Problem: Nodes need to be executed in a specific order, especially with branching.
  • Solution: Implemented a topologicalSort utility function using the topo-sort package.
    • Created edges from connections (fromNodeId to toNodeId).
    • Added self-edges for nodes without connections to ensure they are included.
    • Handled cyclic dependencies by checking for "cyclic" in the error message and throwing a workflow cycle error.
    • Mapped sorted node IDs back to node objects.
  • The executeWorkflow function now returns the topologically sorted nodes.

1.5. Executor Registry and Node Execution

• Executor Registry:
  • Created executorRegistry.ts in features/executions/lib.
  • Defined executorRegistry as a Record<NodeType, NodeExecutor>.
  • Created a getExecutor function to retrieve the appropriate executor based on node.type.
• Node Executor Interface:
  • Defined NodeExecutorParams in features/executions/types.ts including data, nodeId, context, step, and publish (for real-time updates).
  • data is a generic Record<string, unknown> to accommodate varying node inputs.
  • context is WorkflowContext (initially empty, expands with each node's output).
  • stepTools includes getStepTools and ingest.
• Executor Implementation (Manual Trigger):
  • Created manualTriggerExecutor.ts in features/triggers/manualTrigger.
  • The executor simply passes through the context, emitting loading and success states via the publish function.
• Executor Implementation (HTTP Request):
  • Created httpRequestExecutor.ts in features/executions/components/httpRequest.
  • Defined httpRequestData with optional endpoint, method, and body.
  • Error Handling: Added checks for missing endpoint and variableName, throwing nonRetryableError.
  • Request Execution: Used ky (or step.fetch) to make HTTP requests.
  • Response Handling: Parsed JSON responses or fell back to text.
  • Context Update: Spreads the previous context and adds the HTTPResponse under the variableName provided by the user.
  • Key Collision Fix: Introduced variableName as a required field in the UI (dialogue) and executor to prevent overwriting results from multiple nodes with the same output key. Used Zod for validation with a regex for valid variable names.
  • Templating (Handlebars): Integrated Handlebars for dynamic endpoint URLs and request bodies.
    • Registered a JSON.stringify helper for object serialization.
    • Used handlebars.compile to process template strings with context from previous nodes.
    • Added error handling for invalid JSON parsing and potential endpoint compilation issues.
• Real-time Updates (Ingest Realtime):
  • Setup: Installed ingest-realtime package. Configured client.ts with realtimeMiddleware.
  • Channels: Defined channels (e.g., httpRequestChannel) with topics (status) and types (nodeId, status - loading, success, error). Registered channels in functions.ts.
  • Publishing: Destructured publish from the ingest context in executors and used publish(channel.topic(nodeId, status)) to emit events.
  • Custom Hook (useNodeStatus): Created a hook to subscribe to ingest real-time events, filter by channel, topic, and node ID, and update the node's status indicator (loading, success, error).
  • Server Actions (actions.ts): Created server actions to fetch real-time subscription tokens.
  • UI Integration: Used useNodeStatus in node components (e.g., httpRequestNode.tsx) to display dynamic statuses.
  • Error Handling: Wrapped request execution in try...catch blocks to publish error statuses. Configured retries: 0 in ingest functions for faster failure feedback during development.

2. Trigger Nodes

2.1. Google Forms Trigger

• Node and Dialogue: Created googleFormTriggerNode.tsx and googleFormTriggerDialogue.tsx, copying and adapting from the manual trigger.
• Webhook URL Generation: Dynamically generated a webhook URL (/api/webhooks/google-form?workflowId=...) using process.env.NEXT_PUBLIC_APP_URL or localhost fallback.
• UI: Included a copy-to-clipboard button for the webhook URL and detailed setup instructions for Google Apps Script.
• Google Apps Script: Provided a TypeScript utility (generateGoogleFormScript) to generate the necessary Google Apps Script code, including replacing the webhook URL.
• Webhook Route: Created /api/webhooks/google-form/route.ts to handle POST requests.
  • Extracted workflowId from query parameters.
  • Parsed the request body, extracting relevant form data (e.g., formData, workflowId).
  • Triggered an ingest job using a new sendWorkflowExecution utility function, passing the workflowId and formData as initial data.
• Executor: Created a simple googleFormTriggerExecutor that primarily emits loading and success states, passing context to the next node.
• Ingest Integration: Registered the googleFormChannel in functions.ts and updated the executeWorkflow function to handle initial data from triggers.
• Testing: Used enro for local tunneling to expose localhost to the internet for Google Forms to send webhooks. Configured Google Apps Script with the enro URL and the workflow ID. Added a nonRetryableError if the workflowId is missing in the webhook request. Added a 200 OK response to the webhook route to prevent retries.

2.2. Stripe Trigger

• Node and Dialogue: Copied and adapted the Google Forms trigger structure for Stripe.
  • Created stripeTriggerNode.tsx, stripeTriggerDialogue.tsx, stripeTriggerExecutor.ts, stripeTriggerChannel.ts, actions.ts, and updated functions.ts.
  • Modified the dialogue to prompt for Stripe-specific configuration (webhook URL, event types - though initially broad).
  • Removed the username field and adapted instructions for Stripe dashboard setup.
• Webhook Route: Created /api/webhooks/stripe/route.ts.
  • Extracted workflowId and parsed the request body, including event metadata (eventId, eventType) and raw data.
  • Triggered an ingest job with workflowId and stripeData as initial context.
  • Added a 200 OK response to prevent retries.
• Testing: Used Stripe CLI (stripe listen) and enro for local testing. Configured Stripe webhooks in the Stripe dashboard to point to the enro forwarded URL. Added paymentIntentSucceeded event as an example.
• Security: Noted the lack of webhook security (anyone could trigger the endpoint) and suggested implementing signature verification using Stripe's signing secret or a universal protection mechanism like Swix.

3. AI Nodes (Gemini, OpenAI, Anthropic)

• Prerequisites: Ensured AI SDK packages (ai, ai-sdk-google, ai-sdk-openai, ai-sdk-anthropic) and corresponding Prisma schema types were added.
• Node and Dialogue Implementation:
  • Copied and adapted the HTTP request node/dialogue structure for each AI provider (Gemini, OpenAI, Anthropic).
  • Form Schema: Defined schemas including variableName, model (initially a dropdown, later hardcoded due to reliability issues), systemPrompt (optional), and userPrompt (required).
  • Model Selection: Initially attempted a dynamic model selection dropdown but encountered issues with type safety and API version compatibility. Recommended hardcoding a working model (e.g., gemini-2.0-flash) and noted the need for future refactoring for dynamic model selection.
  • UI Updates: Modified node descriptions to reflect AI-specific prompts.
• Executor Implementation:
  • Created specific executors (e.g., geminiExecutor.ts).
  • Templating: Used Handlebars for dynamic systemPrompt and userPrompt based on context from previous nodes.
  • AI Client Initialization: Used createGoogleGenerativeAI, createOpenAI, createAnthropic from AI SDK, passing API keys (initially from environment variables, later from credentials).
  • AI Call: Utilized step.ai.v[Provider].generateText for AI model interaction.
  • Context Update: Stored the AI response (text or AIResponse) in the context under the variableName.
  • Error Handling: Implemented checks for missing required fields (variableName, userPrompt, credentialId) and published error statuses.
  • Real-time Status: Integrated publish calls for loading, success, and error states using the respective AI channels.
• Credential Integration:
  • Schema: Added Credential model to schema.prisma with name, value (API key), type (enum: OpenAI, Anthropic, Gemini), userId, and relations to User and Node.
  • TRPC Router: Created credentials router with CRUD operations (create, remove, update, get, getMany, getByType). Added protection for user ID to prevent credential access across users. Noted the need for credential encryption in production.
  • Hooks: Developed hooks (useCredentials, useCreateCredential, useRemoveCredential, useUpdateCredential, useSuspenseCredential, useCredentialsByType) mirroring the workflow hooks.
  • UI: Implemented a credentials management page (credentials/page.tsx) with list, create, edit, and delete functionality. Added a dropdown in AI node dialogues to select stored credentials.
  • Executor Update: Modified AI executors to fetch the selected credential using step.run('get-credential') and use its value (API key) for AI client initialization. Removed direct reliance on environment variables for API keys.

4. Credentials Management

• Schema: Defined the Credential model in schema.prisma with id, name, value (encrypted), type (enum: OpenAI, Anthropic, Gemini), userId, and relations to User and Node. Added credentialId to the Node model.
• TRPC Router: Implemented CRUD operations for credentials, ensuring user-specific access and protection against ID injection. Added getByType for dropdown population.
• Hooks: Created corresponding hooks (useCredentials, useCreateCredential, etc.) for client-side data management.
• UI: Developed a dedicated credentials management page with list, create, edit, and delete views, including pagination and search.
• Encryption:
  • Problem: Storing API keys (credentials) as plain text in the database is a security risk.
  • Solution: Used the cryptor npm package to encrypt/decrypt credential values.
    • Generated an encryption key stored in environment variables.
    • Created encrypt and decrypt utility functions in lib/encryption.ts.
    • Modified TRPC procedures (create, update) to encrypt values before saving and decrypt them when fetching.
  • Recommendation: Advised using more robust solutions like AWS Secrets Manager for production environments due to the limitations of symmetric encryption with a single key.

5. Authentication (GitHub & Google OAuth)

• GitHub OAuth:
  • Setup: Registered a GitHub OAuth app in the GitHub Developer Portal, obtaining client ID and client secret. Configured homepage and callback URLs (/api/auth/callback/github).
  • Environment Variables: Stored GITHUB_CLIENT_ID and GITHUB_CLIENT_SECRET in .env.
  • Auth.js Integration: Implemented signIn('github') function within the login/register forms, handling success (redirect) and error (toast notification) states.
• Google OAuth:
  • Setup: Configured Google OAuth credentials via the Google Cloud Console.
    • Created a new project (Nodebase prod).
    • Configured the OAuth consent screen (User Type: External, added app name and user email, skipped app logo for verification avoidance).
    • Created an OAuth client ID for a web application, specifying JavaScript origins (localhost:3000) and redirect URIs (/api/auth/callback/google).
    • Published the app to make it available to users.
  • Environment Variables: Stored GOOGLE_CLIENT_ID and GOOGLE_CLIENT_SECRET in .env.
  • Auth.js Integration: Implemented signIn('google') function, similar to GitHub, handling success and error states. Noted potential issues with duplicate email addresses linking accounts and the need for proper user management.

6. Deployment to Vercel

• Build Preparation: Ran npm run build locally to catch build errors (especially type checking) before deployment. Advised against skipping type checks (typescript.ignoreBuildErrors).
• Vercel Setup:
  • Created a Vercel account.
  • Imported the project from GitHub.
  • Configured environment variables in Vercel's project settings:
    • Database URL (Neon connection string).
    • Better-AL URL (updated to the Vercel deployment URL).
    • NEXT_PUBLIC_APP_URL (updated to the Vercel deployment URL).
    • GitHub/Google Client IDs and Secrets (updated for production).
    • Encryption Key (updated to a secure production key).
  • Database: Recommended using Neon's branching feature for managing development and production databases. Used the development branch URL for the DATABASE_URL initially, suggesting a separate production branch for deployment.
• Redeployment: Redeployed the project on Vercel after updating environment variables.
• Post-Deployment: Verified GitHub and Google sign-in functionality with the production URLs.

7. Execution History

• Schema: Defined the Execution model in schema.prisma with id, workflowId, ingestEventId, status (enum: running, success, failed), startedAt, completedAt (optional), error (optional string), and errorStack (optional string, db.Text). Added a relation to Workflow.
• TRPC Router: Created executions router with get and getMany procedures.
  • getMany: Fetched executions with included workflow data (id, name), ordered by startedAt descending. Filtered by workflow.userId.
  • get: Fetched a single execution, including workflow data.
• Hooks: Developed useSuspenseExecutions and useSuspenseExecution hooks.
• Server Components: Implemented paramsLoader and prefetch functions for fetching execution data on the server.
• Client Components:
  • Created executions/page.tsx to orchestrate the loading and display of execution data.
  • Developed executions/components/executions.tsx (main container), executions/components/executionsHeader.tsx, executions/components/executionsList.tsx, and executions/components/executionsPagination.tsx.
  • Created executions/components/executionItem.tsx to display individual execution details (status icon, workflow name, timing, duration).
  • Implemented getStatusIcon to visually represent execution status (success, failed, running).
  • Developed executions/components/execution.tsx for the detailed view of a single execution, including workflow link, status, timing, duration, output, and error details (message and stack trace) using a collapsible component.
• Ingest Integration:
  • Execution Creation: Modified functions.ts to create an Execution record with status: 'running' and ingestEventId upon starting a workflow job. Used CU_ID2 for ingestEventId to ensure uniqueness.
  • Execution Update: Added updateExecution step to update the Execution record with status: 'success', completedAt, and output (context) upon successful completion.
  • Failure Handling: Added an onFailure handler in executeWorkflow to update the Execution record with status: 'failed', error, and errorStack.

8. Deployment Preparation

• Code Cleanup: Removed development-specific configurations like retries: 0 in ingest functions, replacing them with conditional logic based on process.env.NODE_ENV.
• Local Build Test: Ran npm run build locally to ensure the project builds successfully, catching potential type errors or configuration issues before deploying to Vercel.
• Vercel Deployment:
  • Connected Vercel to GitHub repository.
  • Configured environment variables for production (database URL, Better-AL URL, NEXT_PUBLIC_APP_URL, GitHub/Google OAuth credentials, encryption key).
  • Updated OAuth redirect URIs and homepage URLs to production domains.
  • Redeployed the project on Vercel.
  • Verified functionality post-deployment.

This summary provides a detailed breakdown of the technical aspects covered in the transcript, including specific implementation details, problem-solving approaches, and architectural decisions made during the development of the Nodebase platform.