Web AI on the IDE

Key Concepts

• Context Switching: The mental effort and time lost when a developer shifts focus between different tools and applications during their workflow.
• AI-Powered IDEs: Integrated Development Environments that leverage Artificial Intelligence, particularly Large Language Models (LLMs), to assist developers.
• Cursor: An AI-first code editor that integrates LLMs directly into the development workflow, aiming to reduce context switching.
• VS Code: Visual Studio Code, a popular open-source code editor that serves as a platform for many AI-powered IDE innovations.
• Electron: An open-source framework used to build cross-platform desktop applications, forming the foundation of VS Code and other applications like Slack and Discord.
• Web AI: The concept of running AI models directly within a web browser or, in this context, within an IDE using web technologies.
• OpenMetadata Stack: An industry-standard framework for data engineering, used here for data catalog and lineage.
• Elasticsearch: A search and analytics engine used to store and query metadata for the data catalog.
• Gemini: Google's LLM, used here to process natural language queries for data catalog searches and provide summaries.
• Whisper.js: A JavaScript library for running OpenAI's Whisper model locally, enabling real-time speech-to-text transcription.
• Transformers.js: A library that allows running machine learning models, including Whisper, directly in the browser or IDE.
• Dictation Tools: Software that converts spoken language into written text, aiming to speed up content creation.

Building Cursor Extensions with JavaScript and Local Models

This presentation explores how to enhance software development workflows by building custom extensions for IDEs, specifically focusing on reducing context switching for data developers and integrating AI capabilities locally.

The Problem of Context Switching in Development

• Traditional Workflow: Developers typically juggle multiple applications: IDE, browser (for Stack Overflow, documentation, GitHub), project management tools, and communication platforms (Slack, Teams, Google Meets).
• Impact: This constant context switching leads to significant interruptions, with developers experiencing an average of 13 interruptions per hour. Recovering focus after an interruption can take considerable time.
• AI-Powered IDE Solution: AI-powered IDEs aim to mitigate this by bringing many of these tools directly into the code editor, reducing context switching by an estimated 40%.

The Rise of AI-Assisted Development and VS Code's Role

• AI IDE Landscape: While AI IDEs are gaining traction, VS Code remains dominant in developer preference (76% in a recent survey).
• VS Code as a Platform: VS Code's open-source nature (since 2015) and its adoption of features like Copilot Chat (2023) have made it a foundational platform for AI-driven development. Many innovative solutions, including Cursor, Project IDX (now Firebase Studio), Wind Surf, and Curo, are built upon or inspired by VS Code's architecture.
• Extensibility: Anything developed for VS Code has a high probability of working in other VS Code-based IDEs.

Understanding VS Code Architecture

• Electron Framework: VS Code is built on Electron, an open-source framework for creating cross-platform desktop applications.
• Underlying Technologies: Electron combines Chromium (for the UI) and Node.js (for backend processes).
• Development Technologies: This means VS Code extensions can be built using standard web technologies: JavaScript, HTML, and CSS.

Addressing the Underserved Data Developer Workflow

• Data Development Differences: Data development workflows, focused on data transformation, differ from traditional software development.
• Key Pain Points: The most significant context switching pain points for data developers occur during the searching and understanding of data within data catalogs and lineage tools.
• Proposed Solution: Integrate data catalog and lineage interfaces directly into the IDE, treating the IDE as a manipulable web page.

Building a Data Catalog and Lineage Extension

• Concept: Create a new tab within the IDE (e.g., next to the terminal) to house data catalog and lineage interfaces.
• Technology Stack:
  • OpenMetadata Stack: For data engineering standards.
  • Airflow: To orchestrate data set execution.
  • Elasticsearch: To store and query metadata for the data catalog.
  • Gemini: To enable natural language queries and provide summaries of search results.
• Functionality:
  • Users can search data using natural language.
  • Gemini processes queries and summarizes results at the top of the interface.
  • Users can view column details and data lineage interactively.
• Benefit: This reduces context switching by bringing essential data discovery tools directly into the developer's primary workspace.

Leveraging AI for Accelerated Development: Dictation

• Next Step: Integrate existing agentic AI interfaces within the IDE to further accelerate development.
• Context is Key: LLMs require high-quality, detailed context and specifications for effective results, especially in data development with complex business rules.
• Dictation as a Solution: Dictation tools can significantly speed up the process of providing this context.
• Market Solutions: Tools like Aqua, Whisper Flow, and Willow offer dictation capabilities, promising up to a four-fold reduction in typing time.
• Cost Barrier: These commercial tools can be expensive, especially for developers in regions with devalued currency.

Developing a Local, Free, and Offline Dictation Extension

• Goal: Create a dictation feature within the IDE extension that addresses the limitations of commercial tools.
• Requirements:
  • Real-time Transcription: Immediate text output after speaking.
  • Accuracy: High transcription accuracy to minimize editing.
  • Free: No cost to the user.
  • Offline: No data sent externally, crucial for compliance in fintech environments.
• Web AI Implementation: Utilize Web AI by running models directly within the IDE.
• Technology:
  • Whisper.js: A Hugging Face library for running Whisper locally.
  • Transformers.js: Enables loading and running models locally in real-time.
• Workflow:
  1. Model Download: The first time the extension is used, the user downloads the Whisper model weights (approx. 200 MB).
  2. Voice Input: An interface within the extension captures voice input.
  3. Local Inference: Whisper.js and Transformers.js process the audio locally.
  4. Text Output: The transcribed text is provided to the user.
  5. Contextual Use: The user can copy this transcription and paste it as a prompt into the IDE's AI agent (e.g., Cursor's agent).

Practical Example and Adoption

• Demonstration: An example shows a user dictating a prompt to create a new table for customer segmentation. The transcribed prompt is then fed to Cursor's AI agent, which generates the code.
• Impact: This workflow significantly reduces the time spent typing prompts, realizing the promise of faster development.
• Nubank's Experience: Nubank is already using Cursor extensions internally to accelerate development. An adoption curve shows increasing traction among data developers within five weeks of launch.
• User Feedback: A user quote highlights the extension's usefulness in generating production-ready code quickly.

Further Resources

• An article detailing the implementation of Web AI extensions directly into IDEs is available via a QR code.
• The code used in the presentation is open-sourced for experimentation.
• Contact information for the presenter (X/LinkedIn) is provided for further inquiries.