Visual Studio Copilot in the Vibe Coding Olympics: Did Underdog Caught Up To The Pack?

GitHub Copilot Code Editor Comparison: A Detailed Analysis

Key Concepts:

• GitHub Copilot (Agent Mode): An AI-powered code completion and generation tool within Visual Studio Code.
• OpenAI API: Used for generating images via DALL-E 2, DALL-E 3, and JPT-4 models.
• Agentic Behavior: The ability of the tool to proactively perform tasks without explicit user instruction.
• GitHub Pages: A service for hosting static websites directly from a GitHub repository.
• API Key: A secret key required to access and use the OpenAI API.
• Base64 Encoding: A method of encoding binary data as ASCII characters, often used for embedding images in web pages.

Initial Setup and Prompt

• The video focuses on evaluating GitHub Copilot's performance in a specific coding task.
• The task involves creating an index.html file that uses JavaScript to generate three images in parallel using DALL-E 2, DALL-E 3, and JPT-4, displaying them on the page with loading indicators and error handlers.
• The prompt given to Copilot is: "Read this page [OpenAI API documentation]. I want you to make index html that loads JavaScript. It should use some prompt same prompt to generate three images using each of the image models in parallel. Use GPT image one deli 2 D3 It meant to show progress via images from Delhi 2 to three to Jupy 1 image being the newest. So use fetch. I will give you the API key. Do it in this folder."
• Copilot is used in "agent" mode, which is intended to be the most proactive and autonomous mode.

Performance and Interactions

• The process requires multiple interactions with Copilot.
• Initial interactions involve Copilot fetching the OpenAI API documentation.
• The tool requires user confirmation for various actions, including fetching web pages and executing terminal commands, which is considered a friction point and reduces its "agentic" behavior.
• The initial attempt to fetch the documentation fails, indicating potential issues with network access or the tool's ability to handle external resources.
• Copilot successfully creates an index.html file with approximately 300 lines of code.
• The user then prompts Copilot to open the index.html file in a browser.

Error Handling and Model Specifics

• The initial implementation results in errors, specifically an "Invalid value GP4" error, indicating that the model parameter was not correctly set.
• Copilot displays error indicators, which is a positive feature.
• The user provides the error message to Copilot, which then attempts to correct the model selection.
• After updating the model, a new error arises related to the response format, where the JPT-4 model returns a base64 encoded image instead of a URL.
• The user informs Copilot about the base64 encoding issue and instructs it to handle the JSON response correctly.
• Copilot modifies the code to handle the base64 encoded image data.

GitHub Pages Deployment

• The user prompts Copilot to add a README file and prepare the project for publishing to GitHub Pages.
• Copilot creates a basic README file and a .gitignore file.
• The user then asks Copilot to use the terminal with Git to push the project to GitHub.
• Copilot requires multiple confirmations for each terminal command, further highlighting the lack of full agentic behavior.
• Copilot guides the user through the steps to enable GitHub Pages for the repository, but it doesn't automatically open the correct settings page, requiring an additional user action.

Evaluation and Cost Analysis

• The process took 19 interactions with Copilot.
• The tool felt slow at times and initially failed to fetch the documentation.
• Copilot does not automatically save the API key in local storage, requiring the user to re-enter it each time.
• There is no built-in functionality to save the generated images.
• The tool lacks a direct link to the OpenAI API key management page.
• The generated code does not include copyright or license information.
• The design of the generated webpage is considered acceptable, potentially even better than the presenter's original implementation.
• The cost of using GitHub Copilot is estimated at $10 per month, which includes 300 GPT-4 calls.
• Based on the 7 messages/calls used in the task, the estimated cost for this specific task is $0.23.

Conclusion

• GitHub Copilot performed reasonably well, close to other similar tools like Windsurf and Cursor.
• The presenter was more impressed than expected, given previous negative perceptions of Copilot.
• The design of Copilot's chat interface is considered superior to Cursor, making it easier to read and follow the conversation.
• Despite its strengths, Copilot's lack of full agentic behavior and the need for frequent user confirmations are notable drawbacks.