GitHub Models is here: Better LLM evaluation and prompt versioning

Key Concepts:

• GitHub Models
• Public Preview
• Model Tab
• Side-by-Side Evaluations
• Large Language Models (LLMs)
• Chatbot Feature
• Version Control
• Prompts
• Access Policies
• Data Control
• GitHub Infrastructure

GitHub Models: Public Preview and Functionality

GitHub is launching "GitHub Models" in public preview. This new feature introduces a "model tab" within GitHub repositories. The primary purpose of this tab is to provide teams with a centralized location to conduct "side-by-side evaluations of large language models (LLMs)."

Use Case: Chatbot Feature Development

The video specifically mentions the development of a "chatbot feature" as an example use case. The model tab facilitates the comparison of different LLMs to determine the most suitable model for the chatbot.

Key Features and Controls within GitHub Models

The GitHub Models feature offers several key functionalities:

• Version Control: Enables tracking and managing different versions of the LLMs being evaluated.
• Prompts: Allows for the versioning and management of prompts used to interact with the LLMs. This ensures consistency and reproducibility in evaluations.
• Access Policies: Provides control over who can access and interact with the LLMs and evaluation data.
• Data Control: Offers mechanisms for managing and controlling the data used to train and evaluate the LLMs.

Integration with GitHub Infrastructure

All of these features – version control, prompts, access policies, and data control – are integrated directly within the existing GitHub infrastructure. This allows teams to leverage their existing GitHub workflows and tools for LLM evaluation.

Synthesis/Conclusion:

GitHub Models aims to streamline the process of evaluating and selecting LLMs for various applications, such as chatbot development. By providing a dedicated model tab within GitHub repositories and integrating key features like version control, prompt management, access policies, and data control, GitHub Models offers a centralized and controlled environment for teams to compare and choose the best LLMs for their needs, all within the familiar GitHub ecosystem.