Introducing the GitHub Models tab: Manage & test your AI prompts

Key Concepts:

• GitHub Models Tab: A new feature in GitHub repositories for managing and experimenting with AI prompts.
• Prompts: Instructions given to AI models to generate specific outputs.
• .prompt.yml: A file used to store prompt configurations, including test data.
• Model Parameters: Configurable settings that influence the behavior of AI models.
• Compare Tab: A feature for running experiments to compare different prompts and models.
• Evaluators: Tools used to automatically assess the quality of AI model outputs.
• LLM as a Judge: Using a Large Language Model to evaluate the output of another LLM.
• Code Snippets: Pre-written code examples for implementing prompts in projects.

1. Introduction to GitHub Models Tab

The GitHub Models tab is a new feature designed to help developers build AI features into their applications. It provides a centralized location for managing and experimenting with AI prompts directly within a repository.

2. Prompt Management and Configuration

• Prompts are stored as files within the repository, such as the .prompt.yml file.
• The Models tab allows users to view, edit, and commit changes to prompts like any other code.
• The .prompt.yml file can include test data to evaluate prompt performance.

3. Experimentation with the Compare Tab

• The Compare tab enables users to run experiments to understand how different prompts and models affect the output.
• Users can compare variations in prompts, models, parameters, system prompts, and user prompts side-by-side.
• The page displays sample data included in the .prompt.yml file.

4. Evaluation of Model Outputs

• Evaluators are used to automatically assess the quality of AI model outputs.
• Evaluators can be applied to see which combinations of prompts and models meet desired output criteria.
• The results of the evaluators are displayed directly in the dataset grid.
• An existing evaluator uses an LLM as a judge, resulting in a pass or fail determination.

5. Example of Evaluator Application

• The video demonstrates running two different models against two different prompts.
• The results show whether each request (prompt + input) passed the evaluator, along with other metadata.
• In the example, all four requests passed the evaluator.

6. Model Access Control

• Organization administrators can control which models are allowed for use.
• Admins can either allow all models or specify a list of allowed or disallowed models.

7. Collaboration and Implementation

• The Models tab facilitates collaboration on prompts within a team.
• Code snippets are provided in the Models page to help developers implement prompts in their projects.
• Users can access over 40 models using their GitHub account.

8. Conclusion

The GitHub Models tab provides a comprehensive environment for managing, experimenting with, and implementing AI prompts. It streamlines the process of building AI features into applications by offering tools for prompt configuration, comparison, evaluation, and collaboration. The integration with GitHub repositories allows for version control and collaborative development of AI-powered features.