Microsoft’s POML Explained With a Simple Example! (+Ollama)

Key Concepts

• Poml (Prompt Orchestration Markup Language): An open-source markup language, similar to HTML, designed to standardize prompts for large language models (LLMs).
• Olama: A model provider used to run LLMs locally.
• Prompt Engineering: The process of designing and refining prompts to elicit desired responses from LLMs.
• Standardized Prompts: Using a consistent and structured format for prompts to ensure predictable and reliable results from LLMs.
• Multimedia Data: Incorporating various data types like images, CSV files, and JSON data into prompts.
• Few-Shot Prompting: Providing a few examples in the prompt to guide the LLM's response.
• Agentic System: An AI system capable of autonomous decision-making and action.

Poml: Prompt Orchestration Markup Language

Introduction to Poml

• Poml is introduced as a solution to the inconsistency and errors that arise from using unstructured text prompts with LLMs.
• It's likened to HTML, using tags to structure prompts.
• The goal is to provide a standardized format for prompts, leading to more consistent results.
• Poml is open-source and available on GitHub.

Basic Poml Structure

• The basic structure involves tags such as RO (Role), Task, Multimedia, and DataSource.
• Example: <RO>...</RO>, <Task>...</Task>, <Multimedia>...</Multimedia>, <DataSource>...</DataSource>.
• These tags define the role of the LLM, the task it should perform, and the data sources it should use.
• The example shows how to integrate Poml with Olama using a few lines of code.

Setting Up the Environment

• Prerequisites:
  • Download and install Olama.
  • Pull the GPT OSS model using the command olama pull GPT OSS.
• File Creation:
  • Create two files: app.py (for the application code) and orders_qa.poml (for the Poml code).
• VS Code Extension:
  • Install the Poml extension for Visual Studio Code to preview Poml files.

Creating a Poml File (orders_qa.poml)

• Role: Defines the LLM as a chatbot agent answering customer questions.
• Task: Instructs the LLM to answer questions based on provided data.
• DataSource: Includes various data sources:
  • CSV files (e.g., orders.csv, order_line.csv).
  • JSON data (e.g., order_instruction.json).
• Tags Used:
  • CPTable: Represents a CSV table.
  • StepwiseInstruction: Provides step-by-step instructions.
• Example Data:
  • orders.csv: Contains order data (e.g., order number, total amount).
  • order_instruction.json: Contains instructions for the chatbot (e.g., allowed questions, response format).
• Question: A sample question is included: "How much did I pay for my last order?"

Previewing the Poml File

• The VS Code extension allows previewing the Poml file in two ways:
  • XML-like version: Shows the structured XML format.
  • Rendered version: Displays how the prompt will look to the LLM, with data from CSV and JSON files included.
• The preview shows that data from CSV files and JSON instructions are automatically included in the prompt.

Creating the Application (app.py)

• Import Statements:
  • from poml import poml
  • import requests
  • import json
• Step 1: Load and Render Poml File:
  • Use POML("orders_qa.poml", chat=True) to load and render the Poml file.
• Step 2: Combine Messages (if needed):
  • The code includes a function to combine multiple messages into a single prompt.
• Step 3: Call Model:
  • Define the endpoint and model name (e.g., GPT OSS latest).
  • Set stream=False.
  • Send the full prompt to the LLM.
  • Get the data as JSON and print it.

Running the Application

• Run the code using python app.py.
• The full prompt (role, task, data, question) is printed for reference.
• The LLM's response is displayed.
• The example shows that the initial response might be inaccurate, but subsequent runs can yield correct answers.

Advanced Poml Example: Financial Analysis

• Financial Analysis Poml:
  • Includes more advanced features like Excel sheets and images.
  • Uses the <Multimedia> tag to include images.
• Model Change:
  • The model is changed to quen 2.5 vision language model to handle images.
• Automatic Data Conversion:
  • Poml automatically converts images to Base64 format.
• Running the Advanced Example:
  • The code is modified to use the financial_analysis.poml file.
  • The application is run again, and the LLM processes the image and provides a response.

Conclusion and Further Resources

• Poml simplifies the creation of complex prompts with multiple data types.
• The speaker provides the code in the description for users to try out.
• The speaker recommends watching another video about Microsoft's open-source agentic system.