Beaks & Bytes - AI, LLMs, and F5 AI Gateway take flight

Key Concepts

• LLM (Large Language Model): Advanced text-based word generator, like autocomplete but with nuanced understanding.
• Parameters: Connections within an AI model, influencing its learning capacity. More parameters generally mean more capable analysis.
• Training vs. Inferencing: Training creates the model (GPU-intensive), while inferencing uses it (less GPU-intensive, but still significant at scale).
• OpenAI API: A structured JSON request format for interacting with AI models.
• Prompt: The question or input provided to an AI model.
• System Prompt: Operating instructions for the AI model, guiding its behavior.
• User Prompt: The specific query from the user.
• Assistant Role: The AI model's response to the user prompt.
• Tokens: Units of measurement for text complexity and length in AI models (roughly equivalent to words).
• AI Gateway: A containerized solution for securing and managing AI traffic, parsing JSON and inspecting prompts.
• Prompt Injection: Malicious attempts to bypass the system prompt and manipulate the AI model.

What is an LLM?

• LLM stands for Large Language Model.
• At a high level, a text-based LLM like ChatGPT can be thought of as advanced autocomplete.
• LLMs are excellent word generators with a strong understanding of text nuance.

LLMs vs. Traditional AI

| Feature | Traditional AI | Large Language Models (LLMs) the AI gateway is purpose-built to understand these AI API specs.

• AI gateway is a containerized solution, separate from Big IP.
• AI gateway takes learnings from load balancing, app delivery, and security.
• AI gateway is split between a core container system and processor containers.
• The core container system receives OpenAI API requests, normalizes them, and enforces rate limiting and authentication.
• Processors function as security inspection units, looking for things like prompt injection or sensitive data.
• Processors can enrich data or make pass/fail decisions.
• AI gateway can route prompts to different AI model servers based on logic.
• AI gateway should be placed between the web session and the AI protocol spec.

AI Gateway Example Tabletop

1. Prompt Injection: The first processor checks for malicious attempts to bypass the system prompt.
2. Language ID: A processor tags the prompt with its language (e.g., English, Japanese).
3. Spam Detection: A processor looks for repeated questions or variations to prevent spamming the LLM system.
4. System Processor: Adds extra enforcement messages to every prompt.
5. Custom Logic: Allows for custom processors to implement business logic.

Demo: Bird Facts Chatbot

• Two Open Web UI instances are used, both connected to the same backend model (OpenAI GPT3.5 Turbo).
• Both have a system prompt to only talk about birds and bird facts.
• One instance goes directly to the model (unprotected), while the other goes through F5 AI gateway (protected).
• The AI gateway applies prompt injection detection, repetition detection, and system prompt enforcement.

Demo Scenarios

• Easy Mode: Testing the model's ability to stay on track with the system prompt.
• Hard Mode: Sneakily trying to get the model to write Python code by framing it as bird-related.
• Overtly Malicious: Telling the model to ignore previous instructions and become a Python bot.
• Spam: Repeatedly asking the same question to test spam detection.

Demo Results

• The unprotected model is easily tricked into writing Python code and ignoring the system prompt.
• The protected model blocks prompt injection and spam attempts.
• The language processor tags requests with their language.

Conclusion

The video provides a comprehensive overview of AI gateways, explaining their purpose, functionality, and benefits in securing and managing AI traffic. It highlights the importance of understanding AI protocols, protecting against prompt injection and spam, and enforcing desired model behavior. The demo showcases the effectiveness of AI gateways in preventing malicious activity and ensuring responsible AI usage.