F5 AI Guardrails to Secure Model Inference with Red Hat OpenShift AI

Key Concepts

• AI Guardrails: Security mechanisms that monitor and filter inputs/outputs at the inference layer.
• Inference Layer: The stage where a trained AI model processes data to generate predictions or responses.
• Prompt Injection: A security vulnerability where malicious inputs are used to manipulate an AI model into bypassing its safety protocols.
• PII (Personally Identifiable Information): Sensitive data (e.g., Social Security numbers, credit card numbers) that must be protected from leakage.
• Kubernetes Operator: A method of packaging, deploying, and managing a Kubernetes application.
• Red Hat OpenShift AI: The platform providing the infrastructure for model serving and AI workloads.

---

1. Challenges in Enterprise AI Deployment

Enterprises scaling AI face three primary security hurdles:

• Limited Visibility: Security teams lack real-time insight into the content and intent of AI interactions.
• Growing Threat Vectors: Traditional security tools are insufficient against modern AI-specific threats like jailbreaks, prompt injections, and adversarial attacks.
• Uncontrolled AI Interactions: Without enforcement at the inference layer, organizations risk data leaks, intellectual property (IP) loss, and regulatory non-compliance.

2. F5 AI Guardrails Architecture

F5 AI Guardrails acts as an "in-line" security layer. Every prompt from a user/agent and every response from the model must pass through the guardrail engine before reaching its destination.

• Deployment: Deployed as a Kubernetes-native operator via the Operator Lifecycle Manager on Red Hat OpenShift.
• Infrastructure: The demo utilizes three bare-metal nodes, each with an A40 GPU. Workloads are distributed: one GPU for the Mistral Red Team model, one for the Phi-4 (54) guardrail scanner, and one for the Llama model.

3. Guardrail Methodologies and Types

The system provides a multi-layered defense strategy:

Pre-built Guardrail Packages

• Functionality: Out-of-the-box scanners for common threats.
• Capabilities: Detection of prompt injections, PII, and harmful content (e.g., phishing email generation).
• Enforcement: Operates in "block mode," where any detected threat stops the request immediately before it reaches the model.

Custom Guardrails

Organizations can define specific policies tailored to their internal requirements:

1. GenAI-based: Uses a scanning model (e.g., Phi-4) to detect semantic concepts, such as "internal financial forecasts" or "revenue projections."
2. Keyword Match: Targets specific sensitive terms, such as project code names (e.g., "Project Phoenix").
3. Regex (Regular Expression): Enforces pattern-based rules for structured data, such as employee ID formats (e.g., EMP-######).

4. Step-by-Step Implementation Process

1. Deployment: Install the F5 AI Guardrails operator via the OpenShift software catalog.
2. Configuration: Enable pre-built packages and define custom policies in the moderator UI.
3. Validation: Use the built-in "Playground" to test custom guardrails against sample prompts before pushing them to production.
4. Deployment/Assignment: Assign active guardrails to specific projects.
5. Monitoring/Auditing: Utilize the audit trail to review blocked entries, risk scores, and the specific guardrails that triggered the block.

5. Compliance and Auditability

The system maintains a comprehensive audit trail for every interaction. This includes:

• Decision Logging: Detailed logs showing which guardrails fired and why.
• Risk Scoring: Quantitative assessment of threats.
• Full Message Capture: Retention of the original prompt and response for forensic analysis and policy tuning.

6. Synthesis and Conclusion

F5 AI Guardrails provides a robust, Kubernetes-native security framework that secures the AI inference layer. By combining pre-built threat detection with customizable, context-aware policies, it enables organizations to mitigate risks like prompt injection and data leakage. The integration with Red Hat OpenShift AI ensures that security is not an afterthought but an embedded component of the AI stack, providing the visibility and control necessary for enterprise-grade AI governance.