Advancing AI Inference: Scalability and Security with Red Hat OpenShift AI and F5

Key Concepts

• AI Inference: The process of using a trained AI model to make predictions or decisions on new data.
• OpenShift AI: Red Hat's platform for developing, deploying, and managing AI/ML models.
• API Security: Protecting Application Programming Interfaces (APIs) from unauthorized access and attacks.
• LLM (Large Language Model): A type of AI model trained on vast amounts of text data, capable of generating human-quality text.
• Prompt Injection: A type of attack where malicious input is crafted to manipulate the behavior of an LLM.
• AI Gateway: A security solution specifically designed to protect LLMs from attacks like prompt injection and data leakage.
• RAG (Retrieval Augmented Generation): An AI architecture that combines a language model with an information retrieval system to improve the accuracy and relevance of generated text.
• F5 Distributed Cloud: F5's platform for delivering and securing applications across multiple environments.
• VLMs (Very Large Models): Open-source way for the inference.
• LLMD: Open source project that is like the Kubernetes of AI.

AI Inference: Scaling and Securing

The presentation focuses on the industry trend of AI inference and how to scale and secure it using a joint solution from Red Hat (OpenShift AI) and F5.

Red Hat's AI Perspective

• General Trends: AI is augmenting the workforce, driving productivity and efficiency. Use cases include revenue generation, cost reduction, logistics optimization, and risk management.
• Challenges in AI Adoption:
  • Complexity: Understanding AI capabilities and identifying appropriate use cases.
  • Flexibility: Deploying AI models across diverse environments (edge, cloud, etc.) and avoiding vendor lock-in.
  • Cost: The high cost of infrastructure (GPUs) and model training.
  • Data Governance: Balancing data privacy (GDPR, etc.) with the need for data to train AI models.
  • Security Gaps: Traditional security tools are insufficient for AI/LLM security.
  • Visibility: Lack of end-to-end visibility into AI model training and inference.
• Red Hat's Approach with OpenShift AI:
  • Advocates for smaller, purpose-built models tuned with enterprise-relevant data.
  • Provides a platform (OpenShift AI) that supports any model, including custom-built ones.
  • Enables deployment of AI models where the data resides.
  • Offers lifecycle management and monitoring at scale.
  • Maximizes technology investment by providing flexibility and choice.
• Red Hat AI Portfolio:
  • Real AI: Access to test out what AI is about. Access to instruct lab with granite models that can be used to experiment and see how applicable it is.
  • OpenShift AI: Platform where data scientists can run models in any environment. Any model, any accelerator on any cloud.
  • Red Hat AI inference server: Augmented inference.

F5's Security Solutions for AI

• AI as a Double-Edged Sword: AI can be used for innovation but also for malicious purposes.
• Real-World Examples of AI Security Breaches:
  • Prompt injection attacks leading to system takeover.
  • Sensitive information shared with AI chatbots.
  • Bypassing security coding practices with AI-generated code.
• AI Architecture and API Security:
  • AI inference involves multiple components communicating through APIs.
  • Securing APIs is crucial for overall AI security.
  • Traditional API security is necessary but not sufficient for AI/LLM security.
  • AI/LLM attacks have unique attack surfaces and vectors.
• F5's Multi-Layered Security Approach:
  1. API Security: Using F5 Distributed Cloud Web Application and API Security (WAAP) to protect APIs.
     • Three Pillars:
       • Discovery: Identifying all API endpoints, including shadow and undocumented APIs.
       • Monitoring: Analyzing live API traffic using AI/ML to detect anomalies.
       • Enforcement: Enforcing security policies and compliance.
     • Integration with OpenShift AI: Routing API traffic through F5 Distributed Cloud for security enforcement.
     • Use Cases:
       • API Discovery and Schema Validation.
       • OpenAPI Validation for Request and Response Inspection.
       • Bot Defense and Rate Limiting.
       • Sensitive Data Detection and Redaction.
       • Single Dashboard for Metrics and Anomaly Detection.
  2. AI Gateway: A new product specifically designed to secure LLMs from attacks.
     • Addressing OWASP LLM Top 10 Attacks: Prompt injection, system prompt attacks, data leaks, etc.
     • Features:
       • Prompt Injection Detection.
       • Smart Routing based on language ID.
       • Repetition Detection.
       • Model Routing.
     • Examples:
       • Smart Routing: Directing requests to different models based on language.
       • Prompt Injection Prevention: Blocking malicious prompts designed to extract sensitive information.
       • System Prompt Leakage Prevention: Preventing the model from revealing internal instructions and logic.
  3. AI Rack with F5 Distributed Cloud and OpenShift AI:
     • Scenario: Distributed AI services with end-users accessing RAG services, AI inference running in the cloud (OpenShift AI), and on-prem data storage.
     • F5's Role:
       • Network Connectivity: Building secure Layer 3 connectivity between RAG services and on-prem data storage.
       • Service Mesh: Providing application-level service mesh across clouds to connect RAG services with inference endpoints.

Joint Solution Benefits

• Simplified and Secure AI Deployment: Combining Red Hat's simplification of AI adoption with F5's security solutions.
• Adopting and Deploying AI at Scale with Confidence: Ensuring end-to-end observability and security in any environment.
• Key Takeaway: It's not enough to be simple with AI usage. It's not enough to be just secure. You need both.

Conclusion

The presentation highlights the importance of both simplifying and securing AI inference. Red Hat's OpenShift AI provides a flexible and consistent platform for deploying AI models, while F5's security solutions protect those models from a range of attacks. The joint solution enables customers to adopt and deploy AI at scale with confidence, ensuring both ease of use and robust security.