Open Source Friday with Guardrails AI

Key Concepts

• Responsible AI: Building and deploying AI systems ethically and reliably.
• Guardrails: Infrastructure to verify assumptions and ensure AI system reliability throughout its lifecycle.
• Hallucinations: Instances where AI models generate incorrect or nonsensical information.
• Jailbreaking: Attempts to bypass AI model safety measures to elicit harmful responses.
• PII (Personally Identifiable Information): Data that can identify an individual, requiring careful handling.
• Simulation Environments: Testing AI systems with simulated users and scenarios to identify vulnerabilities.
• Model Uncertainty: Accounting for the inherent unpredictability of AI models in system design.
• AI Agents: Autonomous systems that perceive their environment and take actions to achieve goals.
• LLM (Large Language Model): A type of AI model trained on vast amounts of text data.
• Validators: Specific checks or rules used within guardrails to verify AI system inputs and outputs.
• Guards: Collections of validators applied to AI system inputs or outputs.
• Snow Globe: A simulation environment for testing AI systems with simulated users.

Introduction to Guardrails AI and Shrea

• Shrea, co-founder and CEO of Guardrails AI, discusses building responsible and ethical AI with Kevin Scott.
• Guardrails AI focuses on AI reliability infrastructure, ensuring AI-enabled software functions with deterministic software-level reliability.
• Shrea's background includes classical AI research, self-driving cars, and AI infrastructure.

The "Aha" Moment and the Shift in AI

• The pivotal moment for Guardrails AI was post-ChatGPT, marking a shift from focusing solely on models to building systems that utilize models as a core component.
• This shift broadened the scope of AI development, moving from data collection and training loops to prompt engineering and integrating AI components.
• The development of AI systems now parallels self-driving cars, emphasizing end-to-end latency and system-level considerations.

Problem Statement and Solution

• Guardrails AI addresses the challenge of ensuring AI system reliability throughout its lifecycle, from development to production.
• The open-source Guardrails framework verifies assumptions made about AI systems in real-time, online settings.
• Explicit verification is crucial in AI systems due to the vast and unpredictable output space, unlike deterministic software.
• Guardrails help prevent issues like invalid JSON generation, hallucinations, and jailbreaking.
• The company has expanded to offer simulation environments for testing system behavior under large-scale user simulations.

Developer Feedback Loops and Best Practices

• Developers are increasingly adopting practices from the AI/ML background, such as extensive testing and defining clear rubrics for success and failure.
• Creating diverse datasets and actively monitoring system performance against these datasets is essential.
• Accounting for model uncertainty is critical for successful AI system adoption, as demonstrated by Copilot's UX.

Applications of Guardrails

• Initially, Guardrails was used for specific applications, but it has evolved to be integrated into centralized AI platforms within organizations.
• Platform teams use Guardrails to prevent PII leaks and ensure adherence to company policies.
• With the rise of AI agents, Guardrails is now being applied at the application level to prevent issues like getting stuck in loops or making unsafe requests.

Using Different Models to Check Hallucinations

• Employing multiple models is a successful pattern for building AI agents, enhancing uptime, reliability, accuracy, cost, and latency.
• For hallucination checking, models can be stacked in a waterfall approach, starting with simple lookup models for numeric hallucinations and progressing to more complex models for contextual or factual hallucinations.
• The choice of models involves a trade-off between power and latency.

Scaling Responsible AI in Large Companies

• Successful responsible AI programs in large organizations have diverse recommendations tailored to specific AI applications.
• Recommendations for internal LLMs differ from those for HR support chatbots or customer-facing systems.
• Guardrails should be context-aware, considering security, product alignment, company brand, and risk.
• Frameworks like NIST AI Risk Management Framework (AI RMF) and OWASP Top 10 for LLMs provide valuable guidance.

Enabling Developers to Design Evaluations

• Guardrails can be daunting, so it's important to help developers understand their system's true vulnerabilities.
• Snow Globe, a simulation environment, allows developers to connect their AI chatbot and simulate user interactions at scale.
• This helps identify risks and failures, enabling developers to add only the necessary guardrails.

AI Development Journey and Guardrails

• For organizations, thinking about guardrails early on is crucial to avoid liability.
• For individual application development, it's best to build the end-to-end system first and then intentionally add only the necessary guardrails.
• Testing in various ways is essential, as not all recommended guardrails may be necessary for every system.
• The Deep Learning AI course series by Andrew Ng includes a course on guardrails.

Contributing to the Open Source Platform

• The Guardrails system is designed to be modular, with orchestration and a hub of guardrails.
• Developers can contribute guardrails for specific use cases to the hub.
• The project also has open issues and a Discord community for discussion.

Roadmap for Guardrails

• Focus on model research, including improving models used for guardrails.
• Launched an index/benchmark of guardrails, creating a dataset for common categories and benchmarking performance.
• Continuing to conduct public benchmarks.

Staying on Top of New Model Releases

• Twitter (now X) is a valuable resource for model releases.
• Curating a list of high-signal individuals who discuss AI research, applications, and model announcements is effective.

Vibe Coding vs. Complex AI Models

• Guardrails are context and domain-dependent.
• For vibe coding, concerns are around unsafe code, exposed secrets, code-level vulnerabilities, and injection attacks.
• For customer service chatbots, concerns are different.
• Consider when to apply guardrails, balancing risks and performance. Applying guardrails at code commit may be better than at every LLM invocation.

Guardrails Demo: Architecture and Usage

• Guardrails verify requests sent to and outputs received from LLMs.
• Checks can include PII, company proprietary information, and jailbreak attempts.
• The open-source Guardrails hub provides a variety of guardrails for different use cases.
• The hub is inspired by the Hugging Face model hub.
• Each guardrail introduces some latency, so it's important to choose only the necessary ones.
• A Jupyter notebook demo shows how to create and use guards.
• Guards are collections of guardrails that run on inputs or outputs.
• The demo includes examples of profanity guards, politeness checks, topic guards, and PII detection.
• Policies can be set up to handle violations, such as logging, raising exceptions, masking data, or fixing requests.

Guardrails Under the Hood

• Guardrails use three classes of implementation: rule-based systems (regex), fine-tuned machine learning models, and LLMs as judges.
• Rule-based systems are deterministic code.
• Fine-tuned ML models are classifiers for PII, hallucinations, and profanity.
• LLMs as judges use prompt tuning.

Snow Globe Demo: Simulation Environment

• Snow Globe simulates users interacting with an AI system to identify vulnerabilities before production.
• It addresses the challenge of testing AI systems with a wide variety of inputs.
• The demo shows simulations on a "life coach GPT" chatbot.
• To run simulations, you need a live connection to the chatbot and a text description of it.
• You can optionally connect a knowledge base and historical data.
• You can simulate general users, specific cohorts, or specific behaviors.
• Snow Globe generates synthetic personas with different traits and use cases.
• The simulation generates conversations between the synthetic users and the chatbot.
• This provides data for fine-tuning models, optimizing prompts, and QA.

Tips and Tricks

• Guardrails is open source and available via pip install.
• Snow Globe is generally available, and you can sign up to start simulations.
• Use Snow Globe to optimize and test your AI system.
• Use Guardrails as a last line of defense in post-production, based on the actual risks identified by Snow Globe.

Community Engagement

• Visit guardrailsai.com for links to Discord.
• Connect with Shrea on LinkedIn or Twitter.
• Email [email protected].

Conclusion

Guardrails AI provides essential tools and frameworks for building responsible and reliable AI systems. By focusing on verification, simulation, and community engagement, Guardrails AI empowers developers to create AI applications that are both innovative and trustworthy.