How to Build Trustworthy AI — Allie Howe

Key Concepts

Trustworthy AI, AI Security, AI Safety, ML SecOps, AI Red Teaming, AI Runtime Security, Prompt Injection, Jailbreak, Model Serialization Attacks, Shift Right, Compliance (SOC 2, ISO42001, EU AI Act, HIPAA, FDA guidelines).

What is Trustworthy AI and Why Should You Care?

The presentation addresses the critical need for trustworthy AI, highlighting real-world incidents where AI systems failed, leading to negative consequences. Examples include:

• Chevy Tahoe Chatbot Incident (2023): A chatbot offered a Chevy Tahoe for $1, demonstrating a failure in intended operation and vulnerability to user manipulation.
• Slack Data Leak (2024): Slack was tricked into leaking data from private channels via a prompt injection, resulting in data leakage and potential reputational damage.
• Fortnite's AI Vader: An AI-powered Darth Vader NPC in Fortnite initially exhibited racist and homophobic behavior, showcasing the potential for AI to generate inappropriate content.

The speaker emphasizes that you are responsible for the actions of your AI applications. A recent lawsuit against OpenAI was dismissed because the responsibility falls on the user to understand the limitations of AI and proceed with caution. This responsibility extends to both legal and reputational aspects.

Quote: "If you're using AI, it's likely you that will be responsible both on paper and also just from a brand and reputational standpoint."

The Recipe for Trustworthy AI: AI Security and AI Safety

Trustworthy AI is defined as the combination of AI security and AI safety:

• AI Security: How the outside world can harm your AI application (e.g., prompt injections, model serialization attacks).
• AI Safety: How your AI application can harm the world (e.g., generating harmful content, biased outputs).

Product, engineering, and security teams all play a role in building trustworthy AI. Product teams focus on relevance and helpfulness, security teams on preventing inappropriate content and attacks, and engineering teams on cost and latency.

From DevSecOps to ML SecOps: A New Paradigm

The presentation introduces a shift from traditional DevSecOps to ML SecOps to address the unique challenges of AI engineering.

• DevSecOps Limitations: Traditional security tools integrated into CI/CD pipelines are insufficient for AI, as data scientists and ML engineers work in environments like Databricks and Jupyter notebooks.
• ML SecOps: Machine Learning Security Operations is a new model that considers the specific needs of AI development, including scanning Databricks and Jupyter notebooks for exposed secrets and understanding model provenance (origin, data, training).

Model Serialization Attacks: A significant risk highlighted is model serialization attacks, where malicious code is embedded in a model during serialization and executed upon deserialization. The pickle format is particularly vulnerable.

Example: The presentation demonstrates a model serialization attack using Model Scan from Protect AI. The attack involves embedding code to output AWS secrets into a serialized model. When the model is deserialized, the code executes, leaking the credentials. Model Scan can detect such vulnerabilities by identifying unsafe operators.

Step-by-step process for using Model Scan:

1. pip install modelscan
2. Run Model Scan on the model file.

Quote: "ML SecOps is able to look into and take into consideration places that traditional DevSecOps does not."

AI Red Teaming: Testing for Security and Safety

AI red teaming involves simulating adversarial threats and AI safety concerns to identify vulnerabilities.

• Testing Scenarios: Red teaming can test for prompt injections, jailbreaks, and AI safety issues like generating instructions for building bombs or exhibiting biased behavior.
• Benefits:
  • Influencing runtime guardrails by identifying vulnerabilities and blocking prompts that elicit undesirable responses.
  • Comparing LLMs to detect potential backdoors or deceptive behavior.

AI Runtime Security: The Importance of "Shift Right"

AI runtime security is emphasized as a crucial component of trustworthy AI, particularly due to the non-deterministic nature of AI systems.

• Shift Right Focus: While "shift left" (addressing security early in the development cycle) remains important, runtime security is essential for validating AI inputs and outputs in real-time.
• Attack Vectors: Runtime security addresses direct prompt injections, indirect prompt injections (e.g., through RAG systems scraping websites with hidden prompts), and jailbreaks.

Example: Fortnite AI Vader Architecture

The presentation provides a hypothetical architecture for the Fortnite AI Vader implementation, highlighting potential points for inserting AI runtime security:

1. After User Audio Input: Validate user input for inappropriate content before sending it to 11 Labs for voice-to-text conversion.
2. After Gemini Response: Validate Vader's text response for off-topic or inappropriate content before sending it to 11 Labs for text-to-speech conversion.
3. After 11 Labs Audio Output: Validate the final audio output before sending it back to the user.

Considerations for AI Runtime Security Solutions:

• Cost: Acceptable cost of API calls or processing.
• Latency: Low latency to avoid impacting user experience.
• Accuracy: High accuracy in detecting threats and inappropriate content.

Example: Pillar AI Runtime Security

The presentation showcases Pillar, an application security lifecycle platform, as an example of AI runtime security in action.

• Use Case: An application for finding patients suitable for ALS clinical trials.
• Guardrail: A custom guardrail is configured to prevent the system from modifying patient data in the database.
• Demonstration: When the user attempts to change a patient's FC percentage, Pillar blocks the request, demonstrating the effectiveness of custom guardrails.

Benefits of AI Runtime Security:

• Detecting and preventing PII leaks.
• Mitigating AI safety concerns and toxic responses.
• Enforcing custom guardrails aligned with business goals.

Compliance and Competitive Advantage

The presentation emphasizes the importance of demonstrating trustworthy AI practices for compliance and competitive advantage.

• Compliance Frameworks: SOC 2, ISO42001, EU AI Act, HIPAA, and FDA guidelines are mentioned as relevant compliance frameworks.
• Trust Centers: Showcasing AI security controls in a trust center can build customer trust and streamline sales cycles.

Example: A custom risk in a risk register addresses potential reputational damage due to harmful AI outputs. A corresponding control validates AI outputs, and evidence of this control is shared in a trust center.

Why Build Trustworthy AI?

The presentation concludes by reiterating the importance of building trustworthy AI:

• Alignment of Cybersecurity and Business Risk: AI security is directly linked to business success, as untrustworthy AI can damage reputation and revenue.
• Magnification of Existing Risks: AI can amplify existing cybersecurity vulnerabilities.
• Increasing Regulatory Scrutiny: Compliance with AI-related regulations is becoming increasingly important.
• Unlocking Innovation: Trustworthy AI is essential for realizing the full potential of AI in fields like healthcare.

Quote: "Trustworthy AI is super important because it's going to unlock a lot of revolutionary innovation."

Conclusion

The presentation provides a comprehensive overview of trustworthy AI, emphasizing the shared responsibility, the importance of AI security and safety, and the need for a holistic approach encompassing ML SecOps, AI red teaming, and AI runtime security. By prioritizing trustworthy AI, organizations can mitigate risks, ensure compliance, build customer trust, and unlock the transformative potential of AI.