Prompt Engineering and AI Red Teaming — Sander Schulhoff, HackAPrompt/LearnPrompting

Prompt Engineering and AI Red Teaming Summary

Key Concepts:

• Prompt Engineering: Improving prompts to enhance AI accuracy and performance.
• AI Red Teaming: Identifying vulnerabilities in AI systems by attempting to make them perform undesirable actions.
• Prompt Injection: Exploiting vulnerabilities by manipulating prompts to override developer instructions.
• Jailbreaking: Tricking AI models into bypassing safety filters and generating harmful content.
• Chain of Thought Prompting: Encouraging AI to explicitly show its reasoning steps.
• In-Context Learning: Providing examples within a prompt to guide AI behavior.
• Adversarial Robustness: The ability of AI systems to withstand malicious inputs and attacks.
• Attack Success Rate (ASR): A metric used in red teaming to measure the effectiveness of attacks.

1. Introduction

• Sandra Fulof, CEO of Learn Prompting and Hackaprompt, discusses prompt engineering and AI red teaming.
• She wrote the first guide on prompt engineering and organized the first competition on prompt injection.
• Key takeaways:
  • Prompt engineering is still relevant.
  • Security concerns hinder the deployment of prompted systems.
  • Securing GenAI is difficult, potentially an "impossible problem."

2. Background

• Diplomacy AI: Early work involved deception research, which became relevant with advanced AI.
• MinRL (Minecraft Reinforcement Learning): Training AI agents in Minecraft, combining linguistic and RL elements.
• Learn Prompting: Created the first open-source guide on prompt engineering, cited by major AI companies and governments.
• Hackaprompt: Organized the first prompt injection competition, open-sourcing a dataset of 600,000 prompts used for AI model benchmarking.

3. Fundamentals of Prompt Engineering

• Definition:
  • A prompt is a message sent to a generative AI.
  • Prompt engineering is the process of improving prompts.
• Importance:
  • Improved prompts can boost accuracy by up to 90%.
  • Poor prompts can reduce accuracy to 0%.
  • Prompts are core components of multi-prompt systems and agents.
• History:
  • The concept of prompting existed before the term.
  • The term "prompting" gained traction around the time of GPT-2.
  • "Prompt engineering" emerged around 2021.
• Types of Users:
  • Non-technical users: Conversational prompt engineering (e.g., using chatbots).
  • Technical users: Traditional prompt engineering for specific tasks (e.g., binary classification).

4. Advanced Prompt Engineering: The Prompt Report

• Overview:
  • Comprehensive systematic literature review of prompting techniques.
  • Led by Sandra Fulof with a team of 30 researchers.
  • Covered approximately 200 prompting techniques, including 58 text-based English-only techniques.
• Core Contributions:
  • Taxonomized the different parts of a prompt (e.g., role, examples).
  • Taxonomized hundreds of prompting techniques.
  • Conducted manual and automated benchmarks.
• Role Prompting:
  • Telling the AI to assume a specific role (e.g., "You are a math professor").
  • Historically believed to improve performance, but now considered largely useless for tasks requiring empirical validation.
  • Case study: A "dumb idiot" role outperformed an "intelligent math professor" role on math problems.
  • Still useful for open-ended tasks like writing or summaries.
• Core Classes of Techniques:
  • Thought Inducement:
    • Chain of Thought Prompting: Getting the AI to write out its reasoning steps before providing the final answer.
    • Effective for accuracy-based tasks.
    • Reasoning models (e.g., 01, 03) were inspired by chain of thought.
    • The model's reasoning may not be truthful.
    • Thread of Thought Prompting: Variants of "let's go step by step."
    • Tabular Chain of Thought: Outputting the chain of thought as a table.
  • Decomposition-Based Techniques:
    • Breaking down a problem into subproblems before attempting to solve it.
    • Least-to-Most Prompting: Asking what subproblems must be solved first.
  • Ensembling:
    • Using multiple experts (separate LLMs or instances) with different roles or tool-calling abilities.
    • Taking the most common answer as the final response.
    • Techniques like self-consistency (asking the same prompt repeatedly) are becoming less used.
    • Mixture of Reasoning Experts: A technique where separate LLMs are given different role prompts or tool-calling abilities, and the most common answer is taken as the final response.
  • In-Context Learning (ICL):
    • Providing examples within the prompt to guide AI behavior.
    • Differentiated from few-shot prompting, which specifically refers to giving examples.
    • Models learn from the context of the prompt, even without explicit examples.
    • Few-Shot Prompting: Giving the model examples of what you want it to do.
      • Include as many examples as possible.
      • Exemplar ordering can significantly impact accuracy.
      • Maintain a balanced label distribution.
      • Ensure high label quality.
      • Use a good format for examples (e.g., "input: output").
      • Select examples similar to the task at hand.
  • Self-Evaluation:
    • Having the model output an initial answer, provide self-feedback, and refine its answer.

5. Experiments and Benchmarks

• Compared different prompting techniques on MMLU.
• Found that few-shot and chain of thought combined were the best techniques.
• Study on detecting entrapment (a precursor to suicidal intent) in social media posts.
• Manual prompt engineering was outperformed by automated prompt engineering (DSP).
• The presence of specific names in the prompt, even anonymized, significantly impacted performance.

6. AI Red Teaming

• Definition: Getting AIs to do and say bad things.
• Jailbreaking: Tricking chatbots into bypassing safety filters and generating harmful content.
• Prompt Injection: Exploiting vulnerabilities by manipulating prompts to override developer instructions.
• Distinction between Jailbreaking and Prompt Injection:
  • Jailbreaking: Tricking a model without developer instructions.
  • Prompt Injection: Exploiting a system with developer instructions.
• Real-World Harms (with caveats):
  • Chevy Tahoe for $1: A chatbot was tricked into selling a car for $1.
  • Freda: An AI crypto chatbot that paid users who could trick it.
  • Math GPT: A math-solving application that was exploited to leak keys due to malicious Python code execution.
• Cyber Security vs. AI Security:
  • Cyber security is more binary (protected or not).
  • AI security is probabilistic and difficult to guarantee.
  • SQL injection is solvable by escaping user input, while prompt injection has no strong guarantees.

7. Philosophies of Jailbreaking

• Intractability (Jailbreak Persistence Hypothesis): You can patch a bug in cyber security, but you can't patch a brain in AI security.
• Non-Determinism: The same prompt can produce different responses each time, making measurement difficult.
• Ease of Jailbreaking: New AI models are often jailbroken immediately after release.

8. Hacker Prompt Competition

• First competition on AI red teaming and prompt injection.
• Open-sourced a large dataset used for benchmarking.
• Key takeaways:
  • Defenses like improving prompts or adding guardrails don't work effectively.
  • Automated red teaming tools are effective.
• Taxonomy of Attack Techniques:
  • Obfuscation: Techniques like base64 encoding or translating to low-resource languages to evade filters.
  • Typos: Intentionally introducing typos to bypass filters.

9. Monologue on Agents

• Agents are unlikely to work effectively without solving adversarial robustness.
• Powerful agents operating in the real world are vulnerable to manipulation.
• Examples:
  • A humanoid robot being tricked into throwing eggs at someone.
  • A web-using agent being tricked by malicious ads.
• Data is key to improving adversarial robustness.

10. Conclusion

• Prompt engineering and AI red teaming are critical areas in AI development.
• Securing AI systems, especially agents, is a significant challenge.
• Data-driven approaches and ongoing research are essential for improving adversarial robustness.