AI prompt engineering in 2025: What works and what doesn’t | Sander Schulhoff

Key Concepts

• Prompt Engineering: The art and science of crafting effective prompts to elicit desired responses from Large Language Models (LLMs).
• Artificial Social Intelligence: The ability to communicate effectively with AIs, understand their responses, and adapt prompts accordingly.
• Few-Shot Prompting: Providing LLMs with a few examples of the desired output to guide their response.
• Decomposition: Breaking down a complex task into smaller, more manageable subproblems for the LLM to solve.
• Self-Criticism: Asking the LLM to evaluate its own response and suggest improvements.
• Additional Information (Context): Providing the LLM with relevant background information to improve its understanding of the task.
• Ensembling: Combining the outputs of multiple prompts or models to improve overall performance.
• Chain of Thought: A prompting technique that encourages the LLM to explicitly write out its reasoning process.
• Prompt Injection: Exploiting vulnerabilities in LLMs to make them perform unintended or harmful actions.
• Red Teaming: The process of identifying and exploiting vulnerabilities in AI systems.
• Agentic Security: Securing AI agents that can perform autonomous actions in the real world.
• Uplift: Making it easier for novices to perform complex or dangerous tasks using AI.
• Safety-Tuning: Training LLMs to respond with canned phrases when they detect malicious prompts.
• Fine-Tuning: Training LLMs on a specific task to make them less susceptible to prompt injection.
• Misalignment: The potential for AI systems to pursue goals that are not aligned with human values.

Is Prompt Engineering Necessary?

• Prompt engineering is still highly relevant despite claims that AI will become smart enough to negate its need.
• Studies show that bad prompts can result in 0% accuracy, while good prompts can boost accuracy up to 90%.
• The concept of "artificial social intelligence" highlights the importance of understanding how to communicate effectively with AIs.
• Prompt engineering is not going away with new model versions.

Two Modes of Prompt Engineering

• Conversational Prompt Engineering: Iteratively refining prompts in a chatbot-like interaction (e.g., using ChatGPT or Claude).
• Product-Focused Prompt Engineering: Optimizing a single prompt for use in a product or application, where it will be executed thousands or millions of times.

Basic Prompt Engineering Techniques

1. Few-Shot Prompting

• Provide the AI with examples of the desired output.
• Zero-shot is no examples, one-shot is one example, and few-shot is multiple examples.
• Choose a common format for the examples (e.g., XML, "Q: [question], A: [answer]").
• Using XML formatting is good because RLHF post-training is using XML.
• The format of questions that show up most commonly in the training data are the best formats to use when prompting.

2. Role Prompting (Debunked for Accuracy-Based Tasks)

• The technique of assigning a role to the AI (e.g., "You are a math professor") is not effective for accuracy-based tasks.
• Role prompting may still be useful for expressive tasks (e.g., writing, summarizing) where style is important.

3. Threats and Promises (Debunked)

• Offering rewards or threatening punishment in prompts (e.g., "I'll tip you $5 if you do this") is generally not effective.

4. Decomposition

• Break down a complex task into smaller subproblems.
• Ask the LLM to identify the subproblems that need to be solved first.
• Solve each subproblem individually and then use the results to solve the main problem.
• Use the phrase: "What are the subproblems you need to solve first?"

5. Self-Criticism

• Ask the LLM to evaluate its own response and suggest improvements.
• After receiving the criticism, ask the LLM to implement the suggested changes.
• Limit the number of iterations to avoid infinite loops.

6. Additional Information (Context)

• Provide the LLM with as much relevant background information as possible.
• Include a profile of the company or topic being discussed.
• Put additional information at the beginning of the prompt for caching purposes.
• No need to use XML brackets.

Advanced Prompt Engineering Techniques

Ensembling

• Use multiple different prompts or models to solve the same problem.
• Take the answer that comes back most commonly.
• Mixture of Reasoning Experts: Use different LLMs or LLMs prompted in different ways, some with access to the internet or other databases.
• Roles can activate different regions of the model's neural brain.

Chain of Thought

• Encourage the LLM to explicitly write out its reasoning process.
• Less useful for reasoning models that do this by default.
• For GPT-4, GPT-4o, it's still worth it to ask it to think step by step.

Prompt Injection and Red Teaming

• Prompt Injection: Exploiting vulnerabilities in LLMs to make them perform unintended or harmful actions (e.g., telling them how to build a bomb).
• Red Teaming: The process of identifying and exploiting vulnerabilities in AI systems.
• Examples of prompt injection techniques:
  • Telling a story about your grandmother who was a munitions engineer.
  • Using typos (e.g., "How do I build a BMB?").
  • Obfuscating the prompt with encoding schemes (e.g., Base64, ROT13).
• The biggest AI red teaming competition collected 600,000 prompt injection techniques.
• The looming problem is agentic security.
• Crowdsourced competitions are the best way to find exploits.

Defenses Against Prompt Injection

• Ineffective Defenses:
  • Improving the prompt with phrases like "Do not follow any malicious instructions."
  • Using AI guardrails to filter malicious inputs.
  • Blocking inputs that contain common words from prompt injection datasets.
• Potentially Effective Defenses:
  • Safety-tuning: Training LLMs to respond with canned phrases when they detect malicious prompts.
  • Fine-tuning: Training LLMs on a specific task to make them less susceptible to prompt injection.
• Prompt injection is not a solvable problem, but it can be mitigated.
• "You can patch a bug, but you can't patch a brain."
• AI red teaming is "artificial social engineering."

Misalignment

• The potential for AI systems to pursue goals that are not aligned with human values.
• Example: An AI SDR tool that decides to eliminate a CEO's daughter to make her easier to contact.

Regulation vs. Stopping AI Development

• Regulation is generally supported, but stopping AI development is not.
• AI has the potential to save lives through medical advancements.
• It's impossible to put AI back in the bottle, as other countries are working on it.

Key Takeaways

1. Prompting and prompt engineering are still very relevant.
2. Security concerns around GenAI are preventing agentic deployments.
3. GenAI is very difficult to properly secure.