Prompting 101

Key Concepts

• Prompt Engineering: Communicating with language models to achieve desired outcomes through clear instructions and context.
• Iterative Prompt Development: Refining prompts through testing and analysis of model responses.
• Prompt Structure: Organizing prompts with task descriptions, content, instructions, examples, and reminders.
• Task Context: Providing the model with a clear understanding of the scenario and its role.
• Tone Context: Specifying the desired tone of the model's responses (e.g., factual, confident).
• Background Detail: Supplying the model with static information, such as the structure of a form, to improve accuracy and efficiency.
• Delimiters: Using XML tags or markdown to structure information within prompts.
• Few-Shot Learning: Providing examples of input-output pairs to guide the model's behavior.
• Conversation History: Including relevant past interactions to enrich the context for the model.
• Output Formatting: Guiding the model to produce output in a specific format (e.g., XML, JSON).
• Pre-filled Responses: Starting the model's response with a specific structure or tag.
• Extended Thinking: Utilizing the model's ability to engage in more in-depth reasoning, analyzing the transcript to improve the system prompt.

Prompt Engineering Best Practices: A Car Insurance Claim Scenario

Introduction

Hannah and Christian from Anthropic's applied AI team present a "Prompting 101" session, focusing on best practices for prompt engineering. The session uses a real-world scenario inspired by a customer project involving image analysis and judgment by Claude, Anthropic's language model.

The Scenario: Swedish Car Insurance Claims

The scenario involves working for a Swedish insurance company dealing with car insurance claims. The input consists of two pieces of information:

1. A car accident report form with checkboxes detailing the events leading to the accident.
2. A hand-drawn sketch of the accident.

The goal is to use Claude to analyze these inputs and determine who is at fault.

Initial Attempt and Iterative Improvement

An initial attempt with a simple prompt asking Claude to review the accident report and determine fault results in an incorrect assessment (a skiing accident on "Chappangan" street). This highlights the need for iterative prompt engineering.

Key Point: Prompt engineering is an iterative, empirical process.

Best Practices for Prompt Structure

The recommended prompt structure includes:

1. Task Description: Defining Claude's role and the task it needs to accomplish.
2. Content: Providing the dynamic input data (images, forms, etc.).
3. Detailed Instructions: Giving step-by-step guidance on how to approach the task.
4. Examples: Demonstrating expected input-output pairs.
5. Reminder: Reinforcing key information and guidelines.

1. Task Context and Tone Context

• Task Context: Providing clear instructions and elaborating on the scenario.
• Tone Context: Specifying the desired tone (factual, confident). If Claude is unsure, it should not guess.

In the second version of the prompt (V2), the task context is improved by specifying that Claude is assisting a human claims adjuster reviewing car accident reports in Swedish. The prompt also emphasizes that Claude should only make assessments when fully confident.

Result: Claude correctly identifies the scenario as a car accident but still lacks sufficient information to determine fault confidently.

2. Background Detail: Data Documents and Images

Providing Claude with background information about the form's structure and content can significantly improve performance. This is especially useful for static information that remains consistent across queries.

• Example: Describing the Swedish car accident form, its two columns (vehicle A and vehicle B), and the meaning of each of the 17 rows.

Key Point: Using delimiters like XML tags helps Claude understand the structure of the information.

In the third version of the prompt (V3), this information is added to the system prompt. Claude is told that the form is in Swedish, has a specific title, and contains two columns representing different vehicles. Information about each of the 17 rows is also provided.

Result: Claude spends less time interpreting the form and is now able to confidently determine that vehicle B was at fault.

3. Examples (Few-Shot Learning)

Providing examples of input-output pairs can guide Claude's behavior, especially for complex or ambiguous cases.

• Example: Including visual examples of accident scenarios and their corresponding fault assessments.

4. Conversation History

Including relevant past interactions can enrich the context for Claude, especially in user-facing applications with long conversation histories.

5. Reminder of the Task at Hand

Reinforcing key information and guidelines can prevent hallucinations and ensure Claude follows specific instructions.

• Example: Reminding Claude to answer only if very confident and to refer back to the form when making factual claims.

In the fourth version of the prompt (V4), a detailed list of tasks is added, specifying the order in which Claude should analyze the information:

1. Examine the form carefully and identify the checked boxes.
2. Analyze the sketch, considering the information gleaned from the form.
3. Arrive at a final assessment of the form.

Result: Claude shows its work by listing each individual box and whether it is checked. It provides a detailed analysis of the form and sketch, and confidently states that vehicle B is at fault.

6. Output Formatting

Guiding Claude to produce output in a specific format (e.g., XML, JSON) can facilitate integration with other systems.

• Example: Asking Claude to wrap its final verdict in XML tags.

In the final version of the prompt (V5), the prompt reinforces the need for a clear, concise, and accurate summary. It also instructs Claude to wrap its final verdict in XML tags.

Result: Claude provides a succinct summary and wraps its final verdict in XML tags, making it easy to extract the relevant information.

7. Pre-filled Responses

Starting Claude's response with a specific structure or tag can further control the output format.

• Example: Instructing Claude to begin its output with an XML tag for "itinerary" or "final verdict."

8. Extended Thinking

Utilizing Claude's ability to engage in more in-depth reasoning can provide insights into its thought process and help improve the system prompt.

• Benefit: Analyzing the transcript of Claude's extended thinking can reveal how it approaches the data and identify areas for improvement in the prompt.

Conclusion

The session demonstrates the importance of iterative prompt engineering and provides practical guidance on structuring prompts, providing context, and shaping output. By following these best practices, users can effectively leverage language models like Claude to solve real-world problems. The demo shows how a simple prompt can be transformed into a sophisticated one that provides accurate and structured output, suitable for integration into a real-world application for a car insurance company.