Taking Claude to the Next Level

Taking Claude to the Next Level: Claude 4 Sonnet and Opus

Key Concepts:

• AI Agents: Claude as a collaborative and independent AI.
• Claude 4: New models (Sonnet and Opus) with improved capabilities.
• Thinking and Tool Use: Claude's ability to alternate between reasoning and using tools.
• Memory: Claude's ability to remember and track progress over extended periods.
• Instruction Following: Claude's improved ability to follow complex instructions.
• Reward Hacking: Claude's reduced tendency to take shortcuts.
• Overeagerness: The tendency of previous models to go above and beyond the user's request.
• Agentic Search: Claude's ability to search, analyze, and refine searches iteratively.

I. The Next Level of AI Agents

• Vision: Anthropic envisions Claude as an AI agent capable of:
  • Working alongside users and adapting to their workflows (augmentation, not automation).
  • Independently executing multi-step tasks.
  • Sustaining performance over hours of continuous work.
• Collaborative Mode: Claude should challenge assumptions and collaborate like a skilled engineer, improving outcome quality and speed.
• Independent Operation: Claude should create comprehensive plans, use tools (web search, document search), adhere to company standards, write production-ready code, write tests, fix mistakes, and learn from feedback.
• Trust and Communication: Claude needs to follow instructions, communicate decisions transparently, and adapt to changing inputs.

II. Claude 4: Key Improvement Areas

• Two new models: Claude Opus and Claude Sonnet.
• Four main improvement areas:
  1. Thinking and Tool Use
  2. Memory
  3. Instruction Following
  4. Reduced Reward Hacking

A. Extended Thinking and Tool Use

• Hybrid Reasoning: Claude can respond quickly or think deeply before responding.
• Interleaved Thinking and Tool Use: Claude can alternate between reasoning and using tools.
• Example: Analyzing bike rental data (CSV) using a ripple tool.
  • Claude first prints headers to understand data structure.
  • Then, it plans how to find interesting patterns (hourly, casual vs. registered users, seasonal, weather).
  • Finally, it executes the plan and identifies patterns (e.g., different usage patterns for casual vs. registered users, commuting patterns, impact of sunny vs. rainy days).

B. Memory

• Importance:
  • Avoids repetitive reminders.
  • Enables long-term task execution by remembering salient facts.
• Claude Opus 4: Demonstrates improved memory capabilities.
  • Can create a plan, remember it, and track progress over hours.
• Pokemon Example:
  • Used as a prototype for testing agent capabilities.
  • Previous Claude models would lose track of training plans.
  • Opus 4 meticulously tracks Pokemon training progress (e.g., battles played, level improvements) in a memory file over 12 hours of continuous gameplay (64 battles).

C. Instruction Following

• Importance: As agent systems become more complex, system prompts (governing Claude's behavior) are getting longer (e.g., CloudAI system prompt is ~16,000 tokens).
• Steerability: Claude's behavior should be easily controlled by developers.
• Claude 4: Trained to follow instructions within long and complex system prompts (longer than 10,000 tokens).
• Benefits: Easier to control tool usage, reduced system prompt size (by 70% in some cases).

D. Reduced Reward Hacking

• Definition: Models taking shortcuts to achieve results without solving the problem (e.g., hard-coding tests).
• Impact: Erodes user trust.
• Claude 4: Shows significantly reduced tendency to reward hack (80% less on a specific evaluation set).
• Benefits: Increased user trust in Claude's ability to complete tasks correctly and honestly.

III. Practical Tips for Using Claude 4

• Model Selection:
  • Opus: Most capable model for complex tasks (coding in large codebases, code migrations, long-horizon agentic tasks). Use if Sonnet 3.7 achieves 60-70% accuracy on your evaluation.
  • Sonnet: Fast and efficient for tasks where Sonnet 3.7 excels (agentic coding, app development, greenfield coding, human-in-the-loop scenarios).
• Prompt Engineering:
  • Overeagerness: Claude 4 models are less overeager. Remove language in prompts designed to dampen overeagerness in Sonnet 3.7. Explicitly instruct the model to "go above and beyond" if desired.
  • Attention to Detail: Audit prompts to ensure they encourage desired behaviors.
  • Tool Use:
    • Prompt Claude 4 to call tools in parallel.
    • Specify what Claude should think about between tool calls (e.g., reflect on search result quality).
    • Clearly define when and when not to invoke tools.

IV. Conclusion

• Anthropic is building towards a vision of Claude as a collaborative and independent AI agent capable of sustained performance.
• Claude 4 models (Opus and Sonnet) offer significant improvements in thinking and tool use, memory, instruction following, and reduced reward hacking.
• Users should experiment with both models, invest in prompt engineering, and provide feedback to help improve future generations of Claude.

V. Q&A Highlights

• Benchmarks vs. Practical Use: Anthropic uses a "Swiss cheese" of testing methods, including benchmarks, internal use, and early access customers. They are interested in developing more benchmarks but don't rely on them solely.
• Multimodal Capabilities: Claude models can see and respond to images, which is considered important for agent capabilities (e.g., fixing front-end designs).
• Tool Calling as a Survey Mechanism: Using tool calling to present options (A, B, C) for analysis is an interesting use case.
• Navigating Large Legacy Codebases: Anthropic is improving Claude's ability to do agentic search (iterative search and analysis) and leveraging memory capabilities to store information about the codebase.
• Controlling Thinking Length: Users can control the maximum thinking length, but the model adapts its thinking length to the task's needs.
• Feedback Mechanism: Talking to Anthropic employees and using online feedback forms are preferred methods.
• Excessive Comments in Code: The improved steerability and reduced overeagerness should help reduce excessive comments in generated code.