Shaping Model Behavior in GPT-5.1— the OpenAI Podcast Ep. 11

Key Concepts

• GPT-5.1: The latest iteration of OpenAI's models, focusing on reasoning capabilities and improved user experience.
• Reasoning Models: Models capable of "thinking" before responding, akin to Daniel Kahneman's System 1 and System 2 thinking, allowing for more refined answers and tool utilization.
• Model Behavior: The focus on shaping how models interact with users, including aspects like warmth, intuition, and instruction following.
• Steerability: The ability of users to guide the model's responses and behavior to achieve their desired experience.
• Personality (Model): Encompasses both explicit response styles (concise, lengthy, emoji usage) and the overall user experience of the model, including UI, latency, and harness.
• Auto-switcher: A mechanism that transitions users between different models (e.g., chat and reasoning) based on context, aiming for a seamless experience.
• Custom Instructions: User-defined preferences that the model should consistently adhere to.
• User Signals Research: Training reward models and gathering signals during Reinforcement Learning (RL) from user data to improve model behavior, particularly in understanding user intent and emotional intelligence (EQ).
• User Experience (UX): The holistic interaction a user has with the model, influenced by various components beyond just the core model.
• Maximizing Freedom, Minimizing Harm: OpenAI's core principle in model development, balancing user agency with safety.
• Safe Completions: A safety feature where the model attempts to fulfill a request while avoiding harmful actions.
• Subjective Domains: Areas where there isn't a single objective truth, requiring models to express uncertainty and explore ideas openly.
• Creativity (Model): The model's ability to generate responses with a wider expressive range, from highly elevated to very simple.
• Memory: A feature where the model stores information about the user from past conversations to personalize future interactions and avoid repetition.
• Prompt Engineering: The art of crafting effective prompts to elicit desired responses from AI models.

GPT-5.1: Advancements and User Experience

Reasoning Capabilities as Default

A significant advancement in GPT-5.1 is that all models within ChatGPT are now reasoning models. This means the model can internally decide to "think" before responding, a process described as "chain of thought." The extent of this thinking is determined by the complexity of the prompt. For simple greetings, no extensive thought process is initiated. However, for more challenging questions, the model will engage in a more deliberate thinking process, allowing it to refine its answers, utilize tools if necessary, and provide a more considered response. This capability is likened to Daniel Kahneman's System 1 (intuitive, fast) and System 2 (deliberative, slow) thinking.

The widespread availability of reasoning models as the default for all users is expected to lead to across-the-board improvements, particularly in instruction following and general intelligence. This enhanced intelligence, stemming from the model's ability to "think before it responds," significantly aids in various use cases, even those not immediately perceived as requiring deep reasoning. This has been validated by improved evaluations (evals) across various metrics.

Addressing User Feedback: Warmth and Intuition

A key focus for GPT-5.1 was to address feedback received after the ChatGPT-5 launch, specifically concerning the model feeling "weaker intuition" and "less warm." Investigations revealed that this perception was influenced by several factors:

• Context Window Limitations: The model's context window was not retaining enough information from previous user interactions. This could lead to the model "forgetting" crucial details shared by the user, creating a feeling of coldness or a lack of understanding. For instance, if a user expresses having a bad day and the model forgets this after several turns, it can feel dismissive. This has been adjusted in GPT-5.1.
• Response Style Discrepancies: The introduction of an auto-switcher in GPT-5, which moved users between chat and reasoning models, resulted in slightly different response styles. This jarring transition could feel cold, especially in sensitive conversations. For example, a user discussing a personal hardship might be switched to a reasoning model for a clinical answer, creating an incongruous experience. Many changes in GPT-5.1 aimed to ensure a warmer aggregate experience despite underlying technical shifts.
• Instruction Following: GPT-5.1 demonstrates significant improvement in following custom instructions. Users often accept model quirks as long as they have control. However, if the model fails to retain these custom instructions or context, it becomes problematic. The custom instructions feature has been enhanced to ensure more consistent adherence.
• Personal Preference and Control: Recognizing that many aspects of model interaction are subjective, OpenAI introduced style and trait features, including "personality" settings, to give users more control over the model's response formats and overall interaction style.

Understanding the "Switcher" and Multiple Models

The concept of a "switcher" and multiple models can be confusing for users. OpenAI explains that their models possess distinct capabilities, making it challenging to stay updated. Product work focuses on creating user interfaces (UIs) that guide users to the appropriate model for their needs. This includes the model switcher itself and its ability to learn which answers are most helpful in different contexts. For instance, reasoning models are evaluated on their ability to provide scientifically accurate and detailed answers for specific prompt types.

The fact that the free tier now uses a reasoning model signifies a commitment to providing the most intelligent model possible to everyone. This opens doors for exploring more advanced applications with frontier models that can think for extended periods, potentially for background tasks or as tools. The future is envisioned as a "system of models" rather than a single monolithic entity, incorporating various reasoning models, lighter reasoning models, the auto-switcher, and different tools backed by distinct models. This interconnected system is expected to unlock more interesting use cases and product implications as models become more intelligent.

Data, Feedback, and Measuring Progress

Leveraging User Feedback and Conversation Links

With a large user base (800 million users), OpenAI receives extensive feedback. A crucial method for processing this feedback is through conversation links. By analyzing actual user conversations, the team can pinpoint specific issues, such as a model responding coldly or with clipped sentences. This allows for targeted solutions and helps identify if a user was part of an experiment with potential edge cases. For the auto-switcher, signals like user satisfaction, factuality of responses, and latency are monitored to optimize its performance. The balance between response quality and speed is critical, as not all users prioritize waiting for a better answer.

Measuring Emotional Intelligence (EQ)

Improving a model's "IQ" is addressed through benchmarks and evals. However, measuring EQ (emotional intelligence) is more open-ended. OpenAI's research agenda includes user signals research, which involves training reward models and using RL signals derived from user data. This research aims to understand user intent and how models can better respond by considering conversational context, user memory, and history.

EQ is also enhanced by a model's ability to listen, remember, and pick up on subtle signals. This involves ensuring the context window effectively carries information, memory is logged correctly, and the model adopts a style that resonates with the user. The personality features are designed to contribute to this perceived EQ by allowing users to interact with a model that has a style they find appealing.

Defining and Shaping Model Personality

Personality: Response Style vs. Overall Experience

The term "personality" for a model is defined in two ways:

1. Response Style/Tone: This refers to specific traits like conciseness, response length, and emoji usage. OpenAI acknowledges that this aspect of the term might evolve, with potential renaming to "response style" or "style and tone."
2. Overall User Experience: For most users, "personality" encompasses the entire interaction with the model. This includes the app's aesthetics (font, latency), the "harness" (context window, rate limiting), and how these elements combine to create a feeling.

The art of shaping model behavior involves understanding what users perceive as "personality" and mapping it back to the underlying components within ChatGPT and the models that contribute to that experience.

The Art of Post-Training and Steerability

Shaping model personality during post-training is a complex balancing act. Researchers consider various capabilities and make subtle tweaks through RL and reward configurations to achieve desired outcomes without sacrificing essential qualities like "warmth."

Users often perceive the entire ChatGPT experience (image generation, voice, text) as the model's personality, viewing it as a unified "omni experience." However, it's an assembly of many components. Future improvements will focus on seamless integration and consistent enhancement of these elements.

A significant challenge is maximizing user freedom while minimizing harm. This means allowing users to explore a wide range of requests without enabling harmful actions. For example, while a user might want to use specific punctuation like em-dashes, the model shouldn't be trained to never use them, as this would restrict user freedom. The art lies in extracting model quirks that contribute to personality without compromising steerability, which is paramount for user control.

OpenAI aims to avoid the trap of making models overly restrictive, which can lead to them refusing most requests. The goal is to create usable and helpful models by finding the right boundaries for decision-making.

Evolving Safety and Nuance

Early versions of ChatGPT were highly restrictive, often refusing requests. The current approach, exemplified by "safe completions," aims to fulfill requests earnestly while avoiding harmful actions. This is a continuous negotiation to define appropriate boundaries.

The ability to handle subjective domains is crucial. Models need to express uncertainty, engage with user-provided ideas, and remain anchored in objective truths when available. This allows for more open-ended conversations where users can self-direct the discussion.

The creativity of models is also a significant area of development. GPT-5.1 offers a much wider expressive range, allowing for highly elevated or very simple language. This is particularly challenging in post-training because many creative tasks lack a definitive "ground truth" answer, making it an art to balance subjective improvements with objective ones.

Future Directions and User Guidance

Towards Greater Customization and Steerability

With over 800 million weekly active users, a single model personality is insufficient. OpenAI aims for a future where models are highly customizable and steerable, allowing users to achieve their desired experience. The current "personality" work is a foundational step, with ongoing testing and iteration.

The power of customization was illustrated by an anecdote where a researcher, by priming the model with their expertise and context, received a proposal that mirrored their lab's recent breakthrough. This highlights the immense potential of models when users know how to customize them, a skill humanity is still developing. OpenAI is focused on developing tools and features within ChatGPT to help users advance their understanding and maximize their use of these models.

The Evolution of Prompt Engineering and Inferred Understanding

The concept of prompt engineering, which involved users explicitly instructing models (e.g., "you are a grad student"), is evolving. As models become more intelligent and develop memory, they are expected to infer user expertise and context, enabling them to communicate in a way that aligns with the user's background without explicit prompting.

However, a key principle for Product Managers (PMs) is that users should always be aware of what the model is inferring about them and have the tools to modify these inferences. Features like memory can be turned on/off or deleted in settings, ensuring user control. This proactive problem-solving, where the model anticipates user needs, is balanced with user agency.

Memory: Enhancing Personalization and Proactivity

Memory allows models to store information about users from past conversations, eliminating the need for repetitive self-introductions. This stored context helps the model provide more grounded and useful responses.

An example of memory's impact is a personalized news digest that proactively pulls research and information based on user interests and past conversations. This demonstrates how memory extends beyond simply recalling past interactions to actively anticipating user needs. This proactive feature is a key way to help users get the most out of these models.

Challenges in Feedback and Debugging

A significant challenge in improving model behavior is when users report that "something feels different" but cannot articulate the exact cause. Anecdotal feedback and screenshots, while helpful, often lack the necessary metadata for debugging. The share feature in ChatGPT is invaluable, as it provides links that allow the team to inspect the context the model had and debug user feedback more effectively.

Future Excitement and User Advice

Unlocking New Possibilities

The speakers express excitement about the incredible capabilities of the models and what people will build with them. The ChatGPT app is seen as having immense opportunity, with users increasingly recognizing the potential of AI. The idea of "intelligence too cheap to meter" suggests that readily available, highly intelligent models will unlock numerous possibilities.

The progress of Large Language Models (LLMs) shows that smarter models unlock new use cases, which in turn should lead to new form factors for interacting with AI.

Advice for Users

To get the best experience, users are advised to:

• Test with challenging questions: Use prompts on topics you know well to pressure-test the model and observe its improvements.
• Keep experimenting: Updates are frequent, so a negative experience today might be different in the future. Continuous engagement is key.
• Ask the model for prompt assistance: Models have become much better at helping users craft more effective prompts.

Personal Style Choices

When asked about their personal style choices for ChatGPT, one speaker admits to using the default setting as it's what they train on. They frequently switch between different settings to understand user experiences. However, they find a combination of "nerd" (exploratory, unpacks things) and a "country Albertan" style enjoyable, though it can lead to humorous situations like the model saying "howdy" in professional contexts.