Statistician & Fortune 100 advisor on making smarter decisions with AI

Key Concepts

• Decision Intelligence: Turning information into better action across all scales and settings.
• Data-driven vs. Data-inspired/Data-decorated: The difference between using data to drive decisions versus using it to support pre-existing opinions.
• Confirmation Bias: The tendency to interpret information in a way that confirms one's existing beliefs.
• Pre-commitment: The discipline of deciding how data will be used to drive decisions before looking at the data.
• Generative AI Value Gap: The disparity between individual users finding value in generative AI and organizations struggling to measure that value.
• Thunking vs. Thinking: Thunking is routine, pre-decided execution, while thinking is engaged, creative problem-solving.

Decision Intelligence: An End-to-End Approach

Cassie Kozyrkov defines Decision Intelligence as the discipline of turning information into better action, regardless of scale or setting. It bridges the gap between various decision-making disciplines, including psychology, social sciences, managerial sciences, and data/mathematical sciences. This end-to-end approach is crucial because technology allows for effortless execution, raising questions about whether we are asking the right questions and understanding the answers we receive. As we use increasingly powerful language with machines, we must be aware of the consequences and the true meaning of our words.

Data-Driven vs. Data-Inspired Decisions

Kozyrkov highlights the common misconception that many decision-makers believe they are data-driven when they are merely data-inspired or data-decorated. This occurs when data is used to support pre-existing opinions rather than to genuinely inform the decision-making process. This can be likened to using data as a "mood board" instead of a "recipe" or "blueprint."

Example: A manager who already favors a particular marketing campaign might cherry-pick data points that support their preference, ignoring contradictory evidence.

Combating Confirmation Bias with Pre-Commitment

Confirmation bias significantly impacts how we interpret information based on our pre-existing beliefs. To combat this, Kozyrkov emphasizes the importance of pre-commitment: establishing the structure for decision-making before examining the data. This involves setting "goalposts" before "kicking the ball," preventing the manipulation of data to fit a desired outcome. Leaders must take responsibility for this pre-commitment process, bridging the gap between engineering/data language and leadership/decision-making language.

Decision Intelligence in the Age of Generative AI: Navigating Endless Right Answers

Generative AI presents a challenge by generating numerous "good-ish" possibilities, potentially overwhelming decision-makers. Drawing on psychology research, Kozyrkov notes that too many options can hinder decision-making.

Example: Choosing between two jam flavors is easier than choosing between 16.

With generative AI, the sheer volume of options (e.g., 16,000 or 16 million) necessitates strategies for limiting choices and focusing on desired outcomes. This is particularly relevant when the distance between options is small, such as choosing between similar vacation destinations like Paris and Madrid. In such cases, connecting with personal values and goals becomes crucial for breaking ties. AI can assist in generating options, but it should not replace human judgment and goal-setting.

The Generative AI Value Gap: Measuring ROI at Scale

While individuals may find immediate value in generative AI, organizations often struggle to measure its ROI at scale. This "generative AI value gap" arises because these technologies don't inherently define success metrics. Leaders must define what success looks like for the system as a whole, including establishing cutoffs for acceptable answers.

Example: If creating social media copy automatically, define what makes one piece of copy "better" than another and by how much.

Leveraging AI as a Thought Partner

AI can serve as a valuable thought partner by helping decision-makers consider perspectives they may have overlooked. By asking AI to identify ignored factors or unacknowledged assumptions, leaders can break out of established patterns and achieve better outcomes.

Example: Using AI to brainstorm 50 additional considerations for a decision, even if most are irrelevant, can uncover a crucial insight that was previously missed.

Applications of AI: Drug Discovery and Language Translation

Kozyrkov highlights drug discovery as an area where AI is significantly accelerating progress. AI supplements human capabilities in memory (holding abstract concepts) and language (transmitting information). Generative AI democratizes access to technology by allowing users to interact with machines in their own language, rather than requiring specialized coding skills. However, she cautions that natural language is inherently imprecise, unlike mathematics.

The Proto-Genie: Guardrails and Safety Nets

Kozyrkov likens generative AI to a "proto-genie," where prompts are akin to "proto-wishes." This analogy underscores the need for guardrails and safety nets to manage the potential for surprise, uncertainty, complexity, and chaos. It's crucial to ensure that prompts are well-constructed and that the system's outputs align with intended goals, especially when scaling up AI applications within an organization.

Personal Use of AI: Augmentation, Not Abdication

Kozyrkov uses AI to enhance her effectiveness by automating drudgery, particularly in language translation. She emphasizes that translation extends beyond simple language conversion to include tasks like converting bullet points into fleshed-out emails and vice versa. However, she firmly believes that AI should not be used for thinking on her behalf.

Example: Unlike her father who uses a coin toss to make indifferent decisions, Kozyrkov argues against letting AI, a "probability engine," dictate important choices.

Humans must remain in control, defining what is important and using AI as a tool to achieve their goals.

The Future of Work: Optimizing for Thinking

Kozyrkov introduces the concept of "thunking" (routine execution) versus "thinking" (engaged problem-solving). She predicts that AI will automate more and more thunking, creating a challenge for managing thinking effectively. Simply compressing work schedules to force "pure thinking" is unlikely to succeed. Instead, organizations must find ways to optimize for creative and engaged moments.

Challenge: Measuring creativity is difficult, as traditional metrics focus on easily quantifiable aspects like time spent, words typed, or customers served.

Kozyrkov emphasizes the importance of incorporating activities that promote creativity, even if they seem unproductive. She finds data entry soothing and uses it as a way to relax and stimulate creative thought. The key is to address how leadership will handle the "empty space" created by automation, ensuring that it is used to foster creative ideas, healthy cultures, and productive work environments. This is a critical challenge that needs to be addressed within the next three to five years.

Conclusion

The conversation with Cassie Kozyrkov emphasizes the importance of a thoughtful and strategic approach to decision-making in the age of AI. By understanding the principles of Decision Intelligence, combating confirmation bias, and focusing on human values and goals, organizations can effectively leverage AI to enhance productivity, foster creativity, and achieve better outcomes. The key takeaway is that AI should be used as a tool to augment human capabilities, not to replace human judgment and critical thinking.