Do You Trust Your AI’s Inferences? — Sahil Yadav, Hariharan Ganesan, Telemetrak

Key Concepts

AI Trust, Explainability, Traceability, Guardrails, Scalability, Adaptive Control, Human-in-the-Loop, XTOPS, MLOps, Mean Time to Resolve Explainable Errors (MTRE), Trust Adjusted Risk in Dollars, Silent Failures, AI Governance.

Introduction

Sahil and Hari discuss the critical issue of trust in AI systems, drawing on their experience deploying AI across various industries. They highlight the growing adoption of AI but emphasize the significant gap in AI governance, leading to potential "silent failures" with substantial financial and operational consequences.

The AI Trust Gap: Statistics and Examples

• Adoption vs. Governance: While 78% of companies are adopting AI (McKinsey) and 95% are investing in it (EVI), only 11% focus on AI governance.
• Silent Failures: These are difficult to predict and quantify but can cost millions or billions of dollars.
• Examples:
  • Telecom disruption due to AI decision-making, costing millions per minute of network downtime.
  • A gas sensor misinterpreting data, endangering human lives.
  • Supply chain AI errors leading to millions in losses due to SKU mismanagement.

Pillars of Trustworthy AI

• Explainability: Understanding the reasoning behind AI inferences. It's crucial to know "what's under the hood" to avoid "flying blind."
• Traceability: Maintaining a "flight recorder" of all data and changes, enabling recreation of situations and problem-solving.
• Guardrails: Implementing thresholds to prevent significant losses and ensure AI systems stop when necessary.

These pillars are essential for building trust and ensuring scalability in real-world AI deployments.

Pillars of Trust: Detailed Explanation

• Show Its Work: AI should provide simple, understandable explanations for its decisions, eliminating the need for data scientists to translate the system's reasoning.
• Adaptive Control: Building "smart guardrails" that allow the AI system to slow down, change course, or request human assistance when it detects errors or drifts. Analogous to lane assist in vehicles.
• Human-in-the-Loop: Establishing clear roles and playbooks to involve the right experts at the right time with the necessary information, avoiding unnecessary overhead.
• Traceability (Foundation): Digitally signing and tracking every data point and change, similar to tracking a FedEx package from origin to destination.

XTOPS: MLOps with Conscience

• Definition: XTOPS is presented as an evolution of MLOps, incorporating built-in conscience and direct human oversight.
• Life Cycle:
  • Verifiable Traceability: Tracking data provenance and changes from the initial data stage.
  • Actionable Intelligibility: Training models to explain themselves, enabling the detection of reasoning drift.
  • Adaptive Cruise Controls: Implementing guardrails that automatically adjust to new situations and pause when anomalies are detected.
  • Human-AI Teaming: Integrating real-world feedback and enabling human intervention when needed.
• Goal: To ensure every AI decision has a clear "why, when, and who" attached to it, moving from simply launching AI to launching trustworthy AI.

XTOPS vs. MLOps: Key Differences

XTOPS provides an integrated upgrade to foundational MLOps components, specifically for trust:

• Guardrails and Policies: MLOps offers IM and security policies, while XTOPS provides dynamic, AI-aware guardrails that understand context and block risky AI decisions.
• Monitoring and Metrics: MLOps has standard metrics, but XTOPS offers dedicated trust-specific dashboards for leadership and boards.
• Human-in-the-Loop: MLOps uses ad-hoc human intervention, whereas XTOPS creates fast-lane, click-to-fix workflows for quick human review and correction.

XTOPS adds advanced safety and transparency features for high-stakes enterprise AI, reducing firefighting and enabling more innovation.

Measuring Trust: Key Metrics

• Mean Time to Resolve Explainable Errors (MTRE): The time required to understand and fix unexpected AI behavior. A faster MTRE indicates a more agile team, fewer defects, and quicker problem-solving.
• Trust Adjusted Risk in Dollars: Assigning a monetary value to the consequences of trust failures, including fines, lost customers, and reputational damage.

The Importance of Trust Metrics

• Problem: Without focused metrics, resolving AI issues can take months, leading to escalating damage from biased or incorrect decisions.
• Impact: Failures can result in direct fines, increased engineering effort, regulatory scrutiny, and loss of trust and brand value.
• Potential Costs: A serious incident, such as a privacy bug or bias in a credit card system, could cost up to $700 million.
• Purpose: These metrics are not just for defense but for building resilient, reliable, and trustworthy AI-powered products.

Case Study: Guard Hat

• Company: Guard Hat, focused on worker safety in hazardous environments.
• AI Platform: Uses wearable IoT devices to collect health and environmental data, analyzing it in real-time to predict and prevent incidents.
• Problem: High false positive rate (70%) due to GPS input issues, leading users to ignore alerts and creating a safety risk.
• Without XTOPS: Long MTRE, with most time spent identifying the problem and no system to detect GPS drift.
• With XTOPS:
  • Day 0: Alert ignored during an incident.
  • Day 2: Attribution telemetry flags the anomaly.
  • Day 7: Solution deployed to fix or reroute GPS drift.
• Real-World Timeline: The initial problem took 8 months to solve, but the experience led to the development of the XTOPS framework, which then enabled similar problems to be solved in 7 days.

Convincing the CIO: The Financial Argument

• Risk Exposure: Estimated at $2.5 million per site per year.
• Direct Impact: Saving $500,000 per site per year in fines.
• Indirect Impact: Preventing a higher percentage of incidents due to improved AI system performance.

Outcomes of Implementing XTOPS at Guard Hat

• Reduced false alerts.
• Increased trust score, leading to better user engagement with alerts.
• Improved telemetry for understanding AI inferences.
• Enhanced control, such as the ability to switch GPS sources.
• Effective human-in-the-loop processes with dashboards for notifications and action.

Conclusion

The presentation emphasizes the critical need for trustworthy AI systems, highlighting the gap between AI adoption and governance. XTOPS is presented as a solution, building upon MLOps to incorporate explainability, traceability, and human oversight. By focusing on key metrics like MTRE and trust-adjusted risk, organizations can build more reliable and resilient AI systems, ultimately saving money and building trust with users. The Guard Hat case study demonstrates the practical benefits of implementing XTOPS in a mission-critical application.