Waymo: AI in the physical world powering the future of driving

Key Concepts:

• Autonomous Vehicles (AVs)
• Whimo Driver (Whimo's autonomous driving system)
• AI (Artificial Intelligence) and ML (Machine Learning) in autonomous driving
• Operating Domain/Design Domain
• Sensors: Lidar, Cameras, Radar
• Convolutional Neural Networks (CNNs)
• Transformers
• Large Language Models (LLMs) and Visual Language Models (VLMs)
• Whimo Foundation Model
• Simulation for AV development and evaluation
• Safety and Accessibility

1. The Need for Autonomous Vehicles and Whimo's Mission

• The status quo of road safety is unacceptable, with over a million people dying in road collisions annually (one person every 26 seconds).
• The vast majority of these accidents are due to human error.
• Whimo was founded over 15 years ago to challenge this status quo and create a safer mobility option.

2. Early Days at Google and Key Milestones

• Whimo started in 2009 as the Google self-driving car project, with initial testing in the Shoreline Amphitheater parking lot.
• Early challenges from Google's founders, Larry and Sergey, included:
  • Driving 100,000 miles in autonomous mode.
  • Completing 10 routes (100 miles each) with no human intervention.
• These challenges were completed in about 18 months with a small team.
• The focus shifted from proving the possibility of AVs to building a reliable, scalable, and safe autonomous service.

3. Whimo's Current Operations and Growth

• Whimo operates a 24/7 fully autonomous ride-hailing service in San Francisco, Phoenix, LA, and Austin, with plans to expand to Atlanta, Miami, DC, and more cities.
• Serving over a quarter of a million fully autonomous paid trips weekly, a fivefold increase from the previous year.
• Reached a milestone of 10 million fully autonomous paid trips, with half occurring in the last 5 months.

4. Safety Record and Examples of Superhuman Performance

• Whimo driver is approximately five times safer than human drivers in serious collisions (involving injury or airbag deployment).
• Even stronger results when considering collisions involving pedestrians.
• Swiss Re analysis, using insurance claims as a proxy for fault, found the Whimo driver to be about 10 times better than humans.
• Examples:
  • In Austin, the Whimo driver reacted to a scooter rider stumbling onto the road.
  • An oncoming truck swerved into Whimo's lane, and the Whimo driver avoided a head-on collision.

5. Unique Challenges of Autonomous Driving AI

• Complex, messy, and noisy physical environment with unpredictable human behavior.
• Safety-critical environment requiring extremely low error tolerance.
• Real-time operation with onboard processing where milliseconds matter.
• The "long tail" of driving: rare and unusual events that require robust AI.

6. Sensing Modalities and Data Interpretation

• Whimo uses three main sensing modalities: Lidar, cameras, and radar, each with 360-degree coverage.
• These sensors complement each other and provide redundancy.
• AI is crucial for interpreting the raw data from sensors and making driving decisions.
• Example: In Phoenix, during a dust storm, Lidar was able to detect a pedestrian more clearly than cameras.

7. AI and ML Innovations

• Convolutional Neural Networks (CNNs) provided a significant boost in perception around 2013.
• Transformers, starting in 2017, improved perception and enabled breakthroughs in understanding intent, predicting behavior, and decision-making.
• Large Language Models (LLMs) and Visual Language Models (VLMs) offer new possibilities by combining Whimo AI expertise with general world knowledge.

8. Applications of VLMs

• Understanding language and text, such as interpreting complex parking signs.
• General scene understanding, allowing the system to interpret the context of a scene and determine appropriate driving actions.
• Research model "Emma" is an end-to-end ML model that takes input from multiple cameras and outputs a trajectory.

9. Whimo Foundation Model

• Combines AV domain expertise with general world knowledge from VLMs.
• Encoder-decoder model:
  • Encoder (perception side) compresses sensor data into a relevant representation.
  • Decoder (generative side) models behavior, predicts actions, and generates trajectories.
• Serves as the foundation for generative models of agent behavior used in simulation.

10. Simulation for Evaluation and Training

• Key requirements for a good simulator:
  • Large-scale simulation (billions of miles).
  • Closed-loop operation (actions in simulation affect the environment).
  • Realistic sensor and behavioral models.
• Dense 3D reconstruction of environments (e.g., San Francisco) using millions of camera images.
• Use of diffusion techniques to generate controllable models with varying environmental conditions and dynamic traffic situations.

11. Superhuman Performance Example

• In San Francisco, the Whimo driver detected a pedestrian's feet moving under a bus, allowing it to predict their behavior and take defensive action.

12. Generalizability and Future Applications

• The Whimo driver is generalizing well across different geographies (snowy upstate New York, Tokyo) and vehicle platforms.
• Future applications include ride-hailing, local deliveries, long-haul trucking, and personal vehicles.
• By 2030, total vehicle miles traveled are projected to reach 20 trillion.

13. Trust and Accessibility

• Trust is core to Whimo's mission and is earned through safe driving, reliable service, and technology deployment.
• Whimo is designed to be inclusive and accessible to people with disabilities.
• Example: A blind married couple in Phoenix uses Whimo to regain freedom of mobility.

14. Positive Impacts on Society

• Saving lives by reducing accidents.
• Changing our relationship with time (reducing time spent driving).
• Transforming cities (changing streets and parking).
• Making transportation inclusive and accessible.

15. Conclusion

• AI advances have accelerated Whimo's progress, making the Whimo driver generalizable and safe.
• The company is focused on expanding its operating domain and deploying its technology across various applications.
• Trust, safety, and accessibility remain central to Whimo's mission.