Project Astra | Exploring the future capabilities of a universal AI assistant

Key Concepts:

• Project Astra: A research prototype exploring future capabilities of a universal AI assistant.
• Gemini 2.0: Google's newest model enabling agentic capabilities for Project Astra.
• Universal AI Assistant: An AI capable of understanding and responding to various user needs and contexts in real-time.
• Agentic Capabilities: The ability of an AI to act autonomously and proactively to achieve goals.
• Multilingual Capabilities: The ability of an AI to understand and respond in multiple languages.
• Contextual Awareness: The ability of an AI to understand and respond based on the surrounding environment and user history.

Project Astra Overview and Gemini 2.0 Integration

Project Astra is a research prototype designed to explore the potential of a universal AI assistant. It leverages Gemini 2.0, Google's latest model, to enable agentic capabilities. The demonstration involves testing Project Astra in real-world scenarios around London using a Pixel phone and prototype glasses.

Real-World Use Cases and Examples

• Remembering Information: The user asks the agent to remember a door code (1170) from an email, and the agent successfully recalls it later.
• Laundry Instructions: The agent identifies washing instructions from clothing tags (machine wash at 30°C, avoid bleach and tumble drying, iron on low, and dry cleanable) and suggests the appropriate washing machine setting ("30" setting under "Easy Care").
• Providing Information About Places: The agent provides information about Coal Drops Yard (a shopping destination with unique industrial design), Luminary Bakery (a social enterprise bakery), and Tate Modern (a modern and contemporary art museum) based on a list of recommendations.
• Pronunciation Assistance: The agent explains the different pronunciations of "scone" ("skon" rhyming with "gone" and "skown" rhyming with "bone"), noting regional variations.
• Identifying Sculptures: The agent identifies the sculpture "My World and Your World" by Eva Rothschild in Lewis Cubitt Park.
• Plant Suitability: The agent determines that Mountain Fleece can grow well in New York, given its preference for full sun to partial shade and moist, well-drained soil.
• Book Recommendation: Based on a list of books, the agent suggests that the user's friend would like "Half of a Yellow Sun" due to its exploration of complex social and political issues within a historical setting.
• Navigation Assistance: The agent confirms that the 24 bus route goes through Leicester Square, which is close to Chinatown, and lists landmarks along the way (Westminster Abbey, Big Ben, and Trafalgar Square).

Multilingual Capabilities

The video demonstrates Project Astra's multilingual capabilities with friends speaking in French and Tamil.

• French Example: When asked about lanterns in French, the agent responds in French, explaining that the lanterns are part of the gateway to Chinatown in London and have cultural significance.
• Tamil Example: When asked about the entrance to Chinatown in Tamil, the agent responds in Tamil, explaining that it is a significant landmark representing the cultural importance of the area.

Hands-Free Testing with Prototype Glasses

The user tests Project Astra hands-free using prototype glasses.

• Weather Information: The agent provides a weather forecast, stating that it is mostly sunny but rain is expected from 10 p.m. to 6 a.m.
• Identifying Parks: The agent identifies Primrose Hill, noting its panoramic city views and Shakespeare's tree.
• Cycling Restrictions: The agent informs the user that cycling is banned within Primrose Hill but permitted in nearby areas and Regent's Park.
• Supermarket Locations: The agent lists supermarkets along the route back to Camden: Sainsbury's on Camden Road, Morrisons on Chalk Farm Road, and M&S Simply Food on Camden High Street.

Conclusion

Project Astra, powered by Gemini 2.0, demonstrates the potential of a universal AI assistant to provide real-time, contextually aware, and multilingual support across various tasks. The prototype showcases capabilities such as remembering information, providing instructions, identifying objects, offering recommendations, and assisting with navigation, highlighting its potential for future applications.