Books reimagined: use AI to create new experiences for things you know — Lukasz Gandecki, Xolvio

Key Concepts

AI-enhanced reading experiences, Large Language Models (LLMs), scene analysis, character detection, music generation, structured XML metadata, AI voice assistants, semantic search (embeddings), refactoring strategies, human-AI collaboration, open-sourcing.

1. Introduction: Books Reimagined

Sukash Ganditski introduces his project, "Books Reimagined," which uses AI to create new reading experiences. The initial motivation stemmed from difficulty understanding the characters in a book about Donald Trump's re-election. This led to the development of an AI companion app that provided context and images for characters on each page.

2. Demonstrations: The Snow Queen and 1984

• The Snow Queen: Ganditski showcases an early experience built for "The Snow Queen," featuring music and visual effects synchronized with the story. The example is in Polish.
• 1984: A version built for the conference in English. A key feature is the ability to send voice notes to the book and receive immediate, context-aware responses from an AI voice assistant. This addresses the common issues with existing voice assistants (delays, interruptions, inaccurate responses). Users can ask follow-up questions to summarize what happened since the last question. The demo can be checked at bookgenius.net.

3. Enhanced Search Functionality

• Semantic Search: Traditional exact search is insufficient for finding scenes based on descriptions. Semantic search using embeddings allows users to find scenes even if they don't remember the exact wording (e.g., finding the scene where Winston met O'Brien).
• Deep Research: Users can ask complex questions (e.g., "talk about all the ways the party propaganda works"), and the AI will read the book to provide a comprehensive answer.

4. Development Process and Refactoring

• Rapid Iteration with Vanilla JavaScript: Ganditski started with vanilla JavaScript for rapid iteration, acknowledging the initial code was confusing.
• AI-Assisted Development: He emphasizes the speed of generating working code with AI, allowing for quick experimentation and refinement. Throwing away AI-generated code feels less painful than discarding manually written code.
• Waves of Change: The development process involves iterative changes, with the amplitude of changes decreasing over time. Eventually, traditional engineering practices (testing, refactoring) can be applied.
• Refactoring Strategy: Avoid refactoring code that is bad but not painful to maintain. Prioritize refactoring based on a combination of how bad the code is, how painful it is to maintain, and how easy it is to change. He uses the formula: Badness * Pain * Ease of Change to prioritize refactoring efforts. He gives an example of OpenAI audio processing code that is confusing but never needs to be touched, so he avoids refactoring it.

5. Human-AI Collaboration

• Hiding the AI: The goal is to make the AI invisible to the user, creating a seamless experience.
• Human Touch: AI handles the initial draft, but humans are crucial for ensuring quality (e.g., evaluating music, graphics, avatar appropriateness). "The human touch is invaluable."
• Simple Building Blocks: Combining simple AI-powered features creates a magical reading experience.

6. Key Features and Benefits

• 24/7 Availability: AI provides instant answers to questions, unlike human friends.
• Spoiler-Free Search: Natural language search avoids spoilers.
• Immersive Experience: Graphics and music enhance the mood and character recall, creating a movie-like experience. "Music influences the emotions hugely."

7. Production Process

• LLMs for Scene Analysis and Character Detection: LLMs are used to analyze scenes and identify characters.
• Music Generation: AI generates music based on an overall theme (e.g., "Victorian London, noir music" for Sherlock Holmes).
• Structured XML Metadata: AI extracts structured data from the text (e.g., mapping characters to scenes), which is then used to enhance the reading experience. This is very time-consuming for a person to do manually.

8. Open Sourcing the Player

The player is being open-sourced to allow others to create Netflix-style experiences for books.

9. Conclusion

Ganditski encourages developers to explore niches where AI can create new experiences on top of existing technologies. AI makes it possible to produce enhanced books at a scale that was previously impossible due to the high cost of manual production. He emphasizes building AI experiences that "ship in the light and not slides."

Key Concepts Explained

• LLMs (Large Language Models): AI models trained on vast amounts of text data, capable of understanding and generating human-like text.
• Embeddings: Numerical representations of text that capture semantic meaning, enabling semantic search.
• Vanilla JavaScript: Basic JavaScript without the use of libraries or frameworks.