How to Get Your Data Ready for AI Agents (Docs, PDFs, Websites)

Key Concepts

• AI Agents & Knowledge Systems: Building AI agents requires providing them with specific knowledge, often from custom data sources.
• Open Source Document Extraction: Using open-source tools for document extraction offers an alternative to closed-source APIs.
• DocLink Library: A Python library for open-source document extraction and chunking.
• Document Conversion & Parsing: Converting various file types (PDF, HTML, etc.) into a unified data model.
• Chunking: Splitting documents into smaller, semantically meaningful units for efficient retrieval.
• Embeddings & Vector Databases: Converting text chunks into vector embeddings and storing them in a vector database for similarity search.
• Retrieval Augmented Generation (RAG): Retrieving relevant context from a vector database to inform the AI model's response.

1. Document Extraction with DocLink

• Main Point: DocLink is presented as a leading open-source library for document extraction, offering a robust alternative to paid APIs.
• Specific Details:
  • Developed by IBM.
  • Supports various file types: PDF, PowerPoint, DOCX, websites.
  • Transforms documents into a unified data model called the "DocLink document."
  • Includes features like table extraction, which is often a challenge for other libraries.
• Example: The video demonstrates extracting data from a PDF technical report and a website.
• Process:
  1. Install DocLink using pip install docklink.
  2. Use the DocumentConverter class to convert files or websites.
  3. Access the extracted content through the document attribute.
  4. Export the document to formats like Markdown or JSON.
• Code Snippets:
  • converter = DocumentConverter()
  • document = converter.convert(pdf_path)
  • markdown_output = document.to_markdown()
• Sitemap Extraction: The video explains how to extract all pages from a website using its sitemap.xml file.
  • A helper function get_sitemap_urls is used to fetch URLs from the sitemap.
  • The convert_all method is used to process multiple URLs.

2. Chunking with DocLink

• Main Point: DocLink provides built-in chunking methods to split documents into smaller, manageable pieces for embedding.
• Specific Details:
  • Chunking is essential for efficient retrieval and to fit within the token limits of embedding models.
  • DocLink offers two main chunking methods: hierarchical chunker and hybrid chunker.
  • Hierarchical Chunker: Splits documents based on logical components like lists and paragraphs.
  • Hybrid Chunker:
    • Splits chunks that exceed the maximum input tokens of the embedding model.
    • Stitches together chunks that are too small.
    • Uses a tokenizer to ensure chunks fit the chosen model.
• Example: The video demonstrates using the hybrid chunker with the OpenAI text-embedding-3-large model.
• Process:
  1. Import the HybridChunker class from DocLink.
  2. Create a tokenizer wrapper for the OpenAI model.
  3. Initialize the HybridChunker with the tokenizer and max tokens.
  4. Apply the chunker to the document.
• Code Snippets:
  • chunker = HybridChunker(tokenizer=openai_tokenizer, max_tokens=max_input_tokens, merge_pairs=True)
  • chunks = chunker.chunk(document)
• Token Limits: The video emphasizes the importance of considering the maximum input tokens of the embedding model.
  • The text-embedding-3-large model has a specific token limit.

3. Embedding and Vector Database Integration

• Main Point: The video demonstrates how to create embeddings from the chunks and store them in a vector database using LanceDB.
• Specific Details:
  • LanceDB is used as an example due to its ease of use and persistent storage.
  • The video highlights that the principles can be applied to other vector databases like PostgreSQL with PGVector.
  • LanceDB's API allows specifying an embedding model as a function.
  • Pydantic models are used to define the schema of the vector database table.
• Process:
  1. Initialize a LanceDB database.
  2. Define an embedding function using the OpenAI text-embedding-3-large model.
  3. Create a Pydantic model to define the table schema, including text, vector, and metadata fields.
  4. Create a table in LanceDB using the defined schema.
  5. Loop through the chunks, extract relevant data, and add them to the table.
• Code Snippets:
  • db = lancedb.connect(uri)
  • table = db.create_table("docklink", schema=ChunkSchema, mode="overwrite")
  • table.add(chunks)
• Metadata: The video emphasizes the importance of including metadata like file name, page numbers, and title in the vector database.
• Pydantic Note: The video mentions a potential bug in Pydantic where sub-models must be ordered alphabetically to avoid errors.

4. Search and Retrieval

• Main Point: The video demonstrates how to perform similarity searches on the vector database to retrieve relevant context.
• Specific Details:
  • LanceDB's API provides a simple search method for performing similarity searches.
  • The search method allows specifying the query, query type (vector search), and limit.
• Process:
  1. Connect to the LanceDB database.
  2. Load the table.
  3. Use the search method with a query to retrieve relevant chunks.
  4. Convert the results to a Pandas DataFrame for easy viewing.
• Code Snippets:
  • table = db.open_table("docklink")
  • results = table.search(query).limit(5).to_pandas()

5. Chat Application Integration

• Main Point: The video demonstrates how to integrate the knowledge extraction pipeline into a Streamlit chat application.
• Specific Details:
  • Streamlit is used to create a simple interactive chat interface.
  • The application connects to the vector database, searches for relevant context based on user queries, and displays the results.
• Process:
  1. Set up a Streamlit application with chat elements.
  2. Create a function to search the vector database and retrieve relevant context.
  3. Use Streamlit components to display chat messages and retrieved sources.
• Command: streamlit run five-chat.py (executed from the correct directory with the environment activated)

6. Synthesis/Conclusion

The video provides a comprehensive guide to building an open-source document extraction pipeline using DocLink. It covers the key steps of extraction, chunking, embedding, and retrieval, demonstrating how to integrate these components into a functional AI application. The use of DocLink, LanceDB, and Streamlit provides a practical and accessible approach to building knowledge systems for AI agents. The video emphasizes the importance of understanding the underlying concepts and adapting the techniques to different tools and scenarios. The final chat application showcases the power of combining these technologies to create an interactive and informative AI experience.