Open Source Friday with Llmware.ai

Key Concepts

• Open Source Contribution
• Small Language Models (SLMs)
• Retrieval-Augmented Generation (RAG)
• Agent Workflows
• Privacy and Security
• Local and Private AI
• Hardware Acceleration (CPU, GPU, NPU)
• Multimodal AI
• OpenVINO
• PyWebIO
• Fine-tuning
• Deterministic Output
• Transparency
• Community Engagement

Open Source Contribution and Issue Triage

The video begins by addressing how to contribute to open-source projects. A good starting point is helping triage issues, as maintainers often need assistance determining if reported bugs are genuine. Many repositories also label issues as "good first issues," making them suitable for newcomers. Home Assistant is highlighted as an example, showcasing its repository with numerous labeled "good first issues."

Introduction to LLMware

The core of the video features Darren and Nami Oburst from LLMware, an open-source platform for using small language models (SLMs) to run RAG and agent workflows. LLMware prioritizes local, private, safe, and secure AI implementations.

Background and Motivation

Nami's background as a corporate attorney dealing with vast amounts of documents in mergers and acquisitions led her to seek ways to automate document-heavy workflows. The goal was to develop a solution that could operate safely and securely within enterprise boundaries, ensuring data privacy. This led to exploring small language models and their potential for on-device or server-based processing.

Darren's background is in software and software-related services, consulting, and outsourcing. He has experience training models for over 10 years. LLMware emerged from the intersection of their experiences, aiming to apply generative AI to solve specific business problems.

Focus on Small Language Models

LLMware chose to focus on small language models due to their potential for privacy and security. While larger frontier models exist, hardware constraints and the need for data to remain within enterprise walls made SLMs a more viable option. The capabilities of SLMs have significantly improved, making them increasingly competitive for specific tasks.

Nami mentions that when she first met Kevin, small language models weren't popular. Kevin saw the vision and believed in what they were building.

Darren adds that for many enterprise use cases, LLMs are used with closed context text, such as internal documents and data. Smaller models can be fine-tuned to perform tasks like reading, extracting, analyzing, summarizing, and classifying grounded source materials, often achieving the same results as larger models at a lower cost.

Building the LLMware Platform

One of the challenges in using SLMs is their complexity. LLMware aims to simplify their use by handling the logistics of model deployment, managing dependencies, and providing abstractions for seamless model substitution. Frameworks like Langchain and Llama Index were initially focused on larger models like those from OpenAI, whereas LLMware was built with a "small model first" approach.

Open Source and Community Adoption

The open-source nature of LLMware has facilitated technology adoption and community building. The availability of open-source models on platforms like Hugging Face, with over a million models, has been a critical factor. LLMware focuses on providing the tools and infrastructure to deploy and run these models effectively.

Darren notes that the increasing capabilities of open-source models have influenced LLMware's strategy. As base models improve, the need for extensive fine-tuning for specific tasks has decreased.

Innovation and Model Integration

LLMware is designed to allow developers to easily swap in new models as they become available. The platform abstracts away complexities like tokenization schemes and generation parameters, allowing developers to focus on their code. The platform is also adapting to incorporate multimodal capabilities, which are becoming increasingly common in smaller open-source models.

Security and Compliance

Security and compliance are central to LLMware's design. The platform is built for highly regulated industries and includes features to ensure data privacy and security. LLMware minimizes dependencies to reduce security vulnerabilities and encourages users to create their own forks of the library.

Nami states that they built it for highly regulated industries because they are extremely sensitive about the data that they share.

Community Engagement

LLMware actively engages with its community through Discord, YouTube videos, and developer articles. They encourage contributions and provide support to new contributors.

Demo 1: Agent with Custom Tables

Darren presents a demo showcasing an agent with custom tables, highlighting the use of structured data. The demo involves creating a SQL agent that can take natural language queries and generate SQL to query a database.

1. CSV to Database: The process starts with a CSV file containing customer data.
2. Table Creation: LLMware's abstraction layer is used to create a database table from the CSV file, handling validation and schema generation.
3. Agent Creation: A SQL agent is created and loaded with a SQL tool.
4. Natural Language Queries: The agent is then used to answer natural language questions about the data in the table.
5. SQL Conversion: The agent translates the natural language queries into SQL statements.
6. Fact-Based Responses: The SQL statements are executed against the database, and the agent provides fact-based responses based on the data.

The demo uses a local SQLite database, ensuring that all processing happens locally. The journaling capability provides a detailed visualization of the agent's steps.

Human-Agent Interaction

LLMware emphasizes transparency in agent workflows. The platform provides a "whiteboard" or "scratch pad" where all processes write key outputs and descriptions. This allows developers and users to understand what the agent is doing at each step.

Deterministic Output and Fine-Tuning

To ensure deterministic output, LLMware recommends turning the temperature down to zero and disabling sampling during inference. This eliminates stochastic elements and ensures consistent results.

LLMware has fine-tuned a set of models called SLIMs (Structured Language Instruction Models) to output JSON dictionaries. These models are optimized for specific tasks and can be further fine-tuned for enterprise use cases.

Observability and Transparency

LLMware provides detailed metadata at each step of the agent's execution, allowing developers to observe the model's reasoning process. This includes the distribution of tokens considered at each generation step. The platform also includes source verification tools to validate that the output corresponds to the input.

Demo 2: Multimodal AI on AIPC

The second demo showcases multimodal AI capabilities on an AIPC (AI PC) with an Intel Lunar Lake processor. The AIPC has a CPU, GPU, and NPU, allowing for hardware acceleration of AI workloads.

1. Libraries: The demo utilizes PyWebIO for creating a simple UI and OpenVINO for accessing the GPU and NPU.
2. Three Bots: The demo creates three bots:
   • Text Generation Bot: Runs on the CPU and generates text based on a scene.
   • Text-to-Image Bot: Runs on the GPU and generates an image based on the generated text.
   • Topic Classification Bot: Runs on the NPU and classifies the topic of the generated text.
3. Concurrent Processing: The three bots run concurrently, leveraging the different processing units of the AIPC.
4. Real-time Visualization: The demo visualizes the CPU, GPU, and NPU usage in real-time, demonstrating the hardware acceleration.

The demo shows how multimodal AI can be used to create engaging and interactive experiences on local devices.

NPU Utilization

NPU utilization is still in its early stages, but LLMware believes it has significant potential. As chip manufacturers continue to improve NPU capabilities, there will be more opportunities to offload AI workloads to NPUs, improving energy efficiency and performance.

Future Directions

LLMware plans to continue investing in small language models and exploring new use cases for NPU utilization. They aim to democratize AI by making it accessible to everyone, regardless of their resources or data sensitivity. They also plan to integrate document parsing and ingestion capabilities into the platform.

Call to Action

LLMware encourages the community to:

• Check out their GitHub repository (LLMware) and leave a star.
• Install the LLMware package (pip install llmware).
• Join their Discord community.
• Watch their YouTube videos.

Conclusion

LLMware is an open-source platform that is pushing the boundaries of what is possible with small language models. By focusing on privacy, security, and ease of use, LLMware is making AI more accessible to enterprises and developers alike. The platform's innovative features, such as SLIMs, deterministic output, and multimodal AI capabilities, are paving the way for a future where AI is seamlessly integrated into our daily lives.