I used LLaMA 2 70B to rebuild GPT Banker...and its AMAZING (LLM RAG)

Key Concepts

Llama 2 70B, Retrieval Augmented Generation (RAG), Llama Banker, Open Source LLMs, Hugging Face Transformers, Langchain, Llama Index, Sentence Transformers, Text Streamer, GPU utilization, Streamlit deployment, Model caching, Vector Store Index, Prompt Engineering.

Llama 2 70B and the Llama Banker Project

The video details the process of building an open-source Retrieval Augmented Generation (RAG) engine called "Llama Banker" using the Llama 2 70B model. The goal is to replicate the functionality of a similar project built with OpenAI models, but using open-source alternatives. The project aims to answer questions, summarize, and analyze documents (specifically a 300-page annual report) with performance comparable to ChatGPT, but with the advantage of unlimited tokens and lower cost (estimated at $1.69/hour).

Setting up the Environment and Dependencies

The initial steps involve installing necessary dependencies, including:

• PyTorch: The fundamental deep learning framework.
• Langchain: A framework for building applications powered by language models (though the video aims to minimize its use).
• Transformers: Hugging Face's library for working with pre-trained models.
• Sentence Transformers: For generating document embeddings.
• Llama Index: A data framework for building LLM applications.
• SentencePiece: A tokenizer used by some models.

The video emphasizes running the application on RunPod to leverage a GPU powerful enough to handle Llama 2 70B.

Loading and Configuring the Llama 2 Model

The process of loading the Llama 2 model involves:

1. Accessing the Model Weights: Requires requesting access to the Llama 2 repository on the Meta website.
2. Using AutoTokenizer and AutoModelForCausalLM: From the transformers library to load the tokenizer and model, respectively.
3. Caching the Model: Using the cache_dir parameter to save the model to a specific directory.
4. Authentication: Using the use_auth_token parameter to access the restricted Llama 2 repository.
5. Addressing Configuration Issues: The video highlights initial issues with the original Meta weights, specifically related to the pad_token configuration.
6. Alternative Weights: The video uses the upstage weights as an alternative, which were pre-configured for 8-bit loading and rope scaling.
7. GPU Scaling: Loading the model in 8-bit precision allows it to run on a single A100 80GB GPU.

Text Streaming and Model Generation

The video introduces a TextStreamer class to stream the decoded output of the model. The process involves:

1. Passing a Prompt to the Tokenizer: Converting the input text into numerical tokens.
2. Sending Tokens to the GPU: Using the model.device attribute.
3. Using model.generate: To generate the model's output, with parameters like streamer, use_cache, and max_new_tokens.
4. Decoding the Output: Using tokenizer.decode to convert the generated tokens back into text.

Implementing Retrieval Augmented Generation (RAG) with Llama Index

The video transitions to implementing RAG using Llama Index, aiming to answer questions based on the provided annual report.

1. Installing Llama Index: Using pip install llama-index.
2. Creating Prompts: Using SimpleInputPrompt from Llama Index for prompt formatting.
3. System Prompt: Incorporating a system prompt to guide the model's behavior.
4. HuggingFaceLLM Class: Using this class to integrate the loaded Llama 2 model with Llama Index.
5. Handling Document Embeddings: Using Sentence Transformer models for generating embeddings.
6. Langchain Embedding Wrapper: Wrapping Hugging Face embeddings within the Langchain embedding class for compatibility with Llama Index.
7. Service Context: Configuring Llama Index to use the loaded model and embeddings by creating a custom service context using ServiceContext.from_defaults and setting it globally with set_global_service_context.
8. Chunk Size: The video mentions the importance of chunk size in determining how the document is split.

Document Loading and Indexing

The video explores different methods for loading the document:

1. SimpleDirectoryReader: Initially used but found to have issues with skipping pages.
2. Llama Hub and Pi Mu PDF Reader: Using download_loader from Llama Hub to access the PiMuPDFReader for loading the PDF.
3. Vector Store Index: Creating a vector store index using VectorStoreIndex.from_documents to store the PDF chunks and enable querying.

Querying the Index and Evaluating Results

The video demonstrates querying the index using index.as_query_engine and query_engine.query. A specific query ("What was the FY 2022 return on Equity?") is used to validate the accuracy of the RAG system. The model successfully retrieves the correct information from the annual report.

Streamlit Deployment

The video attempts to deploy the Llama Banker application using Streamlit:

1. Creating a Streamlit App: Using st.title and st.text_input to create a basic interface.
2. Integrating Generation Code: Copying the code used for model generation into the Streamlit app.
3. Running the App: Using streamlit run app.py.
4. Port Forwarding: Using RunPod's TCP mapping functionality to expose the app to the internet.
5. Addressing GPU Memory Issues: Encountering GPU memory errors due to Streamlit reloading the model on each query.
6. Model Caching with st.cache_resource: Using the st.cache_resource decorator to prevent Streamlit from reloading the model and resolve the memory issue.
7. Displaying Results: Using st.expander to display the full response object and source text.

Conclusion

The video concludes that Llama 2 shows significant promise in the field of open-source large language model development. The Llama Banker application demonstrates the flexibility and power of Llama 2 for tasks like summarizing financial performance, performing entity extraction, and analyzing sentiment. The video also highlights the challenges of deploying such applications and the importance of techniques like model caching to optimize performance.