What they didn't tell you about Falcon 180b (and will it change?)

Falcon 180B: A Deep Dive

Key Concepts:

• Large Language Model (LLM)
• Parameters
• TI UAE
• Hugging Face Leaderboard
• Open Source
• Pre-trained Model
• Fine-tuning
• Context Window
• GPU Requirements
• Quantization
• Flash Attention
• Prompt Engineering
• Code Generation
• General Knowledge

Introduction

Falcon 180B is a large language model (LLM) with 180 billion parameters, developed by TI UAE. It is positioned as a leading open-source model, potentially rivaling GPT-4 and Google's PaLM 2, despite being smaller. The model is freely available for research and commercial use.

Training and Specifications

• Release Date: September 6, 2023
• Developer: TI UAE
• Type: 180 billion parameter decoder-only model
• Training Data: 3.5 trillion tokens
• Context Window: 2048 tokens
• Performance Claims: Ranks just behind GPT-4 and on par with Google's PaLM 2.

Hardware Requirements and Solutions

• High Compute Requirement: Running Falcon 180B at full precision requires 400GB of VRAM, necessitating powerful GPUs.
• RunPod: A paid service used to access GPU resources. The presenter uses this service to run the model.
• Quantization: A solution to reduce VRAM requirements. Using a quantized version of the model allows it to run on two 80GB A100 GPUs. The Reddit user who quantized the model is mentioned.

Model Loading and Setup

• Libraries: Transformers library from Hugging Face is used.
• AutoModelForCausalLM: Used to load the model.
• Keyword Parameters: cache_dir is used to specify where to save the model.
• Better Transformers Class: Used to speed up predictions using flash attention.
• Tokenizer: Used to process input prompts and decode output text.

Testing and Use Cases

• Sally Benchmark Question: Used as an initial test prompt.
  • Prompt: "Sally a girl has three Brothers each brother has two sisters so how many sisters does Sally have"
  • Correct Answer: One
• Prompt Engineering: Necessary to achieve desired results.
• Code Generation: Tested as a use case.
• General Knowledge: Tested as a use case.
• Text Extraction: The tokenizer.decode() method is used to extract text from the model's output.

Important Considerations

• Pre-trained vs. Fine-tuned Models: The initial Falcon 180B model topping the leaderboards was a pre-trained model. The demonstration uses a chat model, which is a fine-tuned version.
• Fine-tuning Impact: Smaller fine-tuned models based on Llama 2 70B have surpassed the Falcon 180B chat model on leaderboards.
• Community Development: The open-source community is expected to further fine-tune Falcon 180B, potentially leading to even better performance.

Conclusion

Falcon 180B represents a significant advancement in open-source LLMs, offering impressive performance with the potential for further improvement through community contributions. While it has high compute requirements, solutions like quantization make it more accessible. The distinction between pre-trained and fine-tuned models is crucial when evaluating its performance.