360Brew: LLM-based Personalized Ranking and Recommendation - Hamed and Maziar, LinkedIn AI

Key Concepts

• Large Language Models (LLMs) for personalization and ranking
• Promptification
• Zero-shot capability
• In-context learning
• Instruction following
• Model distillation
• Data scaling
• Context length
• Cold start users
• Generalization to new domains
• Model specification, pruning, and quantization
• FP8 and mixed precision
• Attention sparsification

Building a Large Foundation Model for Personalization

• The Problem: Traditional recommendation systems are disjointly optimized, don't leverage advanced architectures, and are slow to roll out.
• The Goal: Build a single large foundation model based on LLMs that understands the user journey holistically and solves all personalization tasks.
• Key Characteristics:
  • Zero-shot capability: Solve new problems/surfaces out-of-the-box by prompting the model.
  • In-context learning: Address cold start user problems by providing a few examples or explaining user interests.
  • Instruction following: Allow users to specify their interests to the model directly.
• Example: Imagine users telling the model their niche interests, and the recommendation system finding relevant information.

Promptification: Converting Data into Prompts

• The Magic of Promptification: Converting user information (history, profiles, interactions) into prompts for LLMs.
• Prompt Structure:
  • Instruction (what the model should do)
  • Member profile information
  • Past interactions
  • Question (e.g., "What do you think the user will do with this new item?")
• Purpose: Formalize the problem for LLMs to understand and process.

Model Development Pipeline

• Starting Point: Open-source LLM.
• Upscaling: "Magic of upcycling" to control model size and throughput vs. quality.
• Training Blocks:
  • Continuous pre-training
  • Fine-tuning
  • Instruction fine-tuning
  • Alignment
• Result: Large model (Blue XL, 150B parameters) with maximized quality.
• Distillation: Reduce model size for online serving (e.g., down to 3B model).
• Automation: Key to managing the complexity of the development cycle.
• Importance of Large Model: Distillation from a large model is crucial; training a small model from scratch doesn't work as well.

Levers for Model Improvement

• Data Scaling:
  • Increasing the amount of training data improves model performance.
  • Leverage extensive user behavior logs (e.g., 6 months to 1 year of data).
• Model Size:
  • Increasing model size (e.g., using Mixture-of-Experts architecture) improves performance.
  • Experiment with sizes like 7B to 8x22B.
• Context Length:
  • Increasing context length (amount of user history fed to the model) improves performance.
  • Performance drops at very long context lengths due to model generalization issues.
  • Key Takeaway: Context length matters significantly for recommendation systems.

Results and Generalization

• Cold Start Improvement: The 360 Brew model significantly outperforms production models for users with few interactions. The gap widens as the number of interactions decreases.
• Generalization to New Domains: The model performs on par with or even beats task-specific production models on completely out-of-domain tasks.
• Significance: Enables faster rollout of new features and surfaces.

Serving Large Models in Production

• Challenges: High QPS (tens of thousands) and low latency (sub-second) requirements.
• Efficiency Levers:
  • Distillation: Step-by-step distillation from a large model (e.g., 150B to 8B to 3B to 1B).
  • Pruning: Gradual pruning of redundant transformer heads and MLPs.
  • Quantization: Using FP8 for activations and model parameters, with mixed precision.
    • Important: LM head (output layer) needs to be in FP32 for good calibration.
  • Specification: Sparsifying attention scores to reduce computation.
    • Not every item needs to attend to every other item.
    • Sparsify attention for output and query.
• Impact: 7x reduction in latency and 30x increase in throughput per GPU.

Technical Details and Publications

• Technical reports and papers published to share lessons learned with the community.

Conclusion

The presentation details a comprehensive approach to building and deploying large language models for personalization and ranking at LinkedIn. Key takeaways include the importance of promptification, the necessity of distillation from large models, the impact of context length, and the effectiveness of techniques like pruning, quantization, and specification for productionizing these models. The results demonstrate significant improvements in cold start performance and generalization to new domains, highlighting the potential of LLMs to revolutionize recommendation systems.