Stanford CS25: V5 I Overview of Transformers

CS25 Transformers Class: Lecture 1 Summary

Key Concepts:

• Transformers
• Pre-training
• Post-training
• Data Strategies (small-scale, large-scale)
• Chain of Thought (CoT)
• Tree of Thought (ToT)
• Program of Thought (PoT)
• Reinforcement Learning with Human Feedback (RLHF)
• Direct Preference Optimization (DPO)
• Reinforcement Learning with AI Feedback (RLAIF)
• Group Relative Policy Optimization (GRPO)
• Continual Learning
• Vision Transformers (ViT)
• Functional Magnetic Resonance Imaging (fMRI)
• Self-Improving AI Agents

1. Introduction to CS25 and Transformers

• The CS25 class aims to provide a comprehensive understanding of transformers and their applications in AI.
• The course features lectures from leading experts in industry and academia.
• This lecture covers the basics of transformers, pre-training, data strategies, post-training, applications, and remaining challenges.

2. Course Logistics

• A new website, cs25.stanford.edu, will host updates and the speaker lineup.
• The Zoom link will be shared for those not affiliated with Stanford or on the waitlist.
• Course takeaways include understanding transformers, exposure to new research, innovative methods, key limitations, open problems, and the future of AI.

3. Introduction to Transformers and Attention

• Word Embeddings: Converting words into dense vectors to capture semantic similarity (e.g., cat and dog are more similar than cat and car).
  • Classical methods: Word2Vec, FastText.
  • Contextual embeddings are now standard to account for context.
• Self-Attention: Learning what to focus on for a given token using query, key, and value matrices.
  • Analogy: Library search (query = topic, key = book summary, value = book content).
• Positional Encodings/Embeddings: Adding order to the sequence, as transformers lack inherent order understanding.
  • Methods: Sinusoids or simple indexing (0, 1, 2...).
• Multi-Head Attention: Using multiple attention heads to capture diverse relationships in sequences.
• Transformers have become prevalent in various fields, including LLMs (GPT-4, etc.), vision, speech, biology, and video.
• LLMs are scaled-up versions of attention and the transformer architecture, trained on large amounts of text data using next-token prediction.
• Emergent abilities appear as models scale up.
• Disadvantages: High computational costs and carbon emissions.
• Larger models generalize well to many tasks and offer few-shot/zero-shot learning capabilities.

4. Pre-training and Data Strategies

• Pre-training involves training a neural network from scratch on a large dataset to gain general capabilities.
• Data is the fundamental fuel for learning.
• Smart data strategies are crucial for maximizing the effectiveness of pre-training.

4.1. Small-Scale Data Strategies: Childlike Datasets

• Investigates the effectiveness of small, childlike datasets for language learning.
• Humans learn efficiently through continuous learning, goal-based interaction, multimodal data, and structured learning.
• Focuses on data differences between human children and LLMs.
• Why small models? Improved training efficiency, interpretability, control, open-source availability, and understanding of human learning.
• Research Question: Is child-directed speech effective training data for language models?
• Hypotheses:
  • Data differences (human interaction vs. internet data).
  • Learning algorithm differences (human brain vs. LLMs).
  • Curriculum differences (simple to complex data).
• Experiment: Trained small GPT2 and RoBERTa models on five datasets:
  • Child: Natural conversation data with children.
  • Tiny Dialogues: Synthetically generated, grammatical, and curricularized conversation data using GPT-4.
  • BabyLM: Diverse mixture of data (Reddit, Wikipedia, etc.).
  • Wikipedia:
  • Open Subtitles: Movie and TV transcriptions.
• Tiny Dialogues: Inspired by conversations leading to learning, ethics, and diversity.
• Curriculum Experiment: Ordered data by ascending age, descending age, and random shuffle.
• Evaluation Metrics:
  • Grammatical and syntactic knowledge.
  • Word similarity (semantic knowledge).
• Results:
  • Childlike data performed worse than heterogeneous data (BabyLM).
  • Tiny Dialogues outperformed the natural conversation data (Child).
  • Curriculum order had negligible impact on performance.
• Conclusion: Other aspects of children's learning (multimodal information, different learning algorithms) may be responsible for their efficient language learning.

4.2. Large-Scale Data Strategies: Two-Phase Pre-training

• Investigates smart data strategies for training large models on billions/trillions of tokens.
• Project with Nvidia: "Maximize Your Data's Potential: Enhancing LLM Accuracy with Two-Phase Pre-training."
• Formalizes and evaluates two-phase pre-training, showing empirical improvements over continuous training.
• Performs fine-grained analysis of data blending for the two phases.
• Prototyping blends on smaller token counts before scaling up.
• Two-Phase Approach:
  • Phase 1: General data for broad learning (diverse data).
  • Phase 2: High-quality, domain-specific data (e.g., math).
  • Balance between quality and diversity in both phases to avoid overfitting.
• Results:
  • Two-phase pre-training outperformed single-phase training.
  • Two-phase pre-training scaled well with model and data size.
  • Performance improved up to 40% of phase two duration, then diminishing returns due to overfitting.
• Conclusion: Well-structured two-phase pre-training with careful data selection is essential for optimizing LLM performance.
• Overall Takeaway: Data effectiveness depends on quality, ordering, structure, and usage, not just quantity.

5. Post-training Techniques

• Adapting pre-trained models to specific tasks and domains.
• Major strategies: Fine-tuning (RLHF), prompt-based methods, retrieval-based methods (RAG).

5.1. Chain of Thought (CoT) and Extensions

• Chain of Thought (CoT): Prompting technique to think step-by-step, similar to human reasoning.
  • Shows intermediate steps for guidance.
  • Provides interpretable insights into model behavior.
• Tree of Thought (ToT): Considers multiple reasoning trajectories and uses self-evaluation to decide on the final output.
• Program of Thought (PoT): Generates code as intermediate reasoning steps, offloading problem-solving to a code interpreter.
• Problem Decomposition:
  • Socratic Questioning: Uses a self-questioning module to propose sub-problems.
  • Computational Graphs: Formulates compositional tasks as computation graphs.
• Key Takeaway: Transformers can solve compositional tasks by reducing reasoning into subgraphs.

5.2. Reinforcement Learning and Feedback Mechanisms

• Reinforcement Learning with Human Feedback (RLHF): Trains a reward model directly from human feedback.
  • Humans rate pairs of responses, and the reward model is trained using a reinforcement learning algorithm (e.g., PPO).
• Direct Preference Optimization (DPO): Directly trains the model to prefer outputs that humans rank higher, more efficient than RLHF.
• Reinforcement Learning with AI Feedback (RLAIF): Replaces human feedback with AI feedback, using a good LLM to provide preference judgments.
  • Less costly than RLHF.
  • Depends on the accuracy of the LLM providing preferences.
• Group Relative Policy Optimization (GRPO): Ranks a group of responses in different orders, providing richer feedback.
  • Stabilizes training and improves reasoning.
• Conmani Optimization (KTO): Modifies the loss function to account for human biases, such as loss aversion.
  • Encourages AI to avoid negative outcomes.
• Personalizing RLHF with Variational Preference Learning: Introduces a latent variable for each user preference profile (e.g., demographics).
  • Trains reward models conditioned on these latent factors.
  • Enables pluralistic alignment, improving reward accuracy for specific demographics.

5.3. Self-Improving AI Agents

• AI Agent: A system that perceives the environment, makes decisions, and takes actions towards a specific goal.
  • Components: Goal-directed, decision-making, iterative, memory, tool use, learning, and adaptation.
• Self-Improvement: Models reflect on their own outputs, leading to iterative improvements.
  • Steps: Reflection, explanation, evaluation, and simulation.
• Techniques:
  • Refinement: Iterative prompting where an LLM critiques and improves its own outputs.
  • Self-Reflection: A model learns from past mistakes and adjusts future responses.
  • ReAct: Combines reasoning with external actions (API calls, retrievals).
  • Language Agent Tree Search (LaTS): Extends ReAct to incorporate multiple planning pathways, using Monte Carlo tree search.

6. Applications of Transformers Beyond Language

6.1. Vision Transformers (ViT)

• ViT splits an image into patches, embeds them to form a sequence, and passes it through a transformer.
• Transformers bring fewer inductive biases than CNNs when trained on large datasets.
• CLIP: Uses ViTs for image encoders, trained through contrastive learning to align image and text representations.
• ViTs are applied to vision-language models like GPT-4o.

6.2. Transformers in Neuroscience

• Functional Magnetic Resonance Imaging (fMRI) captures brain activity.
• fMRI data is high-dimensional, requiring averaging across regions of interest (ROIs).
• Traditional tools used linear pair-wise correlation maps.
• Computer vision techniques allow applying larger models to fMRI data.
• Self-Supervised Pre-training: Masking ROIs and having the transformer predict the masked portion.
• Cross-Attention: Applying attention between two different sequences (e.g., brain regions).
• Results: The model can predict brain activity and identify connections between brain networks.
• The model can be used to predict diseases like Parkinson's with higher accuracy than traditional methods.

7. The Future of Transformers and AI

• Transformers enable more applications across industries.
• Examples: Generalist agents, longer video understanding, domain-specific foundation models, personalized education, advanced healthcare, environmental monitoring, multilingual communication, interactive gaming.
• Missing Elements:
  • Reduced computational complexity.
  • Enhanced human controllability.
  • Alignment with the human brain.
  • Adapted learning and generalization.
  • Multisensory/multimodal embodiment.
  • Infinite and external memory.
  • Infinite self-improvement.
  • Complete autonomy.
  • Emotional intelligence.
  • Ethical reasoning and value alignment.

8. Remaining Weaknesses and Challenges

• Efficiency: Miniaturizing LLMs for smaller devices.
• Interpretability: Understanding the decision-making process of LLMs.
  • Mechanistic interpretability: Understanding how individual components contribute to decisions.
• Diminishing Returns with Scaling: Larger models on more data are not the only solution.
  • Pre-training performance is likely saturating.
  • Too much post-training can lead to catastrophic forgetting.
• Breaking Through Scaling Limits:
  • New architectures (Mamba, state space machines).
  • High-quality data and smart data strategies.
  • Improved training procedures.
• Bringing Capabilities to Smaller Models.
• Theoretical and Interpretability Research: Cognitive science and neuroscience-inspired work.
• Continual/Lifelong Learning: AI systems that can continuously improve after deployment.
  • Updating the brain (weights) of the model continuously.
  • Model editing: Targeting specific nodes for updates.
  • Challenges: Updating skills/capabilities, propagating changes to related facts.
• Continual Learning Techniques:
  • Memit: Mass editing of factual knowledge.
  • Chem: Continually evolving from mistakes.
  • Lifelong Mixture of Experts: Continually adding new experts.
  • CLOB: Continual task learning using prompting.
  • Progressive Prompts: Learning soft prompt vectors for each task.

9. Conclusion

• The lecture provided an overview of transformers, pre-training, data strategies, post-training, applications, and remaining challenges.
• Key areas for future research include continual learning, data efficiency, and interpretability.