Stanford CS231N | Spring 2025 | Lecture 3: Regularization and Optimization

Key Concepts

• Regularization: Techniques to prevent overfitting by penalizing model complexity.
• Optimization: Algorithms to find the best model parameters (weights) by minimizing the loss function.
• Loss Function: A function that quantifies how well a model's predictions match the training data. It has two components: data loss and regularization loss.
• Data Loss: Measures how well the model fits the training data.
• Regularization Loss: Penalizes complex models, encouraging simpler solutions that generalize better to unseen data.
• L1 Regularization: Adds the sum of the absolute values of the weights to the loss function, promoting sparsity (many weights close to zero).
• L2 Regularization: Adds the sum of the squares of the weights to the loss function, encouraging small, spread-out weights.
• Gradient Descent: An iterative optimization algorithm that updates model parameters in the direction of the negative gradient of the loss function.
• Stochastic Gradient Descent (SGD): A variant of gradient descent that uses a small subset (mini-batch) of the training data to estimate the gradient, making it faster for large datasets.
• Mini-batch: A small subset of the training data used in each iteration of SGD.
• Learning Rate: A hyperparameter that controls the step size in gradient descent.
• Momentum: A technique that adds a fraction of the previous update to the current update, helping to accelerate convergence and overcome local minima.
• RMSProp: An optimization algorithm that adapts the learning rate for each parameter based on the magnitude of its recent gradients.
• Adam: An optimization algorithm that combines the benefits of momentum and RMSProp, making it a popular choice for deep learning.
• AdamW: A variant of Adam that decouples weight decay (L2 regularization) from the adaptive learning rate, often leading to better performance.
• Learning Rate Scheduling: Techniques to adjust the learning rate during training, such as step decay, cosine decay, and linear warm-up.
• Linear Scaling Law: An empirical rule of thumb that suggests scaling the learning rate proportionally to the batch size.
• Second-Order Optimization: Optimization methods that use the Hessian matrix (second derivatives) to approximate the loss function as a quadratic, potentially leading to faster convergence but with higher computational cost.
• Hessian Matrix: A matrix of second-order partial derivatives of a function.

1. Recap of Image Classification and Challenges

• Image Classification Task: Mapping an image to a label from a predefined set of labels (e.g., cat, dog, bird).
• Semantic Gap: The difference between how humans perceive an image and how it's represented in a computer (pixel values).
• Challenges in Image Classification:
  • Illumination: Variations in lighting conditions.
  • Deformation: Objects changing shape.
  • Occlusion: Objects being partially hidden.
  • Background Clutter: Objects blending into the background.
  • Intraclass Variation: Objects within the same category looking different.
• Data-Driven Approaches: Using machine learning models trained on data to classify images, instead of hard-coded rules.
• K-Nearest Neighbors (KNN): A simple classification algorithm that assigns a label based on the majority class among the k nearest data points in the training set.
• Hyperparameter Tuning: Using a validation set to choose the best value for hyperparameters (e.g., k in KNN).
• Distance Metrics:
  • Manhattan Distance (L1): Sum of absolute differences between coordinates.
  • Euclidean Distance (L2): Straight-line distance between points.
• Linear Classifier: A model that uses a weight matrix and a bias vector to compute scores for each class.
• Linear Classifier - Algebraic Viewpoint: Each row of the weight matrix represents a class, and the dot product with the input vector yields the class score.
• Linear Classifier - Template Viewpoint: The learned weights can be reshaped into an image-like template for each class.
• Linear Classifier - Geometric Viewpoint: Each row of the weight matrix represents a line (decision boundary) in the input space, separating different classes.

2. Loss Function: Quantifying Model Performance

• Loss Function Definition: A function that measures how well a model's predictions match the ground truth labels.
• Data Loss: Measures the discrepancy between the model's predictions and the training data.
• Softmax Loss (Cross-Entropy Loss): A commonly used loss function for classification that penalizes low probabilities for the correct class and rewards high probabilities.
• Regularization Loss: A term added to the loss function to penalize model complexity and prevent overfitting.
• Regularization Goal: To make the model perform worse on the training data but better on unseen data (test set).
• Regularization Strength (Lambda): A hyperparameter that controls the strength of the regularization penalty.
• Occam's Razor: The principle of preferring simpler models over more complex ones when they explain the data equally well.

3. Types of Regularization

• L2 Regularization:
  • Calculates the sum of the squares of all weights in the model.
  • Encourages small, spread-out weights.
  • Allows for values close to zero.
• L1 Regularization:
  • Calculates the sum of the absolute values of all weights in the model.
  • Promotes sparsity, leading to many weights being exactly zero.
• Preference over Weights: Regularization allows expressing preferences for certain weight distributions (e.g., sparsity).
• Model Simplicity: Regularization can simplify the model, improving generalization to unseen data.
• Optimization Improvement: L2 regularization can improve the optimization process by making the loss function more convex.

4. Optimization: Finding the Best Model Parameters

• Optimization Goal: To find the model parameters (weights) that minimize the loss function.
• Loss Landscape: A visualization of the loss function as a function of the model parameters, where the goal is to find the lowest point.
• Random Search: A simple but inefficient optimization method that randomly samples model parameters and selects the best one.
• Following the Slope (Gradient Descent): An iterative optimization algorithm that updates model parameters in the direction of the negative gradient of the loss function.
• Derivative: The rate of change of a function with respect to one variable.
• Gradient: A vector of partial derivatives of a function with respect to multiple variables, indicating the direction of the steepest ascent.
• Negative Gradient: The direction of the steepest descent, used to update model parameters in gradient descent.
• Numerical Gradient: An approximation of the gradient calculated using a small step size.
• Analytic Gradient: The exact gradient calculated using calculus.
• Gradient Check: Comparing the numerical gradient and the analytic gradient to verify the correctness of the implementation.
• Gradient Descent Algorithm:
  1. Calculate the gradient of the loss function with respect to the weights.
  2. Update the weights by subtracting the gradient multiplied by the learning rate.
  3. Repeat steps 1 and 2 until convergence.
• Step Size: The magnitude of the update to the weights in each iteration of gradient descent.

5. Stochastic Gradient Descent (SGD)

• SGD Definition: A variant of gradient descent that uses a small subset (mini-batch) of the training data to estimate the gradient, making it faster for large datasets.
• Mini-batch: A small subset of the training data used in each iteration of SGD.
• Epoch: One complete pass through the entire training dataset.
• Problems with Gradient Descent and SGD:
  • Narrow Valleys: Slow convergence and oscillations due to large gradients in one direction and small gradients in another.
  • Local Minima: Getting stuck in a suboptimal solution where the gradient is zero.
  • Saddle Points: Points where the gradient is zero but the function is not a minimum, especially common in high-dimensional spaces.
  • Noisy Updates: Using a subset of the data leads to noisy gradient estimates.

6. Momentum: Accelerating Convergence

• Momentum Intuition: Simulating a ball rolling down a hill, where momentum helps to overcome local minima and accelerate convergence.
• Momentum Implementation: Adding a fraction of the previous update to the current update, effectively averaging the gradients over time.
• Velocity Term: A running average of the gradients, used to update the model parameters.
• Momentum Parameter (Rho): A hyperparameter that controls the influence of the previous update on the current update.
• Benefits of Momentum:
  • Overcoming local minima and saddle points.
  • Accelerating convergence in narrow valleys.
  • Averaging out noisy gradients.

7. RMSProp: Adaptive Learning Rates

• RMSProp Definition: An optimization algorithm that adapts the learning rate for each parameter based on the magnitude of its recent gradients.
• Gradient Squared Term: A running average of the squared gradients, used to scale the learning rate for each parameter.
• Decay Rate: A hyperparameter that controls the influence of the previous squared gradients on the current squared gradients.
• Benefits of RMSProp:
  • Adapting the learning rate for each parameter.
  • Taking larger steps in flatter regions of the loss landscape.
  • Taking smaller steps in steeper regions of the loss landscape.

8. Adam: Combining Momentum and RMSProp

• Adam Definition: An optimization algorithm that combines the benefits of momentum and RMSProp, making it a popular choice for deep learning.
• First Moment (Velocity): A running average of the gradients, similar to momentum.
• Second Moment (Gradient Squared): A running average of the squared gradients, similar to RMSProp.
• Bias Correction: A technique to correct for the initial bias in the first and second moment estimates.
• AdamW Definition: A variant of Adam that decouples weight decay (L2 regularization) from the adaptive learning rate, often leading to better performance.
• Benefits of Adam:
  • Adaptive learning rates for each parameter.
  • Momentum to accelerate convergence.
  • Robust to different learning rate scales.
• Adam vs. AdamW: Adam incorporates L2 regularization directly into the gradient calculation, while AdamW applies weight decay separately after the gradient update.

9. Learning Rate Scheduling

• Learning Rate Scheduling Definition: Techniques to adjust the learning rate during training.
• Step Decay: Reducing the learning rate by a fixed factor after a certain number of iterations.
• Cosine Decay: Gradually decreasing the learning rate following a cosine function.
• Linear Decay: Linearly decreasing the learning rate over time.
• Linear Warm-up: Gradually increasing the learning rate from a small value to the initial learning rate at the beginning of training.
• Benefits of Learning Rate Scheduling:
  • Improving convergence speed.
  • Avoiding oscillations.
  • Finding better local minima.
• Linear Scaling Law: An empirical rule of thumb that suggests scaling the learning rate proportionally to the batch size.

10. Second-Order Optimization

• Second-Order Optimization Definition: Optimization methods that use the Hessian matrix (second derivatives) to approximate the loss function as a quadratic.
• Hessian Matrix: A matrix of second-order partial derivatives of a function.
• Taylor Series Expansion: Approximating a function using its derivatives at a given point.
• Benefits of Second-Order Optimization:
  • Potentially faster convergence.
  • More accurate steps towards the minimum.
• Drawbacks of Second-Order Optimization:
  • High computational cost of calculating the Hessian matrix.
  • Large memory requirements for storing the Hessian matrix.
  • Not practical for large neural networks.

11. Conclusion and Recommendations

• Adam or AdamW: A good default choice for training models, especially when starting with a new problem.
• SGD with Momentum: Can sometimes outperform Adam, but requires more careful tuning of hyperparameters.
• Full-Batch Updates: For smaller datasets, consider second-order optimization methods.
• Neural Networks: Next lecture will cover neural networks, which are more complex models that can learn non-linear relationships in the data.
• Non-Linearity: Neural networks use non-linear activation functions (e.g., ReLU) to introduce non-linearity into the model.
• Data Transformation: Neural networks can transform the data into a space where it is linearly separable.

12. Additional Notes

• The choice of optimizer and learning rate schedule depends on the specific problem and model architecture.
• Experimentation and hyperparameter tuning are crucial for achieving good performance.
• Empirical results often guide the selection of optimization techniques.
• Understanding the underlying principles of optimization algorithms can help in troubleshooting and improving model performance.