Stanford CS231N Deep Learning for Computer Vision | Spring 2025 | Lecture 6: CNN Architectures

Key Concepts:

• Convolutional Neural Networks (CNNs)
• Convolutional Layers
• Pooling Layers (Max Pooling, Average Pooling)
• Fully Connected Layers
• Normalization Layers (LayerNorm, BatchNorm, InstanceNorm, GroupNorm)
• Dropout (Regularization)
• Activation Functions (Sigmoid, ReLU, GELU, SELU)
• Receptive Field
• Residual Networks (ResNets)
• Weight Initialization (Kaiming Initialization)
• Data Pre-processing (Mean Subtraction, Standard Deviation Normalization)
• Data Augmentation (Horizontal Flipping, Resizing and Cropping, Color Jitter, Occlusion)
• Transfer Learning (Fine-tuning)
• Hyperparameter Optimization (Random Search)

1. Building CNNs: Layers and Architectures

• Convolutional Layers: Filters slide across the input image, calculating a score at each location by taking the dot product of the filter values with the image values and adding a bias term. The number of filters determines the depth of the output activation map.
• Pooling Layers: Reduce the spatial dimensions (height and width) of the image. Max pooling takes the maximum value within a filter, while average pooling takes the average.
• Fully Connected Layers: Matrix multiplication followed by an activation function.
• Normalization Layers: Normalize input data to a unit Gaussian (mean 0, standard deviation 1) and then scale and shift it using learned parameters.
  • LayerNorm: Calculates the mean and standard deviation for each sample separately across all channels, heights, and widths. Commonly used in transformers.
  • BatchNorm: Calculates the mean and standard deviation per channel across the mini-batch.
  • InstanceNorm: More granular normalization.
  • GroupNorm: Normalizes using different subsets of the input data.
• Dropout: A regularization technique where a fixed percentage of the outputs from a layer are randomly zeroed out during training. This forces the network to have redundant representations and generalize better. At test time, dropout is removed, and the output activations are scaled by the dropout probability.
• Activation Functions: Introduce non-linearities to the model.
  • Sigmoid: Historically used, but suffers from vanishing gradients (small gradients for very negative and very positive values).
  • ReLU: More popular due to a gradient of 1 for positive inputs and 0 for negative inputs. Computationally cheaper than sigmoid.
  • GELU: A smoother activation function that avoids the non-smooth jump in the derivative of ReLU. Used in transformers.
    • Calculates the Gaussian Error Linear Unit, which is the cumulative distribution function of a Gaussian normal.
  • SELU: Similar to GELU.

2. CNN Architectures: Examples and Design Principles

• AlexNet: An early CNN-based paper that performed well on ImageNet using GPUs.
• VGG: A standard architecture from the 2010s, consisting of 3x3 convolutional layers with stride 1 and padding 1, followed by max pooling layers and fully connected layers.
• ResNets (Residual Networks): Introduced residual connections to address the problem of deeper networks having higher training and test errors.
  • Residual Block: Copies the input value (x) over past the convolutional layers and adds it to the output of the convolutional stack (F(x)). This allows the model to easily learn an identity function by setting the convolutional filters to zero.
  • ResNets periodically double the number of filters and downsample the spatial dimension.

3. Weight Initialization

• Proper weight initialization is crucial to avoid issues like vanishing or exploding activations.
• Kaiming Initialization: Initializes weights to the square root of 2 over the input dimension size. This helps maintain a relatively constant mean and standard deviation throughout the layers.

4. Training CNNs: Data Pre-processing and Augmentation

• Data Pre-processing:
  • Calculate the average red, green, and blue pixel values and standard deviations for the training dataset.
  • Subtract the mean and divide by the standard deviation for each input image.
  • ImageNet means and standard deviations are commonly used.
• Data Augmentation: Apply transformations to the image to make it look different but still recognizable for the category class.
  • Horizontal Flipping: Useful for symmetrical objects.
  • Resizing and Cropping: Randomly crop and resize images to a fixed size.
    • Pick the length of the short side of your image.
    • Resize the short side to L and then crop a random patch of 224 by 224 pixels from that image.
  • Test Time Augmentation: Average predictions from multiple crops and resizes of the test image.
  • Color Jitter: Randomize contrast and brightness.
  • Occlusion: Cover parts of the image with black or gray boxes.

5. Transfer Learning

• If you don't have a lot of data, you can use transfer learning.
• Linear Classifier Strategy: Freeze all layers of a pre-trained model (e.g., trained on ImageNet) and replace the final layer with the number of classes in your data set. Only train the final layer.
• Fine-tuning: Train the whole model, but initialize it with weights pre-trained on a large dataset.

6. Hyperparameter Optimization

• Overfit on a Small Sample: Start by trying to overfit on a single data point to ensure the model can memorize it.
• Coarse Grid Search: Try a coarse grid of hyperparameters, focusing on the learning rate.
• Accuracy Curves: Monitor training and validation accuracy to identify overfitting or underfitting.
• Random Search: Randomly sample hyperparameters from a defined range, which is more effective than grid search.

7. Conclusion

The lecture provides a comprehensive overview of training convolutional neural networks, covering CNN architectures, layers, weight initialization, data pre-processing, data augmentation, transfer learning, and hyperparameter optimization. The key takeaway is that building and training CNNs involves a combination of architectural design, careful initialization, data manipulation, and systematic hyperparameter tuning.