Stanford CS231N | Spring 2025 | Lecture 5: Image Classification with CNNs

Image Classification with CNNs

Key Concepts:

• Image Classification
• Linear Classifiers
• Loss Functions (Softmax, SVM)
• Optimization (SGD, Momentum, RMSProp, Adam)
• Neural Networks
• Computational Graphs
• Backpropagation
• Convolutional Neural Networks (CNNs)
• Convolution Layer
• Pooling Layer
• Feature Representations (Color Histograms, HOG)
• Translation Equivariance
• Receptive Field
• Stride
• Padding

Recap of Deep Learning Basics

The first segment of the class covered deep learning fundamentals, including:

• Image Classification with Linear Classifiers: Used as a toy problem to illustrate the basic input-output tensor structure of deep learning problems. Images (grids of pixel values) are classified into human-understandable categories. The output is a score for each category, indicating the likelihood of the image belonging to that category. A weight matrix (W) is used to multiply against the image pixels to predict these scores.
• Loss Functions: Used to evaluate how well a weight matrix (W) performs on a given dataset. Examples include Softmax loss and SVM loss.
• Optimization: The process of searching for the best weight matrix (W) that minimizes the loss function. Algorithms like Stochastic Gradient Descent (SGD), often with momentum, RMSProp, and Adam, are used. The Adam optimizer was awarded the Test of Time Award at ICLR 2025.

Limitations of Linear Classifiers and the Rise of Neural Networks

Linear classifiers have limitations:

• Visual Viewpoint: Linear classifiers summarize all knowledge about a category into a single template, which is insufficient to capture variations in appearance (e.g., cars can be different colors).
• Geometric Viewpoint: Linear classifiers carve up the data space with hyperplanes, which may not be suitable for non-linearly separable data.

Neural networks address these limitations by stacking multiple weight matrices with non-linearities in between, creating a more powerful mechanism for predicting scores.

Computational Graphs and Backpropagation

• Computational Graphs: Data structures that organize the computation of a neural network. Each node represents a functional primitive (e.g., matrix multiply, ReLU). Data flows forward through the graph, and gradients are computed backward using backpropagation.
• Backpropagation: An algorithm that computes gradients of the loss with respect to all parameters in the model. It converts the global problem of gradient computation into a local problem, where each node only needs to know how to compute outputs from inputs and local gradients.

General Recipe for Solving Problems with Deep Learning

1. Encode the problem as tensors.
2. Write down a computational graph to compute output tensors from input tensors.
3. Collect a dataset of input-output tensors.
4. Write down a loss function.
5. Optimize the loss function using gradient descent and backpropagation.

Perceiving and Understanding the Visual World: Introduction to CNNs

The second part of the class focuses on applying deep learning to computer vision problems. Convolutional Neural Networks (CNNs) are introduced as a specialized architecture for processing images.

Feature Representations vs. End-to-End Learning

• Feature Representations: Before neural networks, computer vision relied on hand-engineered feature representations (e.g., color histograms, Histogram of Oriented Gradients - HOG) to convert raw pixel values into meaningful representations.
  • Color Histograms: Capture the distribution of colors in an image, discarding spatial information.
  • HOG (Histogram of Oriented Gradients): Capture the structure of an image by looking at the local direction of edges, discarding color information.
• End-to-End Neural Networks: Learn features directly from raw pixel values using gradient descent, eliminating the need for hand-engineered features. This approach has proven more effective in recent years.

Designing Neural Network Architectures for Images

Instead of designing specific feature extractors, the human designer defines a category of functions through the structure of the computational graph and the sequence of operators.

Deficiencies of Fully Connected Networks for Images

Fully connected networks flatten images into vectors, destroying the spatial structure. CNNs address this by using convolutional layers and pooling layers that respect the two-dimensional structure of images.

Convolutional Neural Networks (CNNs)

CNNs are neural network architectures built from linear layers, non-linearities, convolution layers, and pooling layers. They typically consist of:

1. Body: An interleaved sequence of convolution layers, pooling layers, and non-linearities that extract useful features.
2. Head: Fully connected layers (multi-layer perceptron) that classify the features extracted by the convolutional portion.

The entire system is trained end-to-end via gradient descent.

Historical Context

• LeNet (1998): Early CNN architecture for digit classification.
• AlexNet (2012): A larger CNN architecture that demonstrated the power of deep learning for computer vision, trained on GPUs with more data (ImageNet).
• Dominance of CNNs (2012-2020): CNNs became the dominant approach for almost every computer vision problem, including detection, segmentation, and image captioning.
• Rise of Transformers (2020-Present): Transformers, initially developed for natural language processing, have surpassed CNNs in many computer vision tasks due to their ability to scale to more data and compute.

Convolution Layer

The convolution layer performs template matching between learned filters and small subsets of the input image.

• Instead of stretching the image into a vector, the spatial structure is maintained (3D tensor: channels x width x height).
• Filters are small sub-images (e.g., 5x5 pixels) with the same number of channels as the input.
• The filter is slid across the image, computing a dot product (template matching score) at each location.
• Multiple filters are used to create multiple response planes, which are stacked to form the output.
• A learnable bias vector is often added.

General Formulation:

• Input: N x C_in x H x W (N images, C_in channels, H height, W width)
• Filters: C_out x C_in x K_w x K_h (C_out filters, C_in channels, K_w kernel width, K_h kernel height)
• Output: N x C_out x H' x W' (N images, C_out channels, H' output height, W' output width)

Visualizing Convolutional Filters

• First-layer filters often learn to detect colors (especially opposing colors) and edges.
• Higher layers learn larger spatial structures.

Spatial Arrangement: Convolution Arithmetic

• Without Padding: Output size = (W - K + 1)
• With Padding: Output size = (W - K + 2P + 1)
• Common Practice: Set padding P = (K - 1) / 2 to maintain the same spatial size.

Receptive Fields

The receptive field of a convolution is the region of the original input image that influences one activation in the network. Stacking multiple convolutions increases the effective receptive field.

Strided Convolutions

Strided convolutions skip pixels, downsampling the image and increasing the receptive field more quickly.

• Output Size with Stride: (W - K + 2P) / S + 1

Pooling Layer

Pooling layers downsample feature maps, reducing computation and increasing the receptive field.

• Max Pooling: Divides the input into non-overlapping regions and takes the maximum value in each region.
• Average Pooling: Takes the average value in each region.

Translation Equivariance

Convolution and pooling layers exhibit translation equivariance, meaning that translating the input image and then applying the operator yields the same result as applying the operator and then translating the output. This property ensures that features depend on the content of the image, not its absolute location.

Conclusion

CNNs are powerful architectures for processing images, built from convolution and pooling layers. They learn features directly from data and respect the spatial structure of images. While transformers have become more popular in recent years, CNNs remain a valuable tool in computer vision.