Stanford CS231N | Spring 2025 | Lecture 9: Object Detection, Image Segmentation, Visualizing

Key Concepts

• Vision Transformers (ViTs): Image processing using transformer architecture by splitting images into patches (tokens) and adding positional embeddings.
• Self-Attention: Mechanism allowing the model to attend to different parts of the input sequence when processing it.
• Cross-Attention: Mechanism allowing the model to attend to different parts of two different input sequences when processing them.
• Positional Embedding: Adding information about the position of image patches to the tokens.
• Class Token: A learnable token added to the input sequence to represent the class probability in image classification.
• Layer Normalization: Normalizing the activations of each layer to stabilize training.
• RMSNorm (Root Mean Square Normalization): A simplified normalization technique that doesn't use the mean value.
• SwiGLU MLP: A gated nonlinearity used in MLPs to improve performance.
• Mixture of Experts: Using multiple parallel MLP layers (experts) and routing tokens to a subset of them.
• Semantic Segmentation: Assigning a label to every pixel in an image.
• Fully Convolutional Networks (FCNs): Neural networks that use only convolutional layers for semantic segmentation.
• Downsampling: Reducing the spatial resolution of feature maps.
• Upsampling: Increasing the spatial resolution of feature maps.
• Unpooling: A type of upsampling that uses nearest neighbor or bed of nails methods.
• Transposed Convolution: A learned upsampling method that reverses the convolution operation.
• Instance Segmentation: Identifying and segmenting each individual object instance in an image.
• Object Detection: Identifying and localizing objects in an image using bounding boxes.
• R-CNN (Regions with CNN features): An object detection algorithm that uses region proposals and CNNs.
• Region Proposal Network (RPN): A network that proposes regions of interest in an image.
• YOLO (You Only Look Once): A single-stage object detection algorithm that divides the image into a grid and predicts bounding boxes and class probabilities for each grid cell.
• Non-Maximum Suppression: A technique used to eliminate redundant bounding boxes.
• DETR (Detection Transformer): An object detection algorithm based on transformers.
• Object Queries: Trainable parameters in DETR that represent queries for objects in the image.
• Mask R-CNN: An extension of R-CNN that also predicts segmentation masks for each object instance.
• Saliency: Identifying the most important pixels in an image for a given task.
• Class Activation Map (CAM): A visualization technique that highlights the regions in an image that are most relevant to a particular class.
• Grad-CAM (Gradient-weighted Class Activation Map): An improved version of CAM that uses gradients to weight the feature maps.

Vision Transformers (ViTs)

• Transformers are used for image processing by splitting the image into patches, creating a sequence of tokens.
• Each patch is represented by a D-dimensional vector, often a linear projection of the reshaped patch.
• Positional embeddings are added to the tokens to retain spatial information, which is lost during patch splitting. These can be simple sequence numbers or 2D x,y coordinates.
• The sequence of tokens with positional embeddings is fed into a transformer encoder with self-attention, layer normalization, and MLP layers.
• For image classification, a class token is added to the input sequence. The output corresponding to this token is used to generate class probabilities.
• Alternatively, the output tokens can be pooled and projected into a class probability vector.
• Training is done using backpropagation and loss functions like binary cross-entropy or softmax loss.

Optimizations for Transformers

• Residual Connections and Layer Normalization: Placing layer normalization before self-attention and MLP layers in residual connections to preserve the identity function.
• RMSNorm: A normalization technique that uses the root mean square of features instead of the mean.
• SwiGLU MLP: Replacing simple MLPs with SwiGLU MLPs, which use gated nonlinearity to learn higher-dimensional nonlinearities. This involves adding a third weight matrix (W3) to the standard two (W1, W2).
• Mixture of Experts: Using multiple parallel MLP layers (experts) and routing tokens to a subset of them using a router. This increases the number of parameters and allows the model to learn more robust representations. A is the number of active experts out of E total experts.

Semantic Segmentation

• Semantic segmentation involves assigning a label to every pixel in an image.
• A naive approach of classifying each pixel individually is impractical due to the lack of context.
• Using patches around each pixel provides context but is computationally expensive.
• Fully Convolutional Networks (FCNs) take the entire image as input and output a segmentation map.
• FCNs use downsampling operations (pooling, strided convolution) to reduce the spatial resolution and then upsampling operations to restore the original resolution.
• The network is trained using a pixel-wise softmax loss function, requiring ground truth label maps.
• Upsampling Techniques:
  • Nearest Neighbor: Copying pixel values to the corresponding locations in the upsampled image.
  • Bed of Nails: Placing the original pixel value in one corner of the upsampled region and filling the rest with zeros.
  • Max Unpooling: Saving the locations of max values during max pooling and using those locations during upsampling.
  • Transposed Convolution: Applying a convolution operation on the transposed version of the input to generate a larger output.

U-Net

• U-Net is a widely used FCN architecture for semantic segmentation, especially in medical applications.
• It has a U-shaped architecture with a downsampling (encoding) path and an upsampling (decoding) path.
• A key feature of U-Net is the use of skip connections, where feature maps from the encoding path are copied to the corresponding layers in the decoding path to preserve spatial information.

Instance Segmentation and Object Detection

• Instance segmentation extends semantic segmentation by identifying and segmenting each individual object instance.
• Object detection involves identifying and localizing objects in an image using bounding boxes.
• A simple approach to object detection is to classify the entire image and predict bounding box coordinates, but this is not scalable for multiple objects.
• R-CNN:
  • Uses region proposals to identify potential object locations.
  • Extracts features from each region proposal using a CNN.
  • Classifies the regions and refines the bounding box coordinates.
  • Slow due to running a full CNN for each region proposal.
• Fast R-CNN:
  • Runs a CNN on the entire image to generate a feature map.
  • Extracts features from the feature map corresponding to the region proposals.
  • Classifies the regions and refines the bounding box coordinates.
• Region Proposal Network (RPN):
  • A network that proposes regions of interest in an image.
  • Refines the regions through layers of convolution.
  • Outputs the top K regions with the highest probability of containing an object.
• YOLO:
  • A single-stage object detection algorithm that divides the image into a grid.
  • For each grid cell, it predicts bounding boxes, object probabilities, and class probabilities.
  • Uses non-maximum suppression to eliminate redundant bounding boxes.
• DETR:
  • An object detection algorithm based on transformers.
  • Turns the image into patches and passes them through a CNN to create tokens.
  • Adds positional encoding to the tokens.
  • Uses a transformer encoder to generate output tokens.
  • Uses a transformer decoder with object queries to generate bounding boxes and class labels.
  • Object queries are trainable parameters that represent queries for objects in the image.
• Mask R-CNN:
  • An extension of R-CNN that also predicts segmentation masks for each object instance.
  • Adds a mask prediction branch to the R-CNN architecture.

Visualization and Understanding

• Visualizing neural networks helps understand what the networks are learning.
• Filter Visualization: Visualizing the weights of filters in the early layers of CNNs to see the patterns they are learning.
• Saliency Maps: Identifying the most important pixels in an image for a given task by calculating the gradient of the class score with respect to the pixel values.
• Class Activation Maps (CAM): Highlighting the regions in an image that are most relevant to a particular class by tracing back class predictions to the feature maps.
• Grad-CAM: An improved version of CAM that uses gradients to weight the feature maps, allowing it to be applied to a wider range of CNN architectures.
• Transformer Visualization: Visualizing the attention weights in transformers to understand which parts of the input are most relevant to the output.

Conclusion

The lecture covered core computer vision tasks, including image classification, semantic segmentation, instance segmentation, and object detection. It discussed various algorithms for each task, from traditional CNN-based approaches to more recent transformer-based methods. The lecture also highlighted the importance of visualization and understanding in computer vision, covering techniques like saliency maps, CAM, and Grad-CAM. The key takeaway is that while modern computer vision is advancing rapidly, understanding the fundamentals of these core tasks and visualization techniques is crucial for developing and deploying effective computer vision solutions in various applications.