Stanford CS231N | Spring 2025 | Lecture 2: Image Classification with Linear Classifiers

Image Classification: Linear Classifiers and Data-Driven Approaches

Key Concepts:

• Image Classification: Assigning a predefined label to an image.
• Data-Driven Approach: Using machine learning algorithms to learn from data instead of hard-coding rules.
• Nearest Neighbor Classifier: Classifying an image based on the label of the most similar image in the training data.
• Linear Classifier: A parametric approach that learns weights to map input images to output class scores.
• Hyperparameters: Parameters that are set before training and control the learning process (e.g., k in k-NN).
• Loss Function: A function that quantifies how well a classifier is performing.
• Softmax Classifier: A linear classifier that outputs probabilities for each class.
• Cross-Entropy Loss: A common loss function used with softmax classifiers.

1. Introduction to Image Classification

• Image classification is a core task in computer vision, used as a benchmark for algorithms.
• The goal is to assign one of a set of predefined labels (e.g., dog, cat, truck) to a given image.
• Images are represented as tensors of data, typically with pixel values between 0 and 255 (8-bit).
• A colored image with a resolution of 800x600 has a tensor of 800x600x3 (RGB channels).
• The "semantic gap" refers to the difference between human perception and how a machine perceives an image.

2. Challenges in Image Classification

• Viewpoint Variation: Changing the camera angle alters pixel values even if the object remains still.
• Illumination: RGB values depend on surface material, color, and light source, causing variations under different lighting conditions.
• Background Clutter: Background objects can interfere with object recognition.
• Scale Variation: The size of the object in the image can vary.
• Occlusion: Objects can be partially hidden.
• Deformation: Objects can be deformed (e.g., cats contorting).
• Intraclass Variation: Objects within the same class can have different sizes, colors, patterns, and breeds.
• Context: The surrounding environment can influence object recognition.

3. Data-Driven Approach to Image Classification

• Traditional approaches based on hard-coded rules and edge detectors are difficult to scale and require significant manual effort.
• The data-driven approach involves a three-step process:
  1. Collect a dataset: Gather images and their corresponding labels.
  2. Train a classifier: Use machine learning algorithms to build a model that associates images with labels.
  3. Evaluate the classifier: Test the model on new images and predict their labels.

4. Nearest Neighbor Classifier

• The simplest form of image classification.
• Training: Memorize all training data and labels (O(1) complexity).
• Prediction: Find the most similar training image to the input image and output its label (O(n) complexity, where n is the number of training samples).
• Requires a distance function to measure similarity between images.
• L1 Distance (Manhattan Distance): Sum of absolute pixel differences between two images.
• Implementation in Python using NumPy involves calculating distances, finding the minimum distance, and outputting the corresponding label.

5. K-Nearest Neighbor Classifier

• An extension of the nearest neighbor classifier that considers the k nearest neighbors.
• Uses majority voting to determine the label for a given input image.
• Can be more robust to outliers and noise.
• Problem: Can create "white regions" where there is no clear majority vote.

6. Hyperparameter Tuning

• Hyperparameters (e.g., k in k-NN, distance function) are data set-dependent and problem-dependent.
• Incorrect Approach 1: Choosing hyperparameters that work best for the training data (leads to overfitting).
• Incorrect Approach 2: Choosing hyperparameters that work best for the testing data (cheating, doesn't generalize).
• Correct Approach 1: Splitting the training data into a training set and a validation set. Train on the training set and optimize hyperparameters on the validation set.
• Correct Approach 2: Using cross-validation:
  • Split the training data into k folds.
  • Iteratively use each fold as the validation set and the remaining folds as the training set.
  • Average the accuracies across all folds to evaluate the hyperparameter setting.
  • Apply the best hyperparameter setting to the testing set.

7. Distance Metrics: L1 vs. L2

• L1 Distance (Manhattan Distance): Sum of absolute pixel differences. Visualized as a square.
• L2 Distance (Euclidean Distance): Square root of the sum of squared pixel differences. Visualized as a circle.
• L1 Sensitivity: L1 is sensitive to the orientation of the feature axes. If the features are rotated, the distance function changes.
• L2 Invariance: L2 is invariant to the orientation of the feature axes.
• Feature Preservation: L1 is better when features are meaningful and their information should be preserved.
• Arbitrary Features: L2 is better when features are more arbitrary.

8. CIFAR-10 Dataset and Nearest Neighbor Results

• CIFAR-10: A dataset with 10 classes, each with a number of training and testing images.
• Nearest neighbor achieves around 28% accuracy on CIFAR-10, which is better than random guessing (10%).
• Pixel-based distances are not the best choice, as images with similar pixel values may not be semantically similar.

9. Linear Classifiers

• A parametric approach that learns weights to map input images to output class scores.
• The function f(x, W) = Wx + b maps an input image x to a vector of scores, where W is the weight matrix and b is the bias vector.
• Linear classifiers are building blocks for neural networks.
• The input image is flattened into a vector (e.g., a 32x32x3 image becomes a 3072x1 vector).
• The weight matrix W is of size (number of classes) x (number of features).
• The bias vector b is of size (number of classes) x 1.

10. Visual and Geometric Interpretation of Linear Classifiers

• Each row of the weight matrix W can be interpreted as a template for a specific class.
• Linear classifiers find lines (in 2D) or hyperplanes (in higher dimensions) that separate different classes.
• The bias term allows the decision boundaries to be shifted away from the origin.

11. Limitations of Linear Classifiers

• Cannot classify data that is not linearly separable (e.g., XOR problem, concentric circles).

12. Loss Function and Optimization

• A loss function quantifies how well a classifier is performing.
• The goal is to find the weights W that minimize the loss function.
• Softmax Classifier:
  • Converts the output scores into probabilities using the softmax function: P(Y=i|X=xi) = exp(si) / sum(exp(sj)), where si is the score for class i.
  • The softmax function exponentiates the scores and normalizes them to create a probability distribution.
  • The probabilities sum to 1.
  • The unnormalized log probabilities are called logits.
• Cross-Entropy Loss:
  • The loss function is defined as the negative log likelihood of the correct class: Li = -log(P(Y=yi|X=xi)).
  • This is equivalent to minimizing the Kullback-Leibler (KL) divergence between the predicted and true probability distributions.
  • It is also equivalent to the cross-entropy between the predicted and true distributions.

13. Properties of the Cross-Entropy Loss

• The minimum value of the loss function is 0 (when the probability of the correct class is 1).
• The maximum value of the loss function is infinity (when the probability of the correct class is 0).
• When the weights are initialized randomly, the probabilities of each class are approximately equal (1/C, where C is the number of classes).
• In this case, the initial loss is approximately log(C).

14. Conclusion

• The lecture covered image classification, data-driven approaches, nearest neighbor classifiers, linear classifiers, hyperparameters, distance metrics, and loss functions.
• Linear classifiers are a fundamental building block for neural networks.
• The next lecture will cover optimization techniques for finding the weights that minimize the loss function.