Stanford CS231N | Spring 2025 | Lecture 8: Attention and Transformers

Key Concepts

• Recurrent Neural Networks (RNNs): Neural network architecture for processing sequences.
• Attention: A neural network primitive that operates on sets of vectors, allowing the model to focus on relevant parts of the input.
• Transformers: A neural network architecture that uses self-attention as its core mechanism.
• Encoder-Decoder Architecture: A common structure in sequence-to-sequence models, where an encoder processes the input sequence and a decoder generates the output sequence.
• Context Vector: A vector that summarizes the information in the input sequence, used by the decoder to generate the output sequence.
• Self-Attention: A type of attention mechanism where the query, key, and value vectors are derived from the same input sequence.
• Cross-Attention: A type of attention mechanism where the query and key/value vectors come from different sources.
• Multi-Headed Attention: Running multiple independent attention mechanisms in parallel to capture different aspects of the input.
• Positional Embeddings: Additional data concatenated to input vectors to provide information about their position in the sequence.
• Masked Self-Attention: A technique to prevent the model from attending to future tokens in the sequence, often used in language modeling.
• Permutation Equivariance: A property of self-attention where shuffling the inputs results in the same outputs, but shuffled in the same way.
• Layer Normalization: A technique to normalize the activations of a layer, improving training stability.
• Residual Connections: Adding the input of a layer to its output, helping to alleviate the vanishing gradient problem and improve training.
• Multi-Layer Perceptron (MLP) / Feed Forward Network (FFN): A simple neural network with one or more fully connected layers, used in transformers to process vectors independently.

Recurrent Neural Networks and Sequence-to-Sequence Problems

• RNNs are designed for processing sequences of data, enabling solutions to problems like image captioning (one-to-many) and video classification (many-to-one).
• Sequence-to-sequence problems, such as translation, involve inputting one sequence (e.g., English) and outputting another (e.g., Italian), where the number and order of words may differ.
• The traditional RNN approach uses an encoder-decoder architecture.
  • The encoder RNN processes the input sequence and summarizes it into a context vector.
  • The decoder RNN uses the context vector to generate the output sequence.
• Bottleneck Problem: A limitation of the traditional RNN approach is the context vector, which becomes a bottleneck for long sequences as it needs to encode all information into a fixed-length vector.

Attention Mechanism: Overcoming the Bottleneck

• Motivation: To avoid the bottleneck, the attention mechanism allows the decoder to "look back" at the entire input sequence at each time step.
• Process:
  1. Alignment Scores: Compute alignment scores between the decoder hidden state (query) and each encoder hidden state (keys). A simple linear layer f_att can be used: f_att(concatenate(s_t, h_i)) -> scalar.
  2. Softmax: Apply a softmax function to the alignment scores to obtain a probability distribution over the input tokens.
  3. Context Vector: Compute a weighted sum of the encoder hidden states, using the attention scores as weights. This creates a context vector c_t that summarizes the relevant parts of the input sequence for the current decoder step.
  4. Decoder Update: The decoder RNN uses the context vector, the previous output token, and the previous hidden state to generate the next hidden state and output token.
• Differentiability: The entire process is differentiable, allowing end-to-end training via gradient descent. The network learns to attend to the relevant parts of the input sequence without explicit supervision.
• Initialization of Decoder: The decoder weights are initialized randomly and optimized via gradient descent. The initial hidden state of the decoder can be set to the last hidden state of the encoder, a linear transformation of the last encoder state, or all zeros.
• Visualization: Attention weights can be visualized to understand what the network is focusing on during translation. Diagonal structures in the attention matrix indicate a one-to-one correspondence between input and output words.
• Example: The paper "'Neural Machine Translation' by jointly alerting to align and translate" (2015) demonstrated the effectiveness of attention in machine translation.

Generalizing Attention: From RNNs to a Standalone Operator

• Abstraction: The attention mechanism can be generalized as an operator that takes a set of query vectors and a set of data vectors (with keys and values), and outputs a set of context vectors.
• Components:
  • Query Vectors: Represent the information needed to produce an output.
  • Data Vectors: Represent the information to be summarized.
  • Keys: Used to compute similarity scores with the query vectors.
  • Values: Used to compute the output vectors.
• Process (Standalone Attention Operator):
  1. Similarity Computation: Compute similarities between each query vector and each key vector. A scaled dot product is often used: similarity(q, k) = (q · k) / sqrt(d_k), where d_k is the dimension of the key vectors.
  2. Softmax: Apply a softmax function to the similarity scores to obtain attention weights.
  3. Output Vector: Compute a weighted sum of the value vectors, using the attention weights as weights.
• Benefits of Scaled Dot Product: Scaling the dot product by the square root of the dimension helps prevent vanishing gradients, especially with high-dimensional vectors.
• Matrix Multiplication for Efficiency: Matrix multiplication can be used to efficiently compute all pairwise similarities and weighted sums in parallel.
• Keys and Values: Separating the data vectors into keys and values allows the network to learn different representations for computing similarity and generating the output. This is analogous to a search engine, where the query is matched against the keys, and the value is the information retrieved.

Self-Attention: Attending to Oneself

• Concept: In self-attention, the query, key, and value vectors are all derived from the same input sequence.
• Process:
  1. Linear Projections: Project the input vectors into query, key, and value vectors using three different weight matrices: Q = XW_Q, K = XW_K, V = XW_V.
  2. Attention Computation: Perform the same attention computation as before, using the query, key, and value vectors.
• Benefits: Allows the model to capture relationships between different parts of the input sequence.
• Permutation Equivariance: Self-attention is permutation equivariant, meaning that if the inputs are shuffled, the outputs will be the same, but shuffled in the same way. This implies that self-attention operates on sets of vectors, rather than sequences.
• Positional Embeddings: To incorporate information about the order of the input vectors, positional embeddings can be added to the input vectors.
• Masked Self-Attention: By masking certain entries in the attention matrix, it's possible to control which inputs are allowed to interact with each other. This is useful for tasks like language modeling, where the model should not be able to "look ahead" at future tokens.

Multi-Headed Attention: Capturing Multiple Relationships

• Concept: Run multiple independent copies of self-attention in parallel, each with its own set of weights.
• Process:
  1. Independent Attention Heads: The input is routed to H independent self-attention layers.
  2. Concatenation and Projection: The outputs of the attention heads are concatenated and then projected to produce the final output.
• Benefits: Allows the model to capture different aspects of the input and improve performance.
• Implementation: Multi-headed attention can be implemented efficiently using batch matrix multiplies.

Comparison of Sequence Processing Primitives

• Recurrent Neural Networks (RNNs):
  • Operate on 1D ordered sequences.
  • Not easily parallelizable due to sequential dependencies.
• Convolutional Neural Networks (CNNs):
  • Operate on multidimensional grids.
  • Highly parallelizable.
  • Can be difficult to capture long-range dependencies without large kernels or many layers.
• Self-Attention:
  • Operates on sets of vectors.
  • Naturally generalizes to long sequences.
  • Highly parallelizable.
  • Computationally expensive (n^2 complexity).

Transformers: Attention is All You Need

• Architecture: A transformer is a neural network architecture that uses self-attention as its core mechanism.
• Transformer Block: A basic building block of a transformer, consisting of:
  1. Self-Attention: Allows all vectors to attend to each other.
  2. Residual Connection: Adds the input to the output of the self-attention layer.
  3. Layer Normalization: Normalizes the activations.
  4. Multi-Layer Perceptron (MLP) / Feed Forward Network (FFN): Processes each vector independently.
  5. Residual Connection: Adds the input to the output of the MLP.
  6. Layer Normalization: Normalizes the activations.
• Transformer: A sequence of transformer blocks.
• Applications: Transformers can be used for a wide range of tasks, including language modeling and image classification.
  • Images: Images can be divided into patches, which are then projected into vectors and fed into the transformer.
• Scalability: Transformers have been scaled to hundreds of blocks and trillions of parameters.