Lecture 15 - PCA and ICA | Stanford CS229: Machine Learning Andrew Ng - Autumn 2018

Key Concepts

• Dimensionality Reduction: Techniques like PCA and ICA reduce the number of variables in a dataset while preserving important information.
• PCA (Principal Component Analysis): Identifies directions of maximum variance in data to create principal components for dimensionality reduction. It’s most effective when data lies in a lower-dimensional subspace.
• ICA (Independent Component Analysis): Separates mixed signals into independent sources, exemplified by the “cocktail party problem.” It assumes underlying sources are statistically independent.
• Data Pre-processing: Crucial for PCA, involving zeroing the mean and standardizing variance to ensure features are comparable.
• Mathematical Foundations: Both PCA and ICA rely on linear algebra concepts like eigenvectors, eigenvalues, covariance matrices, and matrix operations.

Principal Component Analysis (PCA)

PCA is a non-probabilistic algorithm used to determine if data resides in a lower-dimensional subspace and, if so, to reduce dimensionality. It differs from Factor Analysis by not modeling the probability density function P(X). The speaker cautions against its overuse, emphasizing it’s most useful for visualization, computational efficiency, or when data demonstrably lies in a lower-dimensional subspace, and suggests regularization is often a more reliable method for preventing overfitting.

PCA aims to find a direction (unit vector u) that minimizes the sum of squared projection distances of data points, or equivalently, maximizes the spread of the projected data. For reducing data to k dimensions, the algorithm identifies the top k eigenvectors of the covariance matrix (Σ) of the pre-processed data.

Implementation Steps:

1. Zero-out the mean of each feature.
2. Standardize the variance of each feature (divide by standard deviation).
3. Compute the covariance matrix (Σ) of the pre-processed data.
4. Calculate the eigenvectors and eigenvalues of Σ.
5. Select the top k eigenvectors (corresponding to the largest eigenvalues).
6. Project the data onto the subspace spanned by the selected eigenvectors.

The original data can be approximately reconstructed from the reduced representation using the principal components. Examples illustrating PCA’s application include height measurements (cm vs. inches), factory vibration sensor data (noise reduction and identifying primary shaking patterns), and pilot skill/enjoyment scores (identifying underlying aptitude). A compelling example involved reducing 50-dimensional neural data from monkey brain electrodes to visualize brain activity during a motor task. The speaker notes that while individual eigenvectors can be noisy, the subspace spanned by the top k eigenvectors is more stable and interpretable.

Independent Component Analysis (ICA)

ICA is an unsupervised learning algorithm that finds independent axes of variation, contrasting with PCA’s focus on principal components. The “cocktail party problem” – separating multiple overlapping audio sources – serves as a motivating example.

ICA aims to “unmix” signals recorded by multiple microphones, each capturing a linear combination of all speakers’ voices. The algorithm models the data as generated by a set of independent sources (S) in Rn, where n is the number of speakers. Each microphone recording (Xij) is defined as a linear combination of the speaker sources using a mixing matrix (A): Xij = Σk (Ajk * Sik). ICA then seeks an “unmixing” matrix (W), the inverse of A, to recover the original sources: S = WX. The rows of W are crucial for source recovery.

ICA Implementation Framework:

1. Data Input: Two or more microphone recordings (X).
2. Source Modeling: Assume data is generated by independent sources (S) in Rn.
3. Mixing Process: Each microphone recording is a linear combination of sources (X = AS).
4. Unmixing: Find the unmixing matrix (W) to recover the original sources (S = WX).

ICA faces axis and sign ambiguities, but these are often inconsequential in audio applications. For initial algorithm development, the number of speakers should equal the number of microphones. Sound is represented as minute variations in air pressure, modeled as periodic functions, as demonstrated with a tuning fork example. ICA is possible because the underlying sources are statistically independent. A visualization using random numbers between -1 and 1 illustrates how mixed data forms a parallelogram, which ICA aims to revert to the original independent distribution. Audio examples demonstrated the separation of two speakers and the improvement in clarity of a counting voice when separated from music. Consistent timestamping between microphones is required for accurate ICA implementation.

Conclusion

Both PCA and ICA are powerful dimensionality reduction techniques, but they serve different purposes. PCA focuses on maximizing variance, making it suitable for data compression and visualization when a lower-dimensional subspace exists. ICA, on the other hand, excels at separating independent sources, as demonstrated by its application to the cocktail party problem. Understanding the underlying assumptions and limitations of each algorithm is crucial for effective application, and the speaker emphasizes that these techniques should not be applied blindly.