K Means Clustering Algorithm | K Means In Python | Machine Learning Algorithms |Simplilearn

K-Means Clustering

Key Concepts: Unsupervised learning, clustering, centroids, distance measures (Euclidean, Manhattan, Cosine), elbow method, convergence.

What is K-Means Clustering?

K-means clustering is an unsupervised learning algorithm used to group unlabeled data into clusters based on similarity. The algorithm aims to partition n observations into k clusters, where each observation belongs to the cluster with the nearest mean (cluster center or centroid), serving as a prototype of the cluster. The value of k (number of clusters) needs to be specified beforehand.

Example: Clustering Cricket Players

A practical example is clustering cricket players into batsmen and bowlers based on their runs scored and wickets taken. The data consists of two characteristics: runs (y-axis) and wickets (x-axis). The goal is to form two clusters (K=2): one representing batsmen (high runs) and the other representing bowlers (high wickets).

K-Means Clustering Process: A Step-by-Step Guide

1. Initialization: Randomly allocate k centroids. These initial points are not necessarily the actual centroids.
2. Distance Calculation: Determine the distance of each data point from each centroid using a distance measure (e.g., Euclidean distance).
3. Assignment: Assign each data point to the closest centroid based on the minimum distance. This forms initial clusters.
4. Centroid Recalculation: Calculate the actual centroid (mean position) for each of the newly formed clusters. Reposition the original randomly allocated centroid to this actual centroid.
5. Iteration: Repeat steps 2-4 until the centroid repositioning stops, indicating that the algorithm has converged. Some data points may be reallocated to different clusters during this iterative process.

Types of Clustering

There are two primary categories of clustering:

• Hierarchical Clustering: Clusters have a tree-like structure.
  • Agglomerative (Bottom-up): Starts with each element as a separate cluster and merges them into successively larger clusters.
  • Divisive (Top-down): Begins with the whole set and divides it into successively smaller clusters.
• Partitional Clustering:
  • K-Means Clustering: Divides objects into k clusters based on similar characteristics. An object can only belong to one cluster.
  • Fuzzy C-Means: Similar to K-means, but objects can belong to more than one cluster with varying degrees of membership.

Applications of K-Means Clustering

K-means clustering has various real-world applications, including:

• Academic Performance: Categorizing students into grades (A, B, C, etc.) based on their scores.
• Diagnostic Systems: Grouping patients based on similar symptoms or medical data.
• Search Engines: Grouping search results together based on relevance.
• Wireless Sensor Networks: Finding cluster heads to collect data in their respective clusters.

Distance Measures in K-Means Clustering

Distance measures quantify the similarity between objects. Common distance measures used in K-means clustering include:

• Euclidean Distance: The straight-line distance between two points. Formula: sqrt(sum((yi - xi)^2))
• Squared Euclidean Distance: The square of the Euclidean distance.
• Manhattan Distance: The sum of the absolute differences of their coordinates. Formula: sum(|xi - yi|)
• Cosine Distance: Measures the angle between two vectors.

Determining the Optimal Number of Clusters (K)

The elbow method is a technique used to determine the optimal value of k. It involves plotting the within-cluster sum of squares (WSS) for different values of k. WSS measures the compactness of the clusters; lower values indicate better clustering. The "elbow" point on the plot, where the rate of decrease in WSS starts to diminish, suggests the optimal value of k.

K-Means Clustering Algorithm: Detailed Steps

1. Input: Data points X1, X2, X3, ..., Xn and the desired number of clusters k.
2. Initialization: Randomly pick k points as initial centroids (C1, C2, ..., Ck).
3. Assignment:
   • Calculate the distance of each data point from each centroid.
   • Assign each data point to the closest centroid.
4. Update:
   • Calculate the new centroids for each cluster (mean position of the data points in the cluster).
5. Iteration: Repeat steps 3 and 4 until the centroids stop changing significantly (convergence).

Demos and Use Cases: Walmart Store Location Optimization

Problem Statement: Walmart wants to open a chain of stores across Florida and needs to find optimal store locations. Opening too many stores close together reduces profit, while stores too far apart limit sales.

Solution: Use K-means clustering to analyze customer addresses (data points) and identify optimal store locations (centroids).

Steps:

1. Data Loading and Preparation: Load customer address data.
2. Visualization: Create a scatter plot to visualize the data distribution and get an initial idea of potential clusters.
3. K-Means Clustering:
   • Specify the number of clusters (k).
   • Assign data points to centroids based on distance.
   • Iteratively recalculate centroids until convergence.
4. Output: The centroids represent the optimal store locations.

Python Implementation (Scikit-learn)

The scikit-learn library provides a readily available K-means implementation.

from sklearn.cluster import KMeans

## Create a KMeans instance, specifying the number of clusters (K)
kmeans = KMeans(n_clusters=4)

## Fit the model to the data
kmeans.fit(X) # X is the data

## Predict the cluster labels for each data point
y_kmeans = kmeans.predict(X)

Color Compression Use Case

K-means clustering can be used for color compression in images. The goal is to reduce the number of colors in an image while preserving its visual quality.

Steps:

1. Image Loading and Reshaping: Load the image and reshape it into a two-dimensional array of pixels.
2. K-Means Clustering: Apply K-means clustering to the pixel data, where k is the desired number of colors.
3. Color Replacement: Replace each pixel's color with the color of its assigned centroid.
4. Reshape and Display: Reshape the data back into the original image dimensions and display the compressed image.

Conclusion

K-means clustering is a versatile unsupervised learning algorithm for grouping data into clusters based on similarity. It involves iteratively assigning data points to centroids and recalculating centroid positions until convergence. The algorithm's effectiveness depends on the choice of k (number of clusters) and the distance measure used. The elbow method helps determine the optimal k. K-means clustering has numerous applications, including customer segmentation, image compression, and anomaly detection.