Azure Container Storage v2 Overview

Key Concepts

• Azure Container Storage (ACS) V2: A managed experience for exposing different types of storage to Azure Kubernetes Service (AKS).
• Container Storage Interface (CSI) Drivers: Standardized way for Kubernetes to interact with various storage types.
• Ephemeral Storage: Local storage on the nodes powering Kubernetes clusters.
• NVMe: High-performance non-volatile memory express storage.
• RAID 0: Disk striping without parity, maximizing performance and capacity.
• App-Level Replication: Handling data durability through application-level mechanisms rather than storage-level replication.
• IOPS: Input/Output Operations Per Second, a measure of storage performance.
• Throughput: The rate at which data can be transferred, measured in GB/s.

1. Introduction to Azure Container Storage V2

• The video focuses on the significant changes in Azure Container Storage V2 compared to V1.
• Kubernetes uses CSI drivers for standardized interaction with different storage types across hyperscalers.

2. Azure Container Storage V1 Overview

• ACS V1 provided a managed experience for AKS, exposing storage options beyond standard CSI drivers.
• Ephemeral Storage: Allowed use of local storage (SSD or NVMe) on nodes. This was a popular option due to its high performance and low latency.
• Azure Elastic SAN: Offered scalable storage via an iSCSI-based target.
• Managed Disks: Supported, but less commonly used due to limitations like the one-to-one mapping of persistent volumes to managed disks when using CSI drivers. ACS V1 allowed for one managed disk to have N number of persistent volumes.
• Ephemeral storage was ideal for latency-sensitive and bursty applications.
• Elastic SAN was suitable for scalable storage needs.

3. Azure Container Storage V2: Changes and Focus

• Dropped Support for Managed Disks: Focus shifted to local ephemeral NVMe disks.
• The primary focus of ACS V2 is on high-performance NVMe local storage.
• NVMe offers approximately five times the speed of regular SSDs.
• Supported VM SKUs: L-series (storage-optimized), some GPU-optimized (NC series), and newer versions of D and E series (general purpose and memory optimized) with NVMe disks.
• VM SKUs with NVMe disks are required for ACS V2.

4. NVMe Disk Configuration and Performance

• ACS V2 creates a RAID 0 stripe across all NVMe disks on a node.
• RAID 0 maximizes performance and capacity by striping data across all disks without parity.
• Persistent volumes created with ACS V2 span all NVMe disks, providing the combined IOPS and throughput of all disks, even for small volumes.
• This setup eliminates the need to fill the disks to achieve full performance.

5. Benefits and Use Cases

• Cost Savings: Leveraging ephemeral storage eliminates the need to pay for managed disks.
• High Performance and Low Latency: Ideal for workloads requiring fast storage access.
• Caching Layers: Suitable for caching solutions like Redis or Memcached, especially when exceeding memory capacity.
• Databases: PostgreSQL can benefit from the low latency and high performance.
• AI Workloads: Caching model weights for AI toolchains like Keto AI.

6. Data Durability Considerations

• Ephemeral storage is not durable; data is lost if the node fails or the VM is deallocated.
• App-Level Replication: Recommended for data durability, rather than relying on storage-level replication.
• Replication Strategies:
  • Read replicas on other nodes.
  • Asynchronous replication to a different AKS cluster in another region.
  • Backups to blob storage using tools like Barman for PostgreSQL.
• The application must handle data durability since ACS V2 does not provide native replication.

7. Performance Improvements in V2

• ACS V2 offers up to seven times the performance of V1.
• V2 is a complete rewrite, not a fork of OpenEBS's Mayastor.
• No dependencies on etcd database or local volume manager.
• No performance tiers to configure; ACS V2 automatically optimizes performance.
• Negligible processor usage with memory usage up to 1GB, even at idle.

8. Performance Benchmark

• A performance test on a cluster with two NVMe disks achieved:
  • Average of 1.2 million IOPS.
  • Bandwidth of just below 5 GB/s.

9. Local CSI Driver

• The local access is provided by a CSI driver written and open-sourced by Microsoft.
• This driver can be used in self-hosted Kubernetes clusters by installing via Helm chart.

10. Management and Future Enhancements

• ACS V2 provides a management layer on top of the local CSI driver.
• No minimum node requirement (can run on one or two nodes).
• No fees associated with ACS V2.
• Future support for Azure Elastic SAN is planned.
• No direct migration from V1 to V2 due to breaking changes.
• V1 and V2 cannot run on the same cluster.

11. Conclusion

• ACS V2 focuses on providing high-performance, low-latency storage using local NVMe disks.
• It simplifies the management of local storage but requires users to handle data durability through app-level replication or backup strategies.
• ACS V2 is the future direction for Azure container storage, offering significant performance improvements and cost savings.