Azure Files Performance Upgrade

Key Concepts

• Azure Files performance improvements
• Metadata operations (file create, open, close, delete)
• Handles (root and file level)
• Directory leases (query directory API)
• SMB protocol
• Multi-channel (Linux)
• Azure Files SSD (Premium)
• IOPS and Throughput
• Network Latency

Metadata Cache

• Problem: Azure Files, being a distributed system, traditionally suffers from latency in metadata operations compared to on-premise file servers with memory-based caches.
• Solution: Introduction of a metadata cache in the Azure Files stack. This cache stores metadata in memory, significantly speeding up operations like file creation, opening, closing, and deletion.
• Performance Gains:
  • File create operations: 30% faster.
  • File deletes: 25% faster.
  • File closes: 45% faster.
  • 3x increase in metadata IOPS.
  • Example: File open/close time reduced from 4ms to sub-2ms.
• Scope: Currently available for Azure Files SSD (Premium) with SMB protocol. Standard and NFS support are on the roadmap.
• Deployment: Rolled out to all users following safe deployment practices. Users needing immediate access can register for the feature to expedite deployment to their storage account.
• Pricing: No pricing changes associated with this feature.

Increased Handle Limits

• Problem: Previous limits on the number of handles (2,000 per file, 10,000 per root) could restrict the scalability of applications requiring many concurrent connections. For example, a login script requiring a handle on the root would limit concurrent logins to 10,000.
• Solution: Increasing the number of handles, enabled by storing handles in memory.
• Details:
  • The exact new limits are still in preview and being determined.
  • The goal is to align the number of handles per file with the number of handles per root.
  • Example: The root handle limit might increase to 30,000.
• Impact: Allows for greater scalability, especially for workloads like VDI profiles. Reduces the need to create and manage multiple file shares solely to accommodate handle limits.

Directory Leases

• Problem: High latency in directory enumeration, especially for large directories, due to frequent round trips to the server.
• Solution: Introduction of directory leases, which enable server-side caching of directory metadata on a per-client basis.
• Functionality:
  • Similar to the actimeo option in Linux NFS mounts, but implemented on the server side.
  • The cache is maintained indefinitely and invalidated when changes occur in the directory.
  • Different lease types (read, write, handle) are broken under specific conditions (metadata changes, conflicting opens, sharing violations).
• Performance Gains: Up to 97% reduction in directory enumeration time for large directories.
• Use Case: Significant performance improvement for Windows clients working with large FSLogix profiles.
• Relationship to Metadata Cache: The metadata cache improves performance for file-level operations (create, delete, open, close), while directory leases improve performance for directory-level operations (directory listing).

Multi-Channel for Linux

• Functionality: Extends the multi-channel capability, previously available for Windows clients, to Linux clients.
• Details:
  • Allows establishing multiple channels (up to four) to the file share, combining their performance.
  • Can increase IOPS from 20,000 to 80,000 per client (assuming the client is capable).
• Requirements: Requires a premium SSD file share.
• Benefit: Increased network bandwidth, IOPS, and throughput for Linux clients.

Network Latency Considerations

• Importance: Network latency remains a significant factor affecting performance.
• Best Practice: Place clients as close as possible to the Azure file share to minimize latency.

Conclusion

Azure Files is receiving significant performance improvements through the introduction of a metadata cache, increased handle limits, directory leases, and multi-channel support for Linux. These changes aim to address performance bottlenecks related to metadata operations, scalability, and directory enumeration, making Azure Files a more viable option for a wider range of workloads. The features are being rolled out automatically, with an option to expedite deployment. Users should consult the documentation for the latest details on limits and configuration options.