Multi-Cloud Storage Migration with Azure Storage Mover

Azure Storage Mover: Multicloud Migration (AWS S3 to Azure Blob)

Key Concepts:

• Azure Storage Mover: A managed service for migrating storage.
• Azure Arc Multicloud Connector: Establishes a connection between Azure and other cloud providers (AWS).
• S3 Bucket: AWS Simple Storage Service.
• Blob Container: Azure Blob storage.
• Data Egress: Data exiting a cloud provider (AWS).

Overview of Azure Storage Mover's New Capabilities

The primary focus of the video is on the expanded capabilities of Azure Storage Mover, specifically its new support for migrating data from other cloud providers, starting with AWS S3 buckets, directly into Azure Blob storage. Previously, Azure Storage Mover primarily focused on on-premises to Azure migrations (SMB/NFS to Azure Files/Blob). The new functionality eliminates the need for complex, manual processes involving tools like AzCopy and simplifies management, operations, orchestration, authentication, and key management.

Prerequisites: Azure Arc Multicloud Connector

A crucial prerequisite for using the AWS S3 to Azure Blob migration is the deployment and configuration of the Azure Arc Multicloud Connector.

Step-by-step:

1. Deploy Azure Arc Multicloud Connector: The connector establishes a link from the Azure control plane (Azure Resource Manager) to the AWS account.
2. Enable Inventory Solution: Within the Azure Arc Multicloud Connector settings, enable the inventory solution, specifically for S3. While other AWS services can be inventoried, S3 is the only mandatory service for the storage migration functionality.
3. CloudFormation Stack: Enabling the inventory solution triggers the creation of an AWS CloudFormation stack, which configures the necessary permissions for connectivity between Azure and AWS.

Deploying and Configuring Azure Storage Mover

The Azure Storage Mover resource itself needs to be deployed within a specific Azure region. While placing it in a region close to the target storage can offer potential optimizations, it's not a strict requirement.

Configuration Steps:

1. Create Storage Mover Resource: This involves specifying the subscription, resource group, name, and region for the Storage Mover.
2. Configure Monitoring: Detailed monitoring data can be enabled to track the progress of storage mover projects and jobs.
3. Create Projects: Projects are logical groupings of migration tasks.
4. Create Jobs: Jobs define the specifics of the migration, including the source (S3 bucket with optional subpath), destination (Blob container with optional subpath), and migration mode.

Migration Modes and Job Execution

The migration mode determines how data is transferred and managed between the source and destination.

• Merge Mode: Content from the source is merged into the target. Existing files in the target that are not present in the source are preserved.
• Mirror Mode: The target is made an exact replica of the source. Files present in the target but not in the source are deleted.

Job execution is a one-time operation. To maintain synchronization, a script (e.g., PowerShell) would be needed to rerun the job periodically. The actual data transfer occurs via a service-to-service copy, meaning no AWS compute resources are used.

Monitoring and Performance

The Azure portal provides monitoring capabilities for tracking the progress of migration jobs. This includes:

• Number of files and folders discovered.
• Volume of data transferred.
• Processing speed and throughput.
• The presenter observed throughputs of around 2 GB per second in his test, but notes that performance can vary depending on factors like file size and the number of small files.

Cost Considerations

While the Azure Storage Mover service itself is free, there are other potential costs:

• AWS Data Egress Fees: Data exiting AWS incurs egress charges. An egress wave process may be available to waive these charges, but this needs to be verified with AWS.
• Transaction Costs: Reading data from the source (AWS S3) and writing data to the destination (Azure Blob storage) incur transaction costs.
• Storage Costs: Standard Azure storage costs apply for the data stored in Blob storage.

Limitations

At the time of recording, there was a limit of 10,000 objects per job. It is crucial to check the official documentation for the latest limits, as they are subject to change.

Key Quotes

• "Azure storage mover now supports other clouds for migrations into Azure... it's removing a lot of the previous complexity."
• "[Mirror mode] will actually make that target look exactly like the source. So if the source doesn't have files that were in the target, it's going to remove them from the target."
• "This is a free service. There is no cost for Azure storage mover itself."

Technical Terms Explained

• Azure Arc: A service that extends Azure management capabilities to infrastructure outside of Azure.
• Multicloud Connector: A component of Azure Arc that enables connectivity and management across multiple cloud providers.
• Service-to-Service Copy: A data transfer method where data is copied directly between cloud services without the need for intermediary compute resources.
• Egress: The process of data leaving a cloud environment.

Logical Connections

The video logically connects the need for multicloud migration with the challenges of using traditional methods (e.g., AzCopy). It then introduces Azure Storage Mover as a solution and explains how it leverages Azure Arc for connectivity. The configuration steps are presented in a sequential manner, highlighting the dependencies between the Azure Arc Multicloud Connector, the Storage Mover resource, projects, and jobs. The discussion of migration modes and cost considerations provides a comprehensive overview of the key factors to consider when planning a migration.

Synthesis/Conclusion

Azure Storage Mover with its new multicloud migration capabilities offers a simplified and managed approach to moving data from AWS S3 to Azure Blob storage. By leveraging Azure Arc, it eliminates the complexities associated with manual migration processes and provides a centralized management interface. While the service itself is free, users need to be aware of potential data egress costs from AWS and transaction costs associated with reading and writing data. The video highlights the ease of use and efficiency of Azure Storage Mover, making it a compelling option for organizations looking to migrate data into Azure.