Tag: Azure Blob Storage

Migrate Amazon S3 bucket to Azure blob Storage

Migrate AWS S3 buckets to Azure blob storage

With the latest version of AzCopy (version 10), you get a new feature which allows you to migrate Amazon S3 buckets to Azure blob storage. In this blog post, I will show you how you can copy objects, folders, and buckets from Amazon Web Services (AWS) S3 to Azure blob storage using the AzCopy command-line utility. This makes it easy to migrate S3 storage to Azure or create a simple backup of your AWS S3 bucket on Azure.

AzCopy will use the Put Block from URL API, which allows you to directly copy files from AWS directly to Azure. This means you will not use a lot of bandwidth from your computer. You can even copy large objects or buckets from S3 to Azure.

Configure access and authorize AzCopy with Azure and AWS

First, you will need to install AzCopy to your machine. After that, you will need to authorize AzCopy with Microsoft Azure and AWS. To authorize with AWS S3, you have to use an AWS access key and a secret access key.

AWS access key and secret access key, and then set these environment variables:

OSCommand
Windowsset AWS_ACCESS_KEY_ID=
set AWS_SECRET_ACCESS_KEY=
Linuxexport AWS_ACCESS_KEY_ID=
export AWS_SECRET_ACCESS_KEY=
macOSexport AWS_ACCESS_KEY_ID=
export AWS_SECRET_ACCESS_KEY=

Copy an AWS S3 object to Azure blob

You can copy a simple object using the following command:

azcopy cp "https://s3.amazonaws.com/tomsbucket/tomsobject" "https://tomsstorageaccount.blob.core.windows.net/tomscontainer/tomsblob"

Copy and migrate Amazon S3 folder to Azure

You can copy a folder from the Amazon S3 bucket to the Azure blob storage:

azcopy cp "https://s3.amazonaws.com/tomsbucket/tomsfolder" "https://tomsstorageaccount.blob.core.windows.net/tomscontainer/tomsfolder" --recursive=true

Copy an Amazon S3 bucket to Azure blob storage

You can also copy one or multiple Amazon S3 buckets to Azure:

azcopy cp "https://s3.amazonaws.com/tomsbucket" "https://tomsstorageaccount.blob.core.windows.net/tomscontainer" --recursive=true

I hope this gives you a quick idea of how you can migrate data from Amazon AWS S3 storage to Azure using AzCopy. If you want to know more, check out the official Microsoft Docs about how to copy data from Amazon S3 buckets by using AzCopy.



Synchronize Folder with Azure Blob Storage using AzCopy

Sync Folder with Azure Blob Storage

With AzCopy v10 the team added a new function to sync folders with Azure Blob Storage. This is great if you have a local folder running on a server or even on a client device which you can to keep synchronized with Azure Blob storage. This will not only upload new or changed files, with the “–delete-destination” parameter you can let AzCopy remove locally deleted files on Azure blob storage and vice-versa.

First, make sure you install and set up AzCopy.

Sync Folder with Azure Blob Storage

You can use the following command to sync a local folder with Azure Blob Storage. This command will only sync changed and new files, it compares file names and last modified timestamps.

Sync Folder with Azure Blob Storage using AzCopy

 
azcopy sync "C:\Temp\images" "https://tomsaccount.blob.core.windows.net/images" --recursive

As mentioned, if you set the “–delete-destination” parameter to “true”, AzCopy deletes files without a prompt. If you want to check first, which files will be removed, before AzCopy deletes a file, set the –delete-destination flag to “prompt”.

To make sure you are not accidentally are deleting data, make sure to enable the soft delete feature before you use the –delete-destination parameter.

Synchronize Folder with Azure Blob Storage using AzCopy

I deleted the file “3.jpg” locally and I ran the azcopy sync again. You can see that file “3.jpg” was removed from the Azure Blob Storage.

Sync to a local folder

To sync Azure Blob Storage to a local folder, you can use the following command.

 
azcopy sync "https://tomsaccount.blob.core.windows.net/images" "C:\Temp\images" --recursive

As of today, the sync feature does only supports local folders with Azure Blobs. Syncing with AWS or from Storage account to Storage account is currently not supported.

I hope this gives you a quick overview of how you can sync folder with Azure Blob Storage, if you want to know more, check out the Microsoft Docs about how you can transfer data using AzCopy. If you have any questions, please let me know in the comments.