Tag: copy

Migrate Amazon S3 bucket to Azure blob Storage

Migrate AWS S3 buckets to Azure blob storage

With the latest version of AzCopy (version 10), you get a new feature which allows you to migrate Amazon S3 buckets to Azure blob storage. In this blog post, I will show you how you can copy objects, folders, and buckets from Amazon Web Services (AWS) S3 to Azure blob storage using the AzCopy command-line utility. This makes it easy to migrate S3 storage to Azure or create a simple backup of your AWS S3 bucket on Azure.

AzCopy will use the Put Block from URL API, which allows you to directly copy files from AWS directly to Azure. This means you will not use a lot of bandwidth from your computer. You can even copy large objects or buckets from S3 to Azure.

Configure access and authorize AzCopy with Azure and AWS

First, you will need to install AzCopy to your machine. After that, you will need to authorize AzCopy with Microsoft Azure and AWS. To authorize with AWS S3, you have to use an AWS access key and a secret access key.

AWS access key and secret access key, and then set these environment variables:

OSCommand
Windowsset AWS_ACCESS_KEY_ID=
set AWS_SECRET_ACCESS_KEY=
Linuxexport AWS_ACCESS_KEY_ID=
export AWS_SECRET_ACCESS_KEY=
macOSexport AWS_ACCESS_KEY_ID=
export AWS_SECRET_ACCESS_KEY=

Copy an AWS S3 object to Azure blob

You can copy a simple object using the following command:

azcopy cp "https://s3.amazonaws.com/tomsbucket/tomsobject" "https://tomsstorageaccount.blob.core.windows.net/tomscontainer/tomsblob"

Copy and migrate Amazon S3 folder to Azure

You can copy a folder from the Amazon S3 bucket to the Azure blob storage:

azcopy cp "https://s3.amazonaws.com/tomsbucket/tomsfolder" "https://tomsstorageaccount.blob.core.windows.net/tomscontainer/tomsfolder" --recursive=true

Copy an Amazon S3 bucket to Azure blob storage

You can also copy one or multiple Amazon S3 buckets to Azure:

azcopy cp "https://s3.amazonaws.com/tomsbucket" "https://tomsstorageaccount.blob.core.windows.net/tomscontainer" --recursive=true

I hope this gives you a quick idea of how you can migrate data from Amazon AWS S3 storage to Azure using AzCopy. If you want to know more, check out the official Microsoft Docs about how to copy data from Amazon S3 buckets by using AzCopy.



AzCopy Azure and Azure Stack

Sync Azure Blob Storage with Azure Stack Blob Storage

There are some scenarios where you want to sync Blob Storage on Azure with Blob Storage on Azure Stack. This easily be done using the AzCopy tool. AzCopy is a command-line utility designed to copy data to and from Microsoft Azure blob and table storage using simple commands with optimal performance. You can copy data from one object to another within your storage account, or between storage accounts in Azure as well as in Azure Stack.

You can get AzCopy on Windows and AzCopy on Linux. You should get at least AzCopy version 7.1.0. In my example I use the version 8.0.0 preview version, which also works fine. Make also sure that you have deployed a supported version of Azure Stack. Check out my blog about updating Azure Stack to get more information.

After the installation you will find the AzCopy.exe in the “C:\Program Files (x86)\Microsoft SDKs\Azure\AzCopy”

How to Sync and copy files from Azure to Azure Stack

You can use AzCopy with Azure Stack as you would do in Microsoft Azure. To copy and Sync files you can use the following command. Be aware that synchronous data transfer between Azure storage and Azure Stack is not supported. There for you will need to specify the the /SyncCopy or –sync-copy parameter.

 
#Windows
 
AzCopy /Source:https://azcopywithazs.blob.core.windows.net/thomasmaurer /Dest:https://azcopywithazs.blob.azurestack.thomasmaurer.ch/thomasmaurer /SourceKey:key1 /DestKey:key2 /S /SyncCopy
 
#Linux
 
azcopy \
--source https://azcopywithazs.blob.core.windows.net/thomasmaurer/ \
--destination https://azcopywithazs.blob.azurestack.thomasmaurer.ch/thomasmaurer/ \
--source-key key1 \
--dest-key key2 \
--include "tm.txt" \
--sync-copy

Of course this works in both directions and also between two different Azure Stack storage accounts.

You can find more about AzCopy on Azure Stack on the documentation page. If you have any questions feel free to comment on the blog post. There are other great storage tools which work great with the Microsoft Hybrid Cloud environment like Visual Studio or Azure Storage Explorer. These scenarios show perfectly how Microsoft built their hybrid cloud environment. Basically you can use the same tools, processes and knowhow in the Public Cloud as well as in your datacenter.



Veeam FastSCP for Microsoft Azure

Veeam FastSCP for Microsoft Azure

Veeam does some great products for your virtualization and datacenter environment such as their Veeam Backup & Replication suite, Veeam Endpoint Backup FREE and Management Packs for System Center Operations Manager. Now a couple of weeks ago Veeam released a cool free tool call Veeam FastSCP fro Microsoft Azure. With Veeam FastSCP (Secure Copy Protocol) for Microsoft Azure, IT Pros and Azure Developers can simply and reliably copy local files to Azure VMs, and copy files in Azure VMs to on-premises.

Veeam FastSCP for Microsoft Azure Diagram

The utility makes your life way easier when dealing with Virtual Machines running on Microsoft Azure IaaS.

  • Secure file copy with no independent encryption or VPN needed
  • Manual file copy to/from Azure VMs without the need to keep the UI open until the file copy completes
  • Automatic scheduling of file copy jobs for nightly or weekly copies to/from Azure VMs
  • A wizard-driven UI to copy files in just a few clicks – with no scripting needed

If you want to download it, check out the Veeam Website.

To set it up the tool connects to the PowerShell endpoint for your IaaS VM. Just add the Virtual Machine and you are ready to go! With that you can do some great things, like simply copy a file to an Azure IaaS VM or even doing scheduled backups of files from inside Azure VMs like Didier Van Hoye did.

 

 



PowerShell

PowerShell: Copy Log files with Date and Content

Some days ago I made a blog post about Select-String in PowerShell. I created a “script” or a better a command to check log files for a specific text and copy the log file to another place using PowerShell.

Get-ChildItem C:\temp -Filter *.log -Recurse | Select-String "Contoso" |  Copy-Item -Destination C:\temp2

Now I added also a Start date and a End date for the log files you wanna search in.

$Startdate = (get-date -year 2011 -month 3 -day 25)
$Enddate = (get-date -year 2011 -month 3 -day 30)
 
Get-ChildItem C:\temp -Filter *.log -Recurse | Where-Object {($_.LastWriteTime.Date -ge $Startdate.Date) -and ($_.LastWriteTime.Date -le $Enddate.Date)} | Select-String "Contoso" |  Copy-Item -Destination C:\temp2

Also check out my blog post to clean up files older than a specific date using PowerShell and how to install PowerShell 6 and PowerShell 7. If you have any questions, please let me know in the comments.