If you are faced to transfer large amounts of data into the AWS cloud, you have to make sure that the data somehow get into S3 when you switch to a cloud service like S3. There are several ways to do this, but one of them is to copy the files using the AWS CLI. There are two commands that enable the data transfer: copy and sync.
While files are simply copied using “aws s3 cp”, “aws s3 sync” allows, for example, synchronization of a local directory with AWS S3. During synchronization, only changed files are transferred, which saves considerable resources with large amounts of data. The basic call for this is:
aws s3 sync . s3://<bucket-name>
This copies all data of the current directory (.) to the specified bucket. Before you do this you have to be connected to the S3 Bucket (“aws configure”). Like all parameters, “sync” also allows other functions to be called, for example to explicitly consider or omit certain files and directories. Here is an example in which only all files / directories are transferred that start with “1”:
aws s3 sync . s3://<bucket-name> --exclude '*' --include "1*"
Incidentally, all S3 commands of the AW CLI can be viewed in the official documentation. For some applications it can also be useful to throttle the maximum transmission speed to S3.
Michael Wutzke has more than 20 years of experience in IT. His focus is on the conception and development
of digital solutions on the Internet. Michael is currently working in finance in Frankfurt. More about the person
Tags: AWS, AWS CLI, AWS S3, aws s3 copy, aws s3 sync, copying data to S3, S3