How to Sync AWS S3 Files using AWS CLI: Step-by-Step Guide

How to Sync AWS S3 Files using AWS CLI: Step-by-Step Guide.

How it to do manually:

1. Log into the AWS console, navigate to S3 Service;

2. Find the right bucket, find the right folder;

3. Open the first file, click download;

4. Go back, and open the next file, over and over again.

With AWS CLI, that entire process took less:

aws s3 sync s3://<bucket>/<path> </local/path>

Solution

1. Install the AWS CLI

Download AWS Client (I`m using v1) here.

NOTE: You can also download it and use it on macOS or Linux systems.

2. Get your access keys

Before using AWS Client we must obtain access keys (this is need for feature client configuration)

1. Log into the IAM Console;

2. Go to Users.;

3. Click on your user name;

4. Go to the Security credentials tab;

5. Click Create access key (you can create only two keys for one user account);

6. You’ll see your Access key ID. Click Show to see your Secret access key.

NOTE: Amazon will show this secret access key only once. If you lost it – you must obtain a new key.

3. Configure AWS CLI

1. Run aws configure and answer the prompts.

2. On the first run of aws configure you will just see [None];

3. In the future you can change any of these values by running aws cli again;

4. The prompts will look like AWS Access Key ID [****************ABCD], and you will be able to keep the configured value by hitting return.

$ aws configure
AWS Access Key ID [None]: <enter the access key you just created>
AWS Secret Access Key [None]: <enter the secret access key you just created>
Default region name [None]: <enter region - valid options are listed below >
Default output format [None]: <format - valid options are listed below >

• Valid region names (documented here) are RegionName:
  • ap-northeast-1 [Asia Pacific (Tokyo)]
  • ap-northeast-2 [Asia Pacific (Seoul)]
  • ap-south-1 Asia [Pacific (Mumbai)]
  • ap-southeast-1 [Asia Pacific (Singapore)]
  • ap-southeast-2 [Asia Pacific (Sydney)]
  • ca-central-1 [Canada (Central)]
  • eu-central-1 [EU Central (Frankfurt)]
  • eu-west-1 [EU West (Ireland)]
  • eu-west-2 [EU West (London)]
  • sa-east-1 [South America (Sao Paulo)]
  • us-east-1 [US East (Virginia)]
  • us-east-2 [US East (Ohio)]
  • us-west-1 [US West (N. California)]
  • us-west-2 [US West (Oregon)]
• Valid output formats are
  • json
  • table
  • text

4. Use AWS CLI!

Command sync recursively copies new and updated files from the source directory to the destination. Only creates folders in the destination if they contain one or more files. At this moment we able to download an entire collection of images with a simple

aws s3 sync s3://cdn.enterinit.com/photos/ ~/d:\AWS