2

How to transfer data between s3 buckets of the different aws account using s3cmd ? 

command will be like this: s3cmd rsync s3://acc1_bucket/folder/ s3://acc2_bucket/folder --recursive

But then how it'll identify second bucket environment ? What is the proper way of doing it ?

Ashish Karpe
  • 277

3 Answers3

6

You can use Minio client aka mc, its Open Source & compatible with AWS S3.

Installing Minio client on Linux

$ wget https://dl.minio.io/client/mc/release/linux-amd64/mc
$ chmod 755 mc
$ ./mc --help

Adding AWS S3 credentials

$ ./mc config host add mys3one https://s3.amazonaws.com BKIKJAA5BMMU2RHO6Izz V7f1CwQqAcwo80UEIJEjc5gVQUSSx5ohQ9GSrr12

$ ./mc config host add mys3two https://s3.amazonaws.com BKIKJAA5BMMU2RHO6IBB V7f1CwQqAcwo80UEIJEjc5gVQUSSx5ohQ9GSaa64

Using mc mirror to transfer object/bucket from one account to another.

$ ./mc mirror mys3one/photos mys3two/photos2

In this example, s3 account with alias "mys3one" with bucketname "photos" is getting mirrored to s3 account with alias "mys3two" with bucketname photo2.

You can automate the same by adding this to cron, so periodically whenever needed content can get synced.

Hope it helps. Disclaimer: I work for Minio

koolhead17
  • 401
3

you simply have to authorize your IAM user to access to the buvket, within the S3 bucket policy, like so:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "whatever",
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "<ARN OF YOUR IAM USER>"
                ]
            },
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation",
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::<YOUR BUCKET NAME>", 
                "arn:aws:s3:::<YOUR BUCKET NAME>/*"
            ]
        }
    ]
}

Then, because this is cross account, you also have to allow your IAM user to perform S3 calls, by attaching a policy to your IAM user, like so:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "whateveryoulike",
            "Effect": "Allow",
            "Action": [
                "s3:*"
            ],
            "Resource": [
                "arn:aws:s3:::<BUCKET_NAME>",
                "arn:aws:s3:::<BUCKET_NAME>/*"
            ]
        }
    ]
}

simply configure your cli, and you will be able to access your bucket cross account.

Tom
  • 657
1

Accounts use different credentials. There's no way to sync from one account to another and provide separate credentials. The only way to do that is to allow public access to the second bucket with a temporary account.

If that is not an option, then your best bet is to:

s3 sync s3://acct1/bucket . --profile acct1 s3 sync . s3://acct2/bucket --profile acct2

Marc Young
  • 141