Google StorageTransfer job from S3 to GCS failed due to The caller does not have permission

Question

I tried to transfer a file from Amazon S3 to Google Cloud Storage using API. I did successfully on the GCP console but when I'm writing a Python script using a service account credential, I got HttpError 403 when requesting https://storagetransfer.googleapis.com/v1/transferJobs?alt=json returned "The caller does not have permission

Also, My service account is already a Storage Object Admin.

The script I'm using

import google.auth
import google_auth_httplib2
import google.oauth2.service_account
from googleapiclient.discovery import build


_DEFAULT_SCOPES = ('https://www.googleapis.com/auth/cloud-platform',)

credentials = (
                    google.oauth2.service_account.Credentials.from_service_account_file(
                        key_path, scopes=_DEFAULT_SCOPES)
                )

http = httplib2.Http()
http_authorized = google_auth_httplib2.AuthorizedHttp(
            credentials, http=http)

conn = build('storagetransfer', 'v1', http=http_authorized, cache_discovery=False)
now = datetime.datetime.utcnow()

project_id='projectid'
transfer_spec = {
    'awsS3DataSource': {
        'bucketName': 's3_bucket',
        'awsAccessKey': {
            'accessKeyId': 'key',
            'secretAccessKey': 'secret',
        }
    },
    'gcsDataSink': {
        'bucketName': 'bucket',
    },
    'objectConditions': {'includePrefixes': ['PREFIX']},
    'transferOptions': {'overwriteObjectsAlreadyExistingInSink': True},
}

transfer_job = {
            'status': 'ENABLED',
            'projectId': project_id,
            'transferSpec': transfer_spec,
            'schedule': {
                'scheduleStartDate': {
                    'day': now.day,
                    'month': now.month,
                    'year': now.year,
                },
                'scheduleEndDate': {
                    'day': now.day,
                    'month': now.month,
                    'year': now.year,
                }
            }
        }

job = conn.transferJobs().create(body=transfer_job).execute()

Does anyone know what's missing?

Docs: https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferJobs/create

were you able to solve this? – user780092 Jan 29 '20 at 23:16 — user780092, Jan 29 '20 at 23:16
No, It hasn’t solved yet – prideloki Jan 31 '20 at 11:40 — prideloki, Jan 31 '20 at 11:40
Having the same issue -- did you ever resolve this? – DanM Jul 22 '20 at 14:47 — DanM, Jul 22 '20 at 14:47

score 0 · Answer 1 · answered Oct 02 '20 at 22:39

0

According to the doc here you need to assign one of the following permission owner/editor/storagetransfer.admin/storagetransfer.user. There is also a python scripting example might be helpful to transfer data from S3 to GCS.

answered Oct 02 '20 at 22:39

Mahboob

1,877
5
19

Google StorageTransfer job from S3 to GCS failed due to The caller does not have permission

1 Answers1