Python Job on Linux EC2 Machine that drops file on Google Drive

Question

I've seen varous articles have address some of what I want, but not all. Using Python how do I upload a file to a specific folder on Google Drive? Additionally, I want to make sure the token never expires as I want it to run from a remote machine.

Edit I have tried the following using a service account.

SCOPES = [
        'https://www.googleapis.com/auth/drive'
    ]
SERVICE_ACCOUNT_FILE = 'path/to/local/creds.json'
credentials = service_account.Credentials.from_service_account_file(
            SERVICE_ACCOUNT_FILE, scopes=SCOPES)
service = discovery.build('drive', 'v3', credentials=credentials)
SS_SERVICE = build('sheets', 'v4', credentials = credentials)

file_metadata = {
    'name': 'A Test File.csv',
    'parents': [{"kind": "drive#folder",
                 "id":'idoffolder'}],
    'mimeType': 'text/csv'
}
req = service.files().create(body=file_metadata,
                                    media_body='path_to_local_csv')
req.uri = req.uri + '&convert=true'
resp = req.execute()

I retrieved the id of the folder from the URL: https://drive.google.com/drive/u/1/folders/idIwanttouse

This code seems to work in that there aren't errors, however I don't see a file called A Test File.csv in the folder I specified.

Interestingly when I run the following:

service.files().list().execute()

I see the file "A Test File.csv" but no other files in my google drive shared account.

A couple notes: I'm trying to add files to a shared business account that I am a member of, additionally I'm using a service account.

Any ideas?

There's already a very similar dupe https://stackoverflow.com/questions/63422138/how-to-upload-file-into-specific-folder-with-google-drive-api-and-python except you said you don't want the token to expire https://stackoverflow.com/questions/63354260/when-does-refresh-token-expires-for-google-drive-api so unless you show what you've tried and where you actually get stuck there's no way to answer the question — Martheen, Mar 01 '21 at 03:51
I" looking for something all encompassing. I'm having trouble even retreiving the key. Google APIs are confusing (at least to me). — ben890, Mar 01 '21 at 13:01
Just start following the steps in the [quickstart](https://developers.google.com/drive/api/v3/quickstart/python) and adapt from there. The token will always expire unless you use service accounts, but depending on what you need it for that may not be viable for you (most applications require a workspace account) — iansedano, Mar 02 '21 at 09:09

Tanaike · Accepted Answer · 2021-03-07T03:40:26.110

3

I believe your goal and your current situation as follows.

You want to upload a CSV file of A Test File.csv to the specific folder in Google Drive by converting to Google Spreadsheet using the service account.
From I see the file "A Test File.csv" but no other files in my google drive shared account., you want to see the uploaded file on your Google Drive.
You want to achieve this using Drive API v3 with googleapis for python.
You have already been able to use Drive API using the service account.

Modification points:

When you want to upload a file to the specific folder using Drive API v3, please set the folder ID of the file metadata like 'parents': ['idoffolder'],. In your script, 'parents': [{"kind": "drive#folder","id":'idoffolder'}], is used. This is used for Drive API v2. I think that this might be one of reasons of your issue.
- In this case, when you want to see the uploaded file in the specific folder of your Google Drive, please share the folder with the email of the service account. By this, you can see it. Please be careful this.
In order to upload a file of local PC, please modify media_body='path_to_local_csv' to media_body=MediaFileUpload("path_to_local_csv", mimetype="text/csv", resumable=True). I think that this might be one of reasons of your issue.
When you want to upload the CSV file by converting to Google Spreadsheet, you can achieve this by including the mimeType to the file metadata.

When above points are reflected to your script, it becomes as follows.

Modified script:

Before you use this script, please set and confirm the variables.

from googleapiclient import discovery
from google.oauth2 import service_account
from apiclient.http import MediaFileUpload

SCOPES = ['https://www.googleapis.com/auth/drive']
SERVICE_ACCOUNT_FILE = 'path/to/local/creds.json'
credentials = service_account.Credentials.from_service_account_file(SERVICE_ACCOUNT_FILE, scopes=SCOPES)
service = discovery.build('drive', 'v3', credentials=credentials)
file_metadata = {
    'name': 'A Test File.csv',
    'parents': ['idoffolder'],
    'mimeType': 'application/vnd.google-apps.spreadsheet'
}
media_body = MediaFileUpload('path_to_local_csv', mimetype="text/csv", resumable=True)
resp = service.files().create(body=file_metadata, media_body=media_body, supportsAllDrives=True).execute()
print(resp)

When above script is run, the CSV file of path_to_local_csv in your local PC is uploaded to the specific folder idoffolder of Google Drive as the Google Spreadsheet.

Note:

In this modified script, it supposes that creds.json of your service account have the permission for uploading the file to the specific folder.
If your will use the service account to impersonate their regular account, please check domain-wide delegation.
About Additionally, I want to make sure the token never expires as I want it to run from a remote machine., in above script, the access token is retrieved every run. By this, I think that this issue might not occur.

References:

edited Mar 07 '21 at 03:40

answered Mar 07 '21 at 02:45

Tanaike

181,128
11
97
165

"In this case, when you want to see the uploaded file in the specific folder of your Google Drive, please share the folder with the email of the service account. By this, you can see it. Please be careful this." I tried doing this using the steps laid out here: https://stackoverflow.com/questions/43243865/how-to-access-team-drive-using-service-account-with-google-drive-net-api-v3 Howveer, I'm getting an error upon running the modified script: – ben890 Mar 07 '21 at 03:06
@ben890 Thank you for replying. From your error message, it was found that your error was changed. And, I have to apologize for the inconvenience. From `File not found: XX`, in this case, it is considered that the service account cannot access to the folder ID. For example, when the service account is not the writer permission, an error of `Forbidden` occurs. So, can you confirm whether the folder is shared with the email of the service account again? And, as a test case, when the file and folder created by the service account is shared with your Google account, you can see it? – Tanaike Mar 07 '21 at 03:26
@ben890 By the way, although I'm not sure whether this is the direct solution of your current issue, I added `supportsAllDrives=True` in the sample script. Could you please confirm it? After you confirmed whether the folder is shared with the email of the service account, can you test it again? These were not the direct solution of your current issue, I apologize again. – Tanaike Mar 07 '21 at 03:40
yes I can confirm that my google account can see the folder and my service account has manager permissions. DOes my service account have to be the creator of the folder? ANy other suggestions? Am I using the right string as the folder_id? Also this is a folder within a shared folder. – ben890 Mar 07 '21 at 15:27
never midn it appears this worked. Thank you! – ben890 Mar 07 '21 at 15:44
I'll award the bounty when it lets me. – ben890 Mar 07 '21 at 15:44
@ben890 Thank you for replying. I'm glad your issue was resolved. – Tanaike Mar 08 '21 at 00:23
Hi Tanaike, how would I modify this to work with excel files? Additionally, how do I pull down files from Drive into dataframes? – ben890 Mar 25 '21 at 15:42
@ben890 About your new 2 questions, I would like to support you. But those are new issues, and that is different from your question. So can you post it as new question? Because when your initial question is changed by comment, other users who see your question are confused. By posting it as new question, users including me can think of it. If you can cooperate to resolve your new issue, I'm glad. Can you cooperate to resolve your new question? – Tanaike Mar 26 '21 at 01:25
posted https://stackoverflow.com/questions/66819224/add-xlsx-documents-to-google-drive-shared-folder-using-python – ben890 Mar 26 '21 at 14:53
1

@ben890 Thank you for your response. Now I noticed that an answer has already been posted. In this case, I would like to respect the existing answer. It will resolve your issue. – Tanaike Mar 27 '21 at 01:19
Hi Tanaike, The answer doesn't actually work for me. I'm unable to look at the file on Google Drive and when I download it, it appears to be corrupted. – ben890 Mar 29 '21 at 03:08
@ben890 Thank you for replying. When I saw the last login time of the answerer, I thought that your comment might not be still seen. So how about waiting it? Because I would like to respect the existing answer. I deeply apologize for this. – Tanaike Mar 29 '21 at 12:58

Python Job on Linux EC2 Machine that drops file on Google Drive

1 Answers1

Modification points:

Modified script:

Note:

References: