0

I am trying to write a python script that writes a Postgres table to a datafile and saves it to the system Google Drive (G:). I would like it to be in google sheet format (.gsheet). But it seems like no matter what I try, my Google Drive won't allow a user to write or move a .gsheet in to the G: drive and it wont allow a file within the google drive to be renamed with a .gsheet extension.

Additional info:

  1. I can write the table to a file with pretty much any extension (including .gsheet) anywhere else on the file structure (ie: C:)
  2. I can write the table to a file with any non-.gsheet extension on the Google Drive (G:)
  3. I can rename any extension to .gsheet in any non-Google Drive (ie: C:) location.
  4. I cannot move any .gsheet file to the G: drive ('file no longer located at location ...' error). Nor can I...
  5. Rename a file in the G: drive with a .gsheet extension ('file too large error').
  6. The craziest of all: If I write a .csv file to the Google Drive, visit the Google Drive online, and select 'Open with GSheet', Google creates a new .gsheet file for viewing, and it appears in my local Google Drive as a .gsheet file.

The 'file no longer located at location ...' errors are accompanied by a system pop up prompting me to log in to my google drive, but since I can write a .csv file to my local Google Drive and it appears when I view it in a browser, aren't I already logged in?

Is there anyway to:

  1. write a .gsheet file to the G: drive
  2. move a .gsheet file to the G: drive
  3. rename a file within the G: drive to a .sheet extension?

I assume I am missing something super obvious, because right now it is looking like I will need to implement the Google Drive API to interact with Google directly to write a .gsheet file to the drive, as opposed to writing it to a synced folder with system functions. At this point, I am more likely to write a .csv file to the G: drive and suffer the extra clicks to 'open in gsheet' in the google drive web view.

Code can be posted upon request.

Thanks

Edit: Code as requested:


import psycopg2 as ppg
import time
import os

db conenction paramters omitted

date = time.strftime("%Y%m%d")

save_e_path = 'G:\\My Drive\\estate\\daily_expense_upload\\expenses_{}.gsheet'.format(date)

save_s_path = 'G:\\My Drive\\estate\\daily_schedule_upload\\schedule_{}.gsheet'.format(date)

expense_file = open(save_e_path, 'a')
schedule_file = open(save_s_path, 'a')

conn = ppg.connect("dbname=\'{}\' user=\'{}\' password=\'{}\'".format(db, user, password))
cur = conn.cursor()

cur.execute("select * from estate.expenses;")
expenses = cur.fetchall()

cur.execute("select * from estate.schedule;")
schedules = cur.fetchall()

for i in expenses:
    expense_file.write('\n')
    for j in i:
        expense_file.write('"' + str(j) + '"' + ',')

for i in schedules:
    schedule_file.write('\n')
    for j in i:
        schedule_file.write('"' + str(j) + '"' + ',')

  • YES! Show us the code please. This is a place where we discuss specific programming issues. This isn't a place to ask to have someone teach you how to code, tell you how to go about solving a fairly complex programming problem, or write code for you. You need to take a real stab at this yourself, and then ask questions here when you get stuck and we can help you to keep moving towards a solution. – CryptoFool Feb 20 '22 at 02:03
  • This is less of a 'what is wrong with my code' promblem and more of a 'is it even possible to write this kind of file to the destination' problem. I know how to write files via python. I know how to put them where I want, but it isnt working for this specific destination for some reason. Thats what I am looking for help with – Timmy Brockett Feb 20 '22 at 02:06
  • 1
    You can go to google developer console and get a access_token for your google drive. Then you can use the token to make request to your google drive through the API. Try this article as a starting point. https://towardsdatascience.com/how-to-manage-files-in-google-drive-with-python-d26471d91ecd – sin tribu Feb 20 '22 at 02:14
  • @sintribu Thanks--I was hoping to avoid the API to write .gsheets to the google drive, but it seems like it might be the only way. Thanks for the reference and confirming Im not totally crazy. – Timmy Brockett Feb 20 '22 at 02:17
  • I honestly don't know if it's the only way or not. Perhaps you can just change write permissions on the G drive. If it's safe to do so that sounds easier. – sin tribu Feb 20 '22 at 02:34
  • @sintribu I have tried and changing permissions doesnt seem to have much effect. If you submit your comment as an answer, I will mark it as accepted unless something else come in over the next day or so. Ive tried everything I can think of to 'sneak' a .gsheet in to the g: drive, but for some reason it is just not working :( – Timmy Brockett Feb 20 '22 at 02:51
  • @TimmyBrockett - fair enough. I redacted my downvote. – CryptoFool Feb 20 '22 at 04:41
  • @CryptoFool Thanks--I take back all the mean things I said about you to the cats. – Timmy Brockett Feb 20 '22 at 06:30

1 Answers1

1

As may be apparent from the comments, you can't just write a .gsheet file to GDrive like that. For starters, it's not even writing it in the format of a GSheet file - not that the actual format is even known (iirc). And clearly, the app is not allowing you to create arbitrary files with a .gsheet extension.

It looks like you're basically writing out your data as a text file, and quite badly. Consider doing it as a CSV dump and then upload to Google Drive, and enable conversion.

See this answer - How to save results of postgresql to csv/excel file using psycopg2? - for a better way to save each of your tables to a CSV file. Btw, do one table at a time instead of loading both into memory with cur.fetchall().

The craziest of all: If I write a .csv file to the Google Drive, visit the Google Drive online, and select 'Open with GSheet', Google creates a new .gsheet file for viewing, and it appears in my local Google Drive as a .gsheet file.

Yes, this would be a way to have your data as a .gsheet file.

To take advantage of the automatic conversion to gsheet without having to use the full GSheet API, read this in the docs - requires using the GDrive API and as an upload instead of copying to local path:

To convert a file to a specific Google Workspace file type, specify the Google Workspace mimeType when creating the file. The following shows how to convert a CSV file to a Google Workspace sheet:

file_metadata = {
    'name': 'My Report',
    'mimeType': 'application/vnd.google-apps.spreadsheet'
}
media = MediaFileUpload('files/report.csv',
                        mimetype='text/csv',
                        resumable=True)
file = drive_service.files().create(body=file_metadata,
                                    media_body=media,
                                    fields='id').execute()
print 'File ID: %s' % file.get('id')
aneroid
  • 12,983
  • 3
  • 36
  • 66
  • (1/2): Thank you for the references and suggestions. I will absolutely look in to them. As mentioned in the comments, it seems the Google API is most straightforward way to get .gsheets files in to a google drive via python. I did consider a pgdump and the more straight forward 'copy to' psql function, but since this was going to run as a scheduled daily script along with other tables and system command functions, I decided to go with a single python script to read out tables. This also eliminates the psql 'fluff' that is included with dump files, even in csv format. – Timmy Brockett Feb 20 '22 at 06:50
  • (2/2): I'm not entirely sure why you think the process for writing a file was poor; you can give a file that is being written by python any extension, .csv or otherwise. If the file name given in the save_x_path variable was .csv would there be an issue? not reading out item-by-item by line-by-line, would give postgres/psycopg2 induced characters (open/close parenthesis on first/last terms and single quotes around every term). Not to mention datetime.date(value) and decimal(value) print-outs for my date and decimal values respectively. – Timmy Brockett Feb 20 '22 at 07:02
  • (3/2) improper fraction...whatever: After checking the link you posted regarding the COPY TO function: That was the first method I used to try to write to the G: drive--and it worked as a .csv file (as stated in the post), but I want to write a .gsheet file to the G: drive. If you try to do that either with python (with psycopg2, os or shutil) or via any windows explorer methods, it will error. The linked method you posted will work since the psql code includes 'CSV HEADER', but if you tell psycopg2 to make the file extension .gsheet, it will error. So it seems the API is the only way to go. – Timmy Brockett Feb 20 '22 at 07:50
  • I meant - use the COPY TO method in the linked post with to create a csv file for yourself, save it in an allowed regular location and then upload that to Google Drive (not GSheets) via the GDrive API, using the code example. That way, you get it as a gsheet and you have to write minimal "csv dump" code yourself. You can call that command from Python too, so it would be part of your daily scheduled Python script; not a separate step. – aneroid Feb 20 '22 at 12:55
  • And when you use Python's [`csv.writer`](https://docs.python.org/3/library/csv.html#csv.writer) or DictWriter, it takes care of the delimiters etc. being correct. In your example, `'"' + str(j) + '"' + ','` would result in a bad csv if a column contained a value like `hello","world` - your expression would make that two columns without escaping that correctly. You've also not put headers in your CSV file, and you start it with a newline. Could've put the newline after the line entry. – aneroid Feb 20 '22 at 12:55
  • Thanks for the suggestion--I will start utilizing the csv.writer python package, it doest seem like a better method. As for (lack of) headers and the initial new line, those were by design. I agree it is not the best for usual situations, but it is preferred in this case. Thanks again. – Timmy Brockett Feb 20 '22 at 16:42
  • You're welcome. And welcome to StackOverflow! If that helped, read: [What should I do when someone answers my question?](https://stackoverflow.com/help/someone-answers), and about [accepting](https://meta.stackexchange.com/a/5235/193893) and [voting](https://stackoverflow.com/privileges/vote-up). – aneroid Feb 20 '22 at 22:30