Sending multiple .CSV files after zipping them and without saving to disk

Question

I am trying to retrieve files from the internet, save them as dataframe (without saving the files on computer), do some calculation, save resulting dataframes in the required format as csv files, zip them and send them back to the orginal source(see below my code). This works perfectly fine but as I have to send hundred of files and was wondering if there is a way of saving the dataframe to the csv file and then zipping it on the go (i.e. without saving it on computer)? Below is my code;

file_names=['A','B','C']
hdll_url = 'http://XYZ'
saving_folder_url = 'C:\Users\Desktop\Testing_Folder'

general = []
for i in file_names:
    url = ('http://')
    data1 = requests.get(url)
    Z1 = zipfile.ZipFile(StringIO.StringIO(data1.content))
    x=pd.read_csv(Z1.open('%s.tsv'%i), sep = '\t', names=["Datetime","Power"])
    x['Datetime'] = pd.to_datetime(x['Datetime'])
    x = x.set_index("Datetime")
    x = x.resample('H', how= 'sum')
    general.append(x)

ABC = pd.DataFrame(general[0]['Power'] + general[1]['Power'] + general[3]['Power'] * 11.363)
ABC.to_csv('%s\ABC.tsv'%saving_folder_url,
                    sep='\t',header=False,date_format='%Y-%m-%dT%H:%M:%S.%f')
ABC_z1 = zipfile.ZipFile('%s\ABC.zip'%saving_folder_url,mode='w')
ABC_z1.write("%s\ABC.tsv"%saving_folder_url)
ABC_z1.close()

ABC_files1 = {'file': open("%s\ABC.zip"%saving_folder_url,'rb')}
ABC_r1 = requests.put(hdll_url, files=ABC_files1, headers = {'content-type':'application/zip',
                        'Content-Disposition': 'attachment;filename=ABC.zip'
                                   } )

XYZ = pd.DataFrame(general[0]['Power'] + general[1]['Power']*100 )
XYZ.to_csv('%s\XYZ.tsv'%saving_folder_url,
                    sep='\t',header=False,date_format='%Y-%m-%dT%H:%M:%S.%f')
XYZ_z1 = zipfile.ZipFile('%s\XYZ.zip'%saving_folder_url,mode='w')
XYZ_z1.write("%s\XYZ.tsv"%saving_folder_url)
XYZ_z1.close()

XYZ_files1 = {'file': open("%s\XYZ.zip"%saving_folder_url,'rb')}
XYZ_r1 = requests.put(hdll_url, files=XYZ_files1, headers = {'content-type':'application/zip',
                        'Content-Disposition': 'attachment;filename=XYZ.zip'
                                   } )

P.S: I am only allowed to send files in .zip format.

score 0 · Answer 1 · answered Jun 20 '16 at 16:01

0

Have a look at BytesIO():

memory_file = BytesIO()
with zipfile.ZipFile(memory_file, 'w') as zf:
    zi = zipfile.ZipInfo('your_filename.txt')
    zi.date_time = time.localtime(time.time())[:6]
    zi.compress_type = zipfile.ZIP_DEFLATED
    zf.writestr(zi, 'your file content goes here')

memory_file.seek(0)

Then hand it to the put method:

requests.put(<your_url>, data=memory_file, header=<your_headers>)

Or something along the lines of that. I didn't test the code, but the first part taken from a flask app, which uses send_file() to hand the file over to the client.

answered Jun 20 '16 at 16:01

nucleon

1,128
1
6
19

thanks, but with this method, I still will be saving files as csv/.txt files before compressing them? – Muhammad Jun 20 '16 at 16:07
Nothing will be written to your hard disk. The BytesIO object lives in memory. – nucleon Jun 20 '16 at 16:11
but it is getting 'your_filename.txt' from somewhere, no? – Muhammad Jun 20 '16 at 16:15
yeah, it was just an example. you have to replace it in your script according to the context. by the way, have a look at http://stackoverflow.com/questions/25064347/sending-multiple-csv-files-to-zip-without-storing-to-disk-in-python – nucleon Jun 20 '16 at 16:16
'your_filename.txt' is the file entry, which is added to the zip archive. the file content is given explicity as 'your file content goes here'. – nucleon Jun 20 '16 at 16:26

Sending multiple .CSV files after zipping them and without saving to disk

1 Answers1