2

I'm currently working on a project where I write output from a pandas dataframe to a .csv file at each iterative step in the process of running a simulation. However, seemingly at random, the program will break and I will get the error:

[Errno 13] Permission denied: 'C:\Users\samue\Desktop\Lab Python\nm12_diam1_pen100000_lam1.5_e-2_a1.75_dt0.0001_rt1000000_T300_drag1_k100_confFalse_2020-09-11 23_11_53.110105.csv'

This is especially weird because it seems to work for anywhere from 500 to 500,000 iterative steps before I get this error message--it seems completely random. Here's the code I'm using to write output:

#name = name of .csv file
path = os.getcwd()
full_name = path + '\\' + name

#if time-point = 0, write the file    
if t[0]==0:
    df.to_csv(full_name, mode='w', index=False)

#if time-point != 0, append the file with new data
else:
    with open(full_name, 'a+') as f:
        df.to_csv(f, mode='a+', header=False, index=False, encoding="utf-16")
        f.close()

I've checked if this is an error with having a read-only folder as the location to which I'm writing the file, but this isn't the case. It's also not an issue with the way that I'm naming the the .csv file (I've tried shorter names (e.g. test.csv) and it doesn't fix the issue).

I also thought it might be an issue of not closing the file after opening it, so I've added f.close(), but this hasn't made it work any more consistently.

I've also tried changing the mode from 'a' to 'a+' and it seems to be functionally the same.

Does anyone have any ideas?

shleemplumbis
  • 21
  • 1
  • Is this the full error message? If not, please post the full message, – DYZ Sep 12 '20 at 18:26
  • 1
    This may or may not be related: You don't have to explicitly `close()` the file because you're using `with`. Here's some more info: https://stackoverflow.com/q/3012488/843953 – Pranav Hosangadi Sep 12 '20 at 18:27
  • 1
    Also, your `if t[0] == 0:...` is unnecessary. Opening a file in `a` appends to a file, but if one doesn't exist it creates a new file. The `a+` makes it a read/write operation. More info here: https://stackoverflow.com/q/1466000/843953 – Pranav Hosangadi Sep 12 '20 at 18:32
  • 1
    Also also, you don't even need to open the file before giving it to `pandas.to_csv()` can take the file path as a string and handle everything for you. https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_csv.html – Pranav Hosangadi Sep 12 '20 at 18:35
  • @DYZ ` ~\Anaconda3\lib\site-packages\pandas\io\formats\csvs.py in save(self) 182 close = False 183 else: --> 184 f, handles = get_handle( 185 self.path_or_buf, 186 self.mode, ~\Anaconda3\lib\site-packages\pandas\io\common.py in get_handle(path_or_buf, mode, encoding, compression, memory_map, is_text) 426 if encoding: 427 # Encoding --> 428 f = open(path_or_buf, mode, encoding=encoding, newline="") 429 elif is_text: ` – shleemplumbis Sep 12 '20 at 21:07
  • @PranavHosangadi I've gotten rid of f.close(), and done your suggestions but I'm receiving the same error. – shleemplumbis Sep 12 '20 at 21:10
  • It seems to work more consistently if I don't have any other csv files in the directory that I'm writing to. Not sure if this helps... – shleemplumbis Sep 12 '20 at 21:15
  • Why do you use two different encodings in the same file? This may be the reason. – DYZ Sep 12 '20 at 21:40
  • @DYZ I've changed the script such that the encoding is always utf-16, but I get the same error: if t[0]==0: df.to_csv(full_name, mode='w', index=False, encoding="utf-16") else: df.to_csv(full_name, mode='a+', header=False,index=False, encoding="utf-16") – shleemplumbis Sep 12 '20 at 21:46

1 Answers1

-1

Try this:

if t[0]==0:
    df.to_csv(full_name, mode='w', index=False)
else:
    df.to_csv(f, mode='a+', header=False, index=False, encoding="utf-16")

You are opening the file with a+ and then again your writing using df.to_csv again with a+. Opening file multiple times with different mode.

Zephyr
  • 11,891
  • 53
  • 45
  • 80
Alim Shaikh
  • 79
  • 3