Permission denied when pandas dataframe to tempfile csv

Question

I'm trying to store a pandas dataframe to a tempfile in csv format (in windows), but am being hit by:

[Errno 13] Permission denied: 'C:\Users\Username\AppData\Local\Temp\tmpweymbkye'

import tempfile
import pandas

with tempfile.NamedTemporaryFile() as temp:
    df.to_csv(temp.name)

Where df is the dataframe. I've also tried changing the temp directory to one I am sure I have write permissions:

tempfile.tempdir='D:/Username/Temp/'

This gives me the same error message

Edit:

The tempfile appears to be locked for editing as when I change the loop to:

with tempfile.NamedTemporaryFile() as temp:
    df.to_csv(temp.name + '.csv')

I can write the file in the temp directory, but then it is not automatically deleted at the end of the loop, as it is no longer a temp file.

However, if I change the code to:

with tempfile.NamedTemporaryFile(suffix='.csv') as temp:
    training_data.to_csv(temp.name)

I get the same error message as before. The file is not open anywhere else.

I don't know enough about tempfile module to give a proper answer. But have a look here (final post): https://github.com/pandas-dev/pandas/issues/1047 just do df.to_csv(temp) (without .name) — FLab, Apr 07 '17 at 17:19
I think that's a typo in his code, that returns this error: "a bytes-like object is required, not 'str'" — thebigspin, Apr 07 '17 at 17:26
Did you mistakenly leave the file open? This will give you the same error. — Andrew L, Apr 07 '17 at 17:27
I think that is what is happening, as I get the same behaviour when I add the suffix .csv in NamedTemporaryFile, but I am not opening the file anywhere else — thebigspin, Apr 07 '17 at 17:32

score 11 · Answer 1 · answered Jun 21 '19 at 17:46

11

I encountered the same error message and the issue was resolved after adding "/df.csv" to file_path.

df.to_csv('C:/Users/../df.csv', index = False)

answered Jun 21 '19 at 17:46

hm6

340
2
13

1

This would create a permanent csv file (`df.csv`) and not a tempfile – thebigspin Sep 18 '19 at 12:40

score 8 · Accepted Answer · edited May 23 '17 at 11:54

8

Check your permissions and, according to this post, you can run your program as an administrator by right click and run as administrator.

We can use the to_csv command to do export a DataFrame in CSV format. Note that the code below will by default save the data into the current working directory. We can save it to a different folder by adding the foldername and a slash to the file

verticalStack.to_csv('foldername/out.csv').

Check out your working directory to make sure the CSV wrote out properly, and that you can open it! If you want, try to bring it back into python to make sure it imports properly.

newOutput = pd.read_csv('out.csv', keep_default_na=False, na_values=[""])

ref

Unlike TemporaryFile(), the user of mkstemp() is responsible for deleting the temporary file when done with it.

With the use of this function may introduce a security hole in your program. By the time you get around to doing anything with the file name it returns, someone else may have beaten you to the punch. mktemp() usage can be replaced easily with NamedTemporaryFile(), passing it the delete=False paramete.

After export to CSV you can close your file with temp.close().

with tempfile.NamedTemporaryFile(delete=False) as temp:
    df.to_csv(temp.name + '.csv')
    temp.close()

edited May 23 '17 at 11:54

Community

1
1

answered Apr 07 '17 at 16:44

RaminNietzsche

2,683
1
20
34

Running as admin didn't change anything, and I tried changing the tempdir to one where I definitely have write permission – thebigspin Apr 07 '17 at 16:54
@thebigspin I changed my answer – RaminNietzsche Apr 07 '17 at 17:16
@thebigspin try this: with tempfile.NamedTemporaryFile(delete=False) as temp: – RaminNietzsche Apr 07 '17 at 17:52
This works (not sure why!?) but then the file isn't deleted at the end of the loop. Tried adding os.remove(temp.name) but then it says file is in use... – thebigspin Apr 07 '17 at 18:01
@thebigspin I changed my answer. please close your file before delete. – RaminNietzsche Apr 07 '17 at 18:07
Can you edit your answer to include only the relevant info (adding delete=false and then temp.close() after using the file) in case someone else hits the same problem? – thebigspin Apr 07 '17 at 18:16
@thebigspin I add this part and make answer like as step-to-step guide for this error :) – RaminNietzsche Apr 07 '17 at 18:33
`NamedTemporaryFile` has a kwarg `suffix` which you can set to `.csv` so that you just have to use `temp.name` in your answer. – scientific_explorer Apr 01 '21 at 08:04

score 2 · Answer 3 · edited Mar 31 '20 at 22:26

Sometimes, it gives that error simply because there is another file with the same name and it has no permission to delete the earlier file and replace it with the new file.

So either name the file differently while saving it, or
If you are working on Jupyter Notebook or a other similar environment, delete the file after executing the cell that reads it into memory. So that when you execute the cell which writes it to the machine, there is no other file that exists with that name.

score 2 · Answer 4 · answered May 08 '18 at 07:37

2

Sometimes，you need check the file path that if you have right permission to read and write file. Especially when you use relative path.

xxx.to_csv('%s/file.csv'%(file_path), index = False)

answered May 08 '18 at 07:37

saneryee

3,239
31
22

make sure to CHMOD the folder you are writing to (777) – dataviews Jul 30 '19 at 19:38

score 1 · Answer 5 · answered Aug 23 '21 at 04:03

I encountered the same error. I simply had not yet saved my entire python file. Once I saved my python file in VS code as "insertyourfilenamehere".py to documents(which is in my path), I ran my code again and I was able to save my data frame as a csv file.

score 0 · Answer 6 · answered Oct 20 '20 at 21:20

0

As per my knowledge, this error pops up when one attempt to save the file that have been saved already and currently open in the background.

You may try closing those files first and then rerun the code.

answered Oct 20 '20 at 21:20

Yash Verma

461
5
4

score -5 · Answer 7 · edited Sep 17 '19 at 12:12

-5

Just give a valid path and a file name

e.g:

final_df.to_csv('D:\Study\Data Science\data sets\MNIST\sample.csv')

edited Sep 17 '19 at 12:12

Butiri Dan

1,759
5
12
18

answered Sep 17 '19 at 11:28

Shaurya Singh

1

This is not saving to a tempfile and so does not address the question – thebigspin Sep 18 '19 at 12:36

Permission denied when pandas dataframe to tempfile csv

7 Answers7

Linked