9

I'm using jupyter notebook pandas to_csv doesn't output the dataframe to a file.

I tried to use to_csv to output a dataframe to a csv file by setting the working directory or specify the directory, and it didn't create any file. The code ran and didn't produce any error message, but when I opened the folder, there was no such file.

I tried a different IO, and it did show the result had been output.

from io import StringIO

output = StringIO()

a.to_csv(output)

print(output.getvalue())

I got the following output:

,a

0,1

1,2

2,3

but again to_csv('filepath/filename.csv') doesn't output any file.

PS: I can read any file in any directory using read_csv().

Update

If I save the file df.to_csv('testfile.csv') then do pd.read_csv('testfile.csv')

I can read the file but cannot see it in the directory.

Also, doing [x for x in os.listdir() if x == 'testfile.csv'] will list the file.

Community
  • 1
  • 1
Li Ai
  • 201
  • 1
  • 3
  • 10
  • try `to_csv('file.csv')` and just use a relative path? it should show up in your folder – Primusa Jan 17 '19 at 01:28
  • are you use mac, windows, linux, etc.? – It_is_Chris Jan 17 '19 at 01:34
  • I'm using windows. I tried every possible way I could think about for to_csv. But now file showed up in the folder. – Li Ai Jan 17 '19 at 01:39
  • 1
    Have you tried an absolute path? The current directory could be different from what you might think it is. – busybear Jan 17 '19 at 01:40
  • I tried 1. absolute directory, 2. use os.chdir() to specify a directory, 3. just use to_csv('filename.csv'). it didn't work in any cases. – Li Ai Jan 17 '19 at 01:42
  • @LiAi what about trying `df.to_csv(os.getcwd()+'\\file.csv')` or `df.to_csv(r'some\file\path\file.csv')` are you viewing a cached version of a network driver? – It_is_Chris Jan 17 '19 at 01:57
  • Did you refresh the output directory? – James Chang Jan 17 '19 at 01:58
  • @Chris just tried both neither worked. So strange - never experienced this before. – Li Ai Jan 17 '19 at 02:13
  • @LiAi if you are not getting an error try reading the saved file (even though you cannot see it in the directory) to see if it exists: first `df.to_csv('testfile.csv')` then `pd.read_csv('testfile.csv')` – It_is_Chris Jan 17 '19 at 02:17
  • @Chris df.to_csv('testfile.csv') then pd.read_csv('testfile.csv') works. I can read the file but cannot see the file. – Li Ai Jan 17 '19 at 02:28
  • @LiAi does `[x for x in os.listdir() if x == 'testfile.csv']` list the file in the directory? Also, because you can read the file, it does not appear to be a python/pandas issues. I can give you the cliche "did you try turning off and on again"? – It_is_Chris Jan 17 '19 at 02:32
  • [x for x in os.listdir() if x == 'testfile.csv'] did list the filename. i did try restarting the computer but it still didn't show up. about to give it a second try. the second try still didn't show the file. – Li Ai Jan 17 '19 at 02:49
  • @LiAi hmmm try opening up the control panel and navigate to `Appearance and Personalization` then under `File Explorer Options` click `Show hidden files and folders` Check `Show hidden files, folders, and drives` and make sure `Hide extensions for known file types` is not checked then apply to folders – It_is_Chris Jan 17 '19 at 02:55
  • @Chris tried what you suggested but still no luck. I can still read the file. Really weird. – Li Ai Jan 17 '19 at 04:02
  • @LiAi this is truly unusual. I wish I could help but that is all I can think of trying. Hopefully someone else will chime in an help. Best of luck to you. – It_is_Chris Jan 17 '19 at 04:56
  • I was using jupyter notebook from Anaconda. Tried using terminal windows and spider to do the same thing and they both worked! so I will trying reinstalling Anaconda. – Li Ai Jan 17 '19 at 14:00
  • it looks really weird now....I reinstalled Anaconda and I still cannot see the files in the folder. BUT, when I opened the Jupyterlab (not jupyter notebook) from Anaconda, I can see all the files I've saved....What happend?? – Li Ai Jan 17 '19 at 14:32
  • Prob solved for myself. See user12352611's anwser. – Li Ai Nov 12 '19 at 02:00

7 Answers7

8

I think the issue is that you're running a Jupyter Notebook so the "current directory" for the notebook is probably somewhere in "C:\Users\user_name\AppData...".

Try running os.getcwd() on its own in your notebook. It probably won't be the same folder as where the *.ipynb file is saved. So as @Chris suggested in comments, this:

df.to_csv(os.getcwd()+'\\file.csv')

... will send your csv into the AppData folder.

You could either change the working directory for the Jupyter notebook, or you could use a fully specified filename like:

df.to_csv('C:\\Users\\<user_name>\\Desktop\\file.csv')

(Note: this also tripped me up in VS-Code while using the interactive iPython execution that happens when you press shift+enter. Interestingly, in VS-Code, if you use ctrl+shift+p and select "Python: Run selection..." it executes in your default terminal, which doesn't have this problem.)

Matt Moehr
  • 123
  • 2
  • 10
5

you probably forgot to add the name of the file after your path, so it will named your file as the last character of your path, which you can see on the home page of jupyter.

should be: df.to_csv('path/filename.csv', ....)

rather than df.to_csv('path.csv'......)

Chris
  • 51
  • 1
  • 3
4

I had the same problem using spyder. In my case it was caused by the internet security tool (COMODO) I used, which somehow executed spyder in a sandbox or so and not allowed it to write to the "normal" directories. Instead it saved the result to a folder named C:VTRoot\HarddiskVolume2\users\. You can try to save a file with a quite unique name a.to_csv('very_unique_filename.csv') and then search in the windows explorer for that filename to find the folder it is stored in. If the reason is some tool like this, changing it's settings may help.

user12352611
  • 56
  • 2
  • 1
    Actually I just found the solution you recommended - it turns out jupyter notebook app is contained in C:VTRoot\HarddiskVolume2\users\ and all the files are there. If it's a *.py file, we can unlock it in the conmodo and then it will write to the folder we like. However, I haven't found a way to unlock the jupyter notebook app. – Li Ai Nov 12 '19 at 01:59
2

Maybe you do not have access to your output folder.

First try the current dir, like to_csv('tmp.csv').

Then check the directory's ownership by using ls -l.

jcyan
  • 51
  • 7
  • I tried ls -l and it didn't show the file that should be there. – Li Ai Jan 17 '19 at 13:29
  • You should use this command in the upper folder, you have to check wether you have right to write in the folder. – jcyan Jan 18 '19 at 02:54
2

This will give you a normal document even though you have used to_csv:

df.to_csv(**'df',** index = False)

Make sure to use 'df.csv' this will ensure the output CSV file.

df.to_csv(**'df.csv'**, index = False)
blackbishop
  • 30,945
  • 11
  • 55
  • 76
hk1414
  • 21
  • 1
0

in my case the files were saved to my apps root folder (as expected).

but i needed to restart Jupyter even though i had auto reload enabled:

  %load_ext autoreload

  %autoreload 2

meaning, reload works for most changes, but it did not work when i added df.to_csv until restarting jupyter notebook

Sonic Soul
  • 23,855
  • 37
  • 130
  • 196
0

You may be using PyCharm for some reason when you create a new CSV it doesn't show the created file in the tree, nevertheless, it was created, you should check the file in the terminal. Or simply close and reopen PyCharm.

Carlos Mauricio Vargas Machado
  • 1