pandas read_csv path issue

Question

I can easily read a csv file if I go:

pd.read_csv(r'C:\\yourpath\yourcsvfile.csv')

But I wanna try path variables instead of hardcode my path. So I tried:

processed_data_path=os.path.join(os.path.pardir,'data','processed')
train_file_path=os.path.abspath(os.path.join(processed_data_path,'train.csv'))
test_file_path=os.path.abspath(os.path.join(processed_data_path,'test.csv'))

the print result is:

C:\yourpath\train.csv
C:\yourpath\test.csv

So it looks alright. But when I tried the code below it throws a file does not exist error:

train_df = pd.read_csv(train_file_path)

where is the problem?

forgot to mention that I'm using windows with python 3.6 – Gene Xu Apr 01 '18 at 05:03 — Gene Xu, Apr 01 '18 at 05:03
show the value that you are assigning to `train_file_path` – Vivek Kalyanarangan Apr 01 '18 at 05:12 — Vivek Kalyanarangan, Apr 01 '18 at 05:12

score 2 · Accepted Answer · answered Apr 01 '18 at 10:12

I would use pathlib module, which is very convenient and natively supported by modern Pandas versions:

Demo:

try:
    from pathlib import Path
except ImportError:             # Python 2
    from pathlib2 import Path

In [49]: p = Path('D:\\')

use overloaded operator / for joining paths:

In [50]: f = p / 'temp' / '.data' / '1.txt'

In [51]: f
Out[51]: WindowsPath('D:/temp/.data/1.txt')

In [52]: pd.read_csv(f)
Out[52]:
        Val
0  0.120000
1  0.320000
2  0.439999
3  0.560000
4  0.599999

score 1 · Answer 2 · answered Apr 01 '18 at 05:14

Maybe adding a check before calling the parser will help you to identify the issue and it is always a good pratice to check path before parsing to make sure we dont't hit the wall of exceptions :

if os.path.exists(train_file_path):
           train_df = pd.read_csv(train_file_path)
else:
           print(train_file_path)

score 0 · Answer 3 · answered Sep 09 '20 at 14:12

For VSCode users: The issue might be that VSCode sets current working directory to workspace root folder. My solution was:

To set current working directory to whatever file you are executing at the time:

File > Preferences > Settings > Python > Data Science > Execute in File Dir

Thanks to user brch: Python in VSCode: Set working directory to python file's path everytime

pandas read_csv path issue

3 Answers3