How big a file will read_csv handle? Works for 10 lines of excel, but not 100000 -- says file doesn't exist

Question

Can't read in a large Excel file using read_csv - python error that file doesn't exist.

Smaller versions of same excel file open easily.

import pandas as pd
data = pd.read_csv("E:\rawdata_50K.csv")
print(data[0:5])

Top 20 lines of excel file load perfectly; the large version does not.

This answer might help: https://stackoverflow.com/a/25962187/6705467 — alexbclay, Aug 07 '19 at 01:23
Possible duplicate of [How to read a 6 GB csv file with pandas](https://stackoverflow.com/questions/25962114/how-to-read-a-6-gb-csv-file-with-pandas) — Tim, Aug 07 '19 at 01:25
@Tim: The problem is the file name, nothing to do with file size. — ShadowRanger, Aug 07 '19 at 03:10

Trenton McKinney · Answer 1 · 2019-08-07T03:00:22.983

2

data = pd.read_csv(r"E:\rawdata_50K.csv")

or

data = pd.read_csv("E:/rawdata_50K.csv")

from pathlib import Path

drive_path = Path('E:/')
file_path = drive_path / 'rawdata_50K.csv'
data = pd.read_csv(file_path)

edited Aug 07 '19 at 03:00

answered Aug 07 '19 at 02:49

Trenton McKinney

To be clear, without the raw-string `r` prefix, the path was to `E:awdata_50K.csv`, which of course does not exist. This is why you *always* use raw strings for Windows paths (and regular expressions). – ShadowRanger Aug 07 '19 at 03:11

score 0 · Answer 2 · answered Aug 07 '19 at 20:58

Thanks so much! The 2nd solution in Answer 5 of the 6GB answer worked well and fast....

Trying suggested methods

import pandas as pd Fileread = pd.read_csv("E:\dataraw.csv", chunksize=500) dfList = [] for df in Fileread: dfList.append(df)

df = pd.concat(dfList,sort=False)

print(df[99950:100000])

and perhaps someone can explain why the same CSV file worked when named as dataraw but did NOT work if renamed rawdata...(????)