12

I read a csv file using pandas:

data_raw = pd.read_csv(filename, chunksize=chunksize)
print(data_raw['id'])

Then, it reports TypeError:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'TextFileReader' object has no attribute '__getitem__'

What can I do to resolve the problem? And how can I change the data_raw into a dataFrame object? I use the python2.7 and pandas v0.19.1

Long Ye
  • 147
  • 1
  • 1
  • 13
  • Show your csv file. It is not clear what your objective is. Make it clear what are you trying to do. `data_raw` is already a `DataFrame` object. Check with `print(type(data_raw))` – Mohammad Yusuf Jan 25 '17 at 06:03
  • Thanks. But the type of data_raw is TextFileReader because of the chunk. (http://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html) . You can also see my another question (http://stackoverflow.com/questions/41843342/typeerror-when-make-a-dataframe-from-panda-read-csv/41847564#41847564). The purpose of the code is read a big csv file(4GB) into a dataFrame. But the RAM of the computer is just about 3GB. – Long Ye Jan 25 '17 at 09:17

3 Answers3

19

When you pass chunksize option to read_csv(), it creates a TextFileReader reader - an open-file-like object that can be used to read the original file in chunks. See usage example here: How to read a 6 GB csv file with pandas When this option is not provided, the function indeed reads the file content.

Community
  • 1
  • 1
DYZ
  • 55,249
  • 10
  • 64
  • 93
  • Thank you very much. But I still don't know how to modify the code if I want to read the data into a DataFrame. Could you help me? BTW the RAM of computer is just 3GB. And CSV file is more than 3GB. – Long Ye Jan 25 '17 at 09:31
  • And I have read the "How to read a 6 GB csv file with pandas". I use the pd.DataFrame.append(chunk, ....) in a for cycle sentence. But it reports "You haven't enough memory "error. – Long Ye Jan 25 '17 at 09:34
18

One way around this problem is to set nrows parameter in pd.read_csv() function and that way you select subset of data you want to load into the dataframe. Of course, drawback is that you wont be able to see and work with full dataset. Code example:

data = pd.read_csv(filename, nrows=100000)
Mihajlo T.
  • 326
  • 3
  • 8
1

You can convert the TextFileReader to a Dataframe. For small data, use:

df = pd.concat(MyTextFileReader, ignore_index=True)

See How to read data in Python dataframe without concatenating?, also for a solution for large data.

questionto42
  • 7,175
  • 4
  • 57
  • 90