pandas: read_csv misses final lines

Asked Oct 10 '19 at 13:12

Active Oct 17 '19 at 13:22

Viewed 173 times

I'm reading a csv file which contains only numerical data, but not all of them are being read.

I'm calling it as follows:

df = pd.read_csv(file_name, sep=';', usecols=Settings.util_columns(), parse_dates=['TIMESTAMP'], infer_datetime_format=True, low_memory=False, quoting=csv.QUOTE_NONE)
print(f'Reading {file_name}')
print(df['TIMESTAMP'].min(), '|', df['TIMESTAMP'].max())

and it outputs the following:

Reading clean_data.csv
2008-12-03 20:30:00 | 2018-10-05 03:50:00

However, clean_data.csv (which is sorted by 'TIMESTAMP') contain TimeStamps up to 2019-08-27.

In addition to this, no warning nor error message is thrown/shown.

I found This question from where I took the quoting=csv.QUOTE_NONE, however it didn't change anything. Consider that my data is numerical.

Using:

python==3.6.6
pandas==0.25.1

UPDATE

My apologize, it was my fault. It turn out that the file which I was reading was not the one I was expecting to read.

Once again, I'm sorry, and thanks for your time.

edited Oct 17 '19 at 13:22

asked Oct 10 '19 at 13:12

Rodrigo Laguna

1,796
1
26
46

3

By any chance, would you be able to share the csv? It's hard to help you if we can't reproduce the issue. – adrianp Oct 10 '19 at 13:19
I'm not sure if I can share it, it's very large (gigabytes of data) :/ – Rodrigo Laguna Oct 10 '19 at 19:25
does `wc -l clean_data.csv` correspond to `len(df)+1`? (for Windows see [here](https://stackoverflow.com/a/5665718/3944322)) – Stef Oct 10 '19 at 20:13
`len(df)` is 413992 `wc -l clean_data.csv` is 455429 I'm working in linux, but the files were generated in windows – Rodrigo Laguna Oct 17 '19 at 12:32
Could you show us, results of command `tail `? – 404pio Oct 17 '19 at 13:30

pandas: read_csv misses final lines

0 Answers0