pandas.read_excel() output OverflowError: Python int too large to convert to C int

Question

I am using ubuntu 18.04, pandas==1.2.1

my excel file looks something like this

seq           userid      point     .....
2.01e^+12       A        231231.15
2.012e^+12      B          123
2.0131e^+12     C           3
2.41e^+12       D         2312
2.41e^+12       E         31.15

max(seq) = 2.41e^+12 max(point) = 231231.15

When I to pd.read_excel("file_name.xlsx") it outputs error message in the title.

From resources:

OverflowError: Python int too large to convert to C long torchtext.datasets.text_classification.DATASETS['AG_NEWS']() -> tells me I need to change csv.field_size_limit to sys.maxsize however couldn't find out how to change excel.field_size_limit
"OverflowError: Python int too large to convert to C long" on windows but not mac -> tells me I need to set seq, point columns to float datatype. which i did using pd.read_excel("file_name.xlsx", converters={'seq':float, "point":float}) however didn't fix.

When I remove two float columns I can read excel file. How can I fix this error?

Can you clarify resources #1. Not sure what you mean by you "couldn't figure out how to change r" — user1558604, Mar 17 '21 at 01:29
do you know what the maximum value is in each column? And are they ints or floats? — user1558604, Mar 17 '21 at 01:36
#2 should have worked, but if they are ints I would use dtype of "UInt64" rather than float. — user1558604, Mar 17 '21 at 01:52
@user1558604 I've edited max values and `converters={"seq":"UInt64"}) still outputs same error message. — haneulkim, Mar 17 '21 at 03:09
Oh, don't use `converters` use `dtype`. I believe with dtype set, the value is read in as that dtype. If converter is set I think it reads in as default then it attempts to convert. — user1558604, Mar 17 '21 at 03:14
@user1558604 are you saying use dtype like `pd.read_excel(dtype="UInt64")` ? it outputs same error message and also same with dtype=str, int, and float — haneulkim, Mar 17 '21 at 05:13
Yeah, that should have worked... Are you using 32 bit python or 64? — user1558604, Mar 17 '21 at 22:08
I had the same problem. In my case, excel assigned a date type to a number column and pandas was trying to convert a 13 digits number to date. Changing the data type in excel from "date" to "general" solved the issue. — Diego, Jun 13 '22 at 18:04

score 0 · Answer 1 · answered Feb 04 '22 at 09:50

0

If you check the options for importing data on pydata.org you'll options like convert_dates=True you can toggle these on or off until your file reads. Assuming the problematic data is a date.

answered Feb 04 '22 at 09:50

KB_Mann

1

pandas.read_excel() output OverflowError: Python int too large to convert to C int

1 Answers1