fillna function to replce NaN in a dataframe raising IOPub data rate exceeded

Question

New to Python and working my way through a Panda import and cleanse.

My code:

df = pd.read_csv('SFIC_RFQs.csv', sep='~', usecols=[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19, 20,21,22,23,24,25,26,27,28,29,30, 31,32])
df.isnull().sum().sum()               # Total Number of NaN = 14594
df.fillna(0)

Error:

IOPub data rate exceeded The notebook server will temporarily stop sending output to the client in order to avoid crashing it. To change this limit, set the config variable --NotebookApp.iopub_data_rate_limit.

I changed C:\Users\pelucas\.jupyter\ jupyter_notebook_config from

#c.NotebookApp.iopub_data_rate_limit = 1000000

to

#c.NotebookApp.iopub_data_rate_limit = 100000000000

Same error

I amended the fillna command to df.iloc[0:1500,0:33].fillna(value=0,inplace=true) so as to see if it worked, It did but as soon as I oushed up to df.iloc[0:1600,0:33] the error above was raised again.

The fillna function is displaying the results of it's action in the Jupyter Notebook which I guess is the issue.

Is there a way to force Jupyter to not show the results of the fillna command? Or if it has to just display plain text?
I imported 33 columns (0:33) but my fillna command has 0:33 impying 34 coluns or is the zero element an internal index?
My read_csv looks a little long, can I abbreviated the column numbers i.e. 0:33?

Try https://stackoverflow.com/questions/23692950/how-do-you-suppress-output-in-ipython-notebook — Ilja Everilä, Jan 18 '18 at 08:30

score 0 · Accepted Answer · answered Jan 18 '18 at 08:34

0

First of all: if you want to use the output of fillna you should add inplace=True to your command or assign this line to another variable. If you'll assign it to antoher variable (new_df = df.fillna(0)) you will also avoid displaying the output.

Instead you can add another line to the block with output that can help you understand the data:

new_df.sample(50)

or

new_df.head(50)

answered Jan 18 '18 at 08:34

AndreyF

1,798
1
14
25

Thanks AndreyF,. Works well. cheers df_No_NaN = df.fillna(0) has no output and df_No_NaN.isnull().sum().sum() is zero – Pete Lucas Jan 18 '18 at 08:53
So no other way to use df.fillna(0) without outputting the result and raising that i/O error? – Pete Lucas Jan 18 '18 at 08:56

fillna function to replce NaN in a dataframe raising IOPub data rate exceeded

1 Answers1