Use this tag for pandas.DataFrame.fillna or pandas.Series.fillna
Fill missing values using the specified method for a DataFrame or a Series with pandas (see also interpolate, ffill, bfill)
Docs:
Questions tagged [fillna]
528 questions
189
votes
7 answers
Python Pandas replace NaN in one column with value from corresponding row of second column
I am working with this Pandas DataFrame in Python.
File heat Farheit Temp_Rating
1 YesQ 75 N/A
1 NoR 115 N/A
1 YesA 63 N/A
1 NoT 83 41
1 NoY …
edesz
- 11,756
- 22
- 75
- 123
139
votes
7 answers
How to pass another entire column as argument to pandas fillna()
I would like to fill missing values in one column with values from another column, using fillna method.
(I read that looping through each row would be very bad practice and that it would be better to do everything in one go but I could not find out…
xav
- 4,101
- 5
- 26
- 32
133
votes
12 answers
Pandas: filling missing values by mean in each group
This should be straightforward, but the closest thing I've found is this post:
pandas: Filling missing values within a group, and I still can't solve my problem....
Suppose I have the following dataframe
df = pd.DataFrame({'value': [1, np.nan,…
BlueFeet
- 2,407
- 4
- 21
- 24
44
votes
6 answers
Pandas fill missing values in dataframe from another dataframe
I cannot find a pandas function (which I had seen before) to substitute the NaN's in a dataframe with values from another dataframe (assuming a common index which can be specified). Any help?
user308827
- 21,227
- 87
- 254
- 417
37
votes
8 answers
How to Pandas fillna() with mode of column?
I have a data set in which there is a column known as 'Native Country' which contain around 30000 records. Some are missing represented by NaN so I thought to fill it with mode() value. I wrote something like this:
data['Native…
Jim
- 405
- 1
- 4
- 6
33
votes
4 answers
Pandas fillna throws ValueError: fill value must be in categories
Discription: both features are in categorical dtypes. and i used this code in a different kernal of same
dateset was working fine, the only difference is the features are in flote64. later i have converted these feature dtypes into…
Ravi Varma
- 333
- 1
- 4
- 8
25
votes
1 answer
Pandas missing values : fill with the closest non NaN value
Assume I have a pandas series with several consecutive NaNs. I know fillna has several methods to fill missing values (backfill and fill forward), but I want to fill them with the closest non NaN value. Here's an example of what I have:
s =…
Clément F
- 3,535
- 6
- 18
- 26
10
votes
1 answer
Elegant way to fillna missing values for dates in spark
Let me break this problem down to a smaller chunk. I have a DataFrame in PySpark, where I have a column arrival_date in date format -
from pyspark.sql.functions import to_date
values = [('22.05.2016',),('13.07.2010',),('15.09.2012',),(None,)]
df =…
cph_sto
- 7,189
- 12
- 42
- 78
8
votes
2 answers
How to efficiently fillna(0) if series is all-nan, or else remaining non-nan entries are zero?
Given that I have a pandas Series, I want to fill the NaNs with zero if either all the values are NaN or if all the values are either zero or NaN.
For example, I would want to fill the NaNs in the following Series with zeroes.
0 0
1 0
2…
Lien
- 127
- 7
8
votes
4 answers
Filling nulls with a list in Pandas using fillna
Given a pd.Series, I would like to replace null values with a list. That is, given:
import numpy as np
import pandas as pd
ser = pd.Series([0,1,np.nan])
I want a function that would return
0 0
1 1
2 [nan]
But if I try using the…
splinter
- 3,727
- 8
- 37
- 82
7
votes
1 answer
Pandas backfill specific value
I have dataframe as such:
df = pd.DataFrame({'val': [np.nan,np.nan,np.nan,np.nan, 15, 1, 5, 2,np.nan, np.nan, np.nan, np.nan,np.nan,np.nan,2,23,5,12, np.nan np.nan, 3,4,5]})
df['name'] = ['a']*8 + ['b']*15
df
>>>
val name
0 NaN a
1 NaN…
RSHAP
- 2,337
- 3
- 28
- 39
7
votes
2 answers
Conditional forward fill in pandas
I have a dataframe:
>>> k
Out[87]:
Date S E cp Last Q code
30 2017-11-10 22500 2017-11-17 P 170.00 828.47 11/17/2017P22500
32 2017-11-10 22625 2017-11-17 P 180.00 646.91 …
dayum
- 1,073
- 15
- 31
6
votes
1 answer
Pandas forward fill, but only between equal values
I have two data frames: main and auxiliary. I am concatenating auxiliary to the main. It results in NaN in a few rows and I want to fill them, not all.
Code:
df1 = pd.DataFrame({'Main':[00,10,20,30,40,50,60,70,80]})
df1 =
Main
0 0
1 10
2 …
Mainland
- 4,110
- 3
- 25
- 56
6
votes
3 answers
Filling na values with merge from another dataframe
I have a column with na values that I want to fill according to values from another data frame according to a key. I was wondering if there is any simple way to do so.
Example:
I have a data frame of objects and their colors like this:
object …
Kuzenbo
- 229
- 4
- 9
5
votes
2 answers
Filling missing values with mean in PySpark
I am trying to fill NaN values with mean using PySpark.
Below is my code that I am using and following is the error that occurred:
from pyspark.sql.functions import avg
def fill_with_mean(df_1, exclude=set()):
stats =…
John
- 279
- 1
- 3
- 16