Replacing NaNs with mutable types in a pandas dataframe

Question

In the process of trying to solve another question, I've hit a roadblock. Here's my data:

     col1    col2       col3    col4       col5 user_ID
0     [1]     [3]         []     NaN        NaN       1
1  [2, 3]     [3]  [1, 2, 3]     NaN        NaN       2
2     [3]  [3, 1]     [3, 1]     NaN        NaN       3
0  [1, 2]     NaN        [1]     [3]        NaN       1
1     [3]     NaN     [2, 3]     [3]        NaN       2
2     [3]     NaN        [3]  [3, 1]        NaN       3
0     [1]     [3]        NaN     NaN         []       1
1  [2, 3]     [3]        NaN     NaN  [1, 2, 3]       2
2     [3]  [3, 1]        NaN     NaN     [3, 1]       3

I want to replace those NaNs with an empty list so I can perform a summation along those columns.

I've tried df.replace, but I get

TypeError: Invalid "to_replace" type: 'float'

I also tried df.fillna and got

TypeError: "value" parameter must be a scalar or dict, but you passed a "list"

How can I fill these NaNs with the empty list []?

Edit: So it turns out this is a duplicate! Since the marked dupe doesn't have an applymap solution, I'll keep this here.

@Ev.Kounis No. All the methods I outlined work for scalar values, but not the empty list. — cs95, Aug 23 '17 at 11:49
@Ev.Kounis Updated with some error warnings. It should make sense this isn't a dupe now. — cs95, Aug 23 '17 at 11:51
for the `fillna` it says [here](http://pandas.pydata.org/pandas-docs/version/0.17.0/generated/pandas.DataFrame.fillna.html#pandas-dataframe-fillna) that _"..it cannot be a list"_ but for the `replace` there is no such requirement — Ma0, Aug 23 '17 at 11:53
@Ma0 True that, but replace does not allow `np.nan` to be replaced, as it seems. — lcnittl, Nov 24 '20 at 22:49

cs95 · Accepted Answer · 2017-08-23T12:04:26.663

A little dirty, but I solved this using df.applymap:

In [671]: df.applymap(lambda x: [] if x != x else x)
Out[671]
     col1    col2       col3    col4       col5  user_ID
0     [1]     [3]         []      []         []        1
1  [2, 3]     [3]  [1, 2, 3]      []         []        2
2     [3]  [3, 1]     [3, 1]      []         []        3
0  [1, 2]      []        [1]     [3]         []        1
1     [3]      []     [2, 3]     [3]         []        2
2     [3]      []        [3]  [3, 1]         []        3
0     [1]     [3]         []      []         []        1
1  [2, 3]     [3]         []      []  [1, 2, 3]        2
2     [3]  [3, 1]         []      []     [3, 1]        3

Replacing NaNs with mutable types in a pandas dataframe

1 Answers1