1

The line with fillna() raises the warning even though it is not performed inplace. Why is that?

import pandas as pd
import numpy as np


tuples = [('foo', 1), ('foo', 2), ('bar', 1), ('bar', 2)]
index = pd.MultiIndex.from_tuples(tuples)

df = pd.DataFrame(np.random.randn(2, 4), columns=index)
df.loc[0, ('foo', 1)] = np.nan

# this works without warning
# df = pd.DataFrame({'foo': [1, np.nan, 3], 'bar': [np.nan, 22, 33]]})  

df1 = df[['foo', 'bar']]
# df1 = df[['foo', 'bar']].copy()  # this does not help
filled = df1.fillna({'foo': 100, 'bar': 200}, inplace=False)

The problem does not appear if foo and bar are ordinary columns, not multiindexed.

cs95
  • 379,657
  • 97
  • 704
  • 746
rafgonsi
  • 83
  • 7

1 Answers1

0

This is a false positive, the warning is not supposed to be raised here. I think the issue is that fillna doesn't understand that "foo" and "bar" apply to specific levels of your MultiIndex columns.

I would recommend calling fillna inside GroupBy as a workaround until this functionality is implemented.

fill = {'foo': 100, 'bar': 200}
df1.groupby(level=0, axis=1).apply(lambda x: x.fillna(fill[x.name]))

          foo                 bar          
            1         2         1         2
0  100.000000  1.040531 -1.516983 -0.866276
1   -0.055035 -0.107310  1.365467 -0.097696

Alternatively, to use fillna directly, specify a dict of tuples (because, MultiIndex),

df1.fillna({('foo', 1): 100, ('foo', 2): 100})

          foo                 bar          
            1         2         1         2
0  100.000000  1.040531 -1.516983 -0.866276
1   -0.055035 -0.107310  1.365467 -0.097696
cs95
  • 379,657
  • 97
  • 704
  • 746
  • Thank you. I applied the second solution and it works :) But actually, it is not false positive. Under the hood, pandas performed inplace na filling, which can be tracked down in debugger. In some cases it left me with nas unfilled. – rafgonsi Jun 03 '19 at 13:06
  • @Rafcik It is a false positive in the sense that the [SettingWithCopyWarning was developed to warn pandas users against bad practices](https://stackoverflow.com/questions/20625582/how-to-deal-with-settingwithcopywarning-in-pandas/53954986#53954986) such a chained assignment. No such thing is going on here. – cs95 Jun 03 '19 at 13:13