3

I am hitting on a corner case in pandas. I am trying to use the agg fn but without doing a groupby. Say I want an aggregation on the entire dataframe, i.e.

from pandas import *
DF = DataFrame( randn(5,3), index = list( "ABCDE"), columns = list("abc") )
DF.groupby([]).agg({'a' : np.sum, 'b' : np.mean } ) # <--- does not work

And DF.agg( {'a' ... } ) does not work either.

My workaround is to do DF['Total'] = 'Total' then do a DF.groupby(['Total']) but this seems a bit artificial.

Has anyone got a cleaner solution?

piRSquared
  • 285,575
  • 57
  • 475
  • 624
joelhoro
  • 890
  • 1
  • 9
  • 20

2 Answers2

4

It's not so great either, but for this case, if you pass a function returning True at least it wouldn't require changing df:

>>> from pandas import *
>>> df = DataFrame( np.random.randn(5,3), index = list( "ABCDE"), columns = list("abc") )
>>> df.groupby(lambda x: True).agg({'a' : np.sum, 'b' : np.mean } )
             a         b
True  1.836649 -0.692655
>>> 
>>> df['total'] = 'total'
>>> df.groupby(['total']).agg({'a' : np.sum, 'b' : np.mean } ) 
              a         b
total                    
total  1.836649 -0.692655

You could use various builtins instead of lambda x: True but they're less explicit and only work accidentally.

DSM
  • 342,061
  • 65
  • 592
  • 494
2

Having an analogous DataFrame.aggregate method is a good idea. Creating an issue here:

https://github.com/pydata/pandas/issues/1623

Wes McKinney
  • 101,437
  • 32
  • 142
  • 108
  • 1
    Cool @Wes MCKinney - perhaps you could also make DF.groupby([]) return DF? Thx – joelhoro Jul 15 '12 at 21:00
  • I also noticed that for some functions, like `np.std` this gives an error: `IndexError: 0-d arrays can't be indexed.`. It seems to choke on the extra structure of a pandas Series. If you use `lambda x: np.std(x.values)` as the passed-in function, this fixes it. But it seems like this should be Pandas responsibility. Maybe a try-catch sort of thing where if doesn't work as-is then also try applying it on column.values before giving up? There may be good reasons for avoiding this; just a suggestion. – ely Jul 15 '12 at 22:38
  • @EMS can you report an issue that reproduces what you're seeing? – Wes McKinney Jul 16 '12 at 16:33