1

When using Pandas v0.25 and greater, I can generate aggregate columns with custom names using an elegant native syntax:

import pandas as pd

data = {"x": [1, 1, 2], "y": [-1, 0, 0]}
pd_df = pd.DataFrame(data)

this_is_valid = pd_df.groupby("x").agg(my_custom_name=("y", "mean"))

However, when I try to do the same using a Dask dataframe, I get an "unexpected keyword argument" error.

import dask.dataframe as dd
dask_df = dd.from_pandas(pd_df, npartitions=1)
blows_up = dask_df.groupby("x").agg(my_custom_name=("y", "mean"))

Is there a straightforward way to accomplish this using Dask (and without resorting to the rename function), or is this functionality just not supported? If not, is there a fundamental reason why it cannot be supported?

David Bruce Borenstein
  • 1,655
  • 2
  • 19
  • 34

1 Answers1

2

Dask has docs on performing custom aggregations. They discuss the case of computing the mean, and how it's more complex than the pandas counterpart:

Many reductions can only be implemented with multiple temporaries. To implement these reductions, the steps should return tuples and expect multiple arguments. A mean function can be implemented as:

custom_mean = dd.Aggregation(
    'custom_mean',
    lambda s: (s.count(), s.sum()),
    lambda count, sum: (count.sum(), sum.sum()),
    lambda count, sum: sum / count,
)
df.groupby('g').agg(custom_mean)

This hints at the complexity involved in handling all types of user-defined aggregation, but provides a pretty good overview of how to implement them.

As for renaming the column, I don't see a way to do that in one step (at the moment). Could be wrong about this, and I'm sure this will probably change in the future. Complex reshape operations in dask are significantly different from their pandas counterparts because they need to work with data partitions and account for a variety of data locations, so it's not trivial to replicate the full pandas API. Balancing performance considerations on a laptop, a distributed cluster, and a high-performance computing facility (the range of dask deployments is quite broad) with the many feature requests from users is a real challenge for the dask developers.

Generally, the answer "why does this not exist" for open source projects is "if you want to contribute it, PRs are welcome!". See dask's development guidelines for a nice intro to contributing.

Michael Delgado
  • 13,789
  • 3
  • 29
  • 54