0

What is the best method to get the simple descriptive statistics of any column in a dataframe (or list or array), be it nested or not, a sort of advanced df.describe() that also includes nested structures with numerical values.

In my case, I have a dataframe with many columns. Some columns have a numerical list in each row (in my case a time series structure), which is a nested structure.

Such nested structures are meant:

  • list of arrays,
  • array of arrays,
  • series of lists,
  • dataframe with nested lists of numerical values in some columns (my case)

How to get the simple descriptive statistics from any level of the nested structure in one go?

Asking for

df.describe()

will give me just the statistics of the numerical columns, but not those of the columns that include a list with numerical values. I cannot get the statistics just by applying

from scipy import stats
stats.describe(arr)

either as it is the solution in How can I get descriptive statistics of a NumPy array? for a non-nested array.

questionto42
  • 7,175
  • 4
  • 57
  • 90

1 Answers1

0

My first approach would be to get the statistics of each numerical list first, and then take the statistics of that again, e.g. the mean of the mean or the mean of the variance would then give me some information as well. In my first approach here, I convert a specific column that has a nested list of numerical values to a series of nested lists first. Nested arrays or lists might need a small adjustment, not tested.

NESTEDSTRUCTURE = df['nestedColumn']

[stats.describe([a[x] for a in [stats.describe(x) for x in NESTEDSTRUCTURE]]) for x in range(6)]

gives you the stats of the stats for a nested structure column. If you want the mean of all means of a column, you can use 

stats.describe([a[2] for a in [stats.describe(x) for x in NESTEDSTRUCTURE]])

as position 2 stands for "mean" in

DescribeResult(nobs=, minmax=(, ), mean=, variance=, skewness=, kurtosis=)

I expect that there is a better descriptive statistics approach that should also automatically understand nested structures with numerical values, this is just a workaround.

questionto42
  • 7,175
  • 4
  • 57
  • 90
  • 1
    No better way. Stats expect uniform numeric arguments, like a numeric numpy array. Rethink why you have nasty nested structures. They don't make computations faster or easier. – hpaulj Jun 15 '20 at 10:22
  • I could also just convert the nested structure into a df/array with one column per each previous list item. As my nested structures cover time series data, it is just not so useful to have houndreds of column statistics then if it is rather about exploring the time series data on the most general level. Then again your answer makes sense, that there is no other way, it is just about what detail of statistics you need: either on *list item level*, or on *list aggregation level*, and there might be no automated method because it is just a decision you have to make. – questionto42 Jun 15 '20 at 10:40