Python: List of Dataframes

Question

I have a list of Dataframes, each dataframe has datetimeindex with 6 columns. I would like to make all dataframes in one by calculating the daily mean.

data:

[                               M1       M2          M3          M4        M5       M6
 sensor                                                                   
 2020-07-03 08:00:00+00:00    3.616141   3.362717  1.627347    2.242732   2.095574  1.013824
 2020-07-03 08:15:00+00:00    4.043727   3.749407  1.790467    2.272293   2.118229  1.011966 
 2020-07-03 08:30:00+00:00    3.872196   3.595969  1.729359    2.221447   2.073431  0.997198
 ...                               ...        ...       ...         ...   
 2020-09-25 08:30:00+00:00    8.996401   1.716477  0.099816    5.604559   1.311362  0.076378
 2020-09-25 08:45:00+00:00    6.645853   1.352785  0.081961    4.112518   1.019947  0.061641
 2020-09-25 09:00:00+00:00    6.066697   1.068805  0.058980    3.991505   0.861015  0.047540 
      
 [2204 rows x 6 columns],
                                 M1       M2          M3          M4        M5       M6
 sensor                                                                   
 2020-02-02 00:00:00+00:00    7.981656   0.047279  0.000461    1.417066   0.010246  0.000100
 2020-02-02 00:15:00+00:00   10.735810   0.063593  0.000620    1.617664   0.011697  0.000114
 2020-02-02 02:15:00+00:00    5.957599   0.035289  0.000344    1.228983   0.008886  0.000087 
 ...                               ...        ...       ...         ...    
 2020-09-30 23:15:00+00:00   41.624778  11.892219  0.917579   15.610420   5.437229  0.419702
 2020-09-30 23:30:00+00:00   40.319985  11.423915  0.875743   14.846530   5.130067  0.393482 
 2020-09-30 23:45:00+00:00   43.567333  12.411852  0.955641   15.126403   5.250202  0.404163 

 [4020 rows x 6 columns],
                           ...]

what I want:

                             M1       M2          M3          M4        M5       M6
sensor                      
2020-02-02 00:00:00+00:00   11.227779   5.133028    1.057358    2.485335    1.310150    0.281292
2020-02-03 00:00:00+00:00   11.681073   5.245068    1.071268    2.560408    1.322918    0.286271
2020-02-04 00:00:00+00:00   11.391060   5.209235    1.067607    2.530223    1.311115    0.281819
...     ...     ...     ...     ...     ...     ...
2020-12-29 00:00:00+00:00   15.413061   3.162552    0.810623    7.999189    2.100743    0.554420
2020-12-30 00:00:00+00:00   15.630278   3.051311    0.777364    8.084270    2.060067    0.540470
2020-12-31 00:00:00+00:00   16.001919   3.063330    0.770053    8.208727    2.060275    0.536775

366 rows × 6 columns

what I did:

def join_data(datalist):
    
    joined = pd.concat(datalist).reset_index()
    means = joined.groupby('sensor').mean()
    return means

but it doesn't return mean of day:

                                 M1       M2          M3          M4        M5       M6
sensor                      
2020-02-02 00:00:00+00:00   12.227779   6.133028    1.000058    1.485335    1.310150    0.281292
2020-02-02 00:15:00+00:00   10.681073   3.245068    0.071268    3.560408    1.000918    0.200271
2020-02-02 00:30:00+00:00   9.391060    4.209235    0.067607    10.530223   1.301115    1.281819
...     ...     ...     ...     ...     ...     ...
2020-09-30 23:15:00+00:00   15.413061   3.162552    0.810623    7.999189    2.100743    0.554420
2020-09-30 23:30:00+00:00   15.630278   3.051311    0.777364    8.084270    2.060067    0.540470
2020-09-30 23:45:00+00:00   16.001919   3.063330    0.770053    8.208727    2.060275    0.536775

21986 rows × 6 columns

is there any suggestion?

No, it returns the mean for a column in one dataframe, and not a list of dataframes. — Ami, Jan 19 '21 at 15:38
Ah. Got you. Then I think you need to `reset_index` for all the dfs before concatenating them. — Yevhen Kuzmovych, Jan 19 '21 at 15:42
In the join_data function, I used reset_index, but the result is not as I want. — Ami, Jan 19 '21 at 15:47
You use it after concatenation. But I think you should use it before: `pd.concat(df.reset_index() for df in datalist)` — Yevhen Kuzmovych, Jan 19 '21 at 15:49
If you still grouping just by `sensor` column, you are grouping by time, not date. Now check the solution that I recommended in my first column — Yevhen Kuzmovych, Jan 19 '21 at 16:24
perfect. 'df.set_index('Date_Time').groupby(pd.Grouper(freq='D')).mean() ' is worked! thanks. — Ami, Jan 19 '21 at 16:34

Python: List of Dataframes

0 Answers0