38

I want to use the Pandas dataframe to breakdown the variance in one variable.

For example, if I have a column called 'Degrees', and I have this indexed for various dates, cities, and night vs. day, I want to find out what fraction of the variation in this series is coming from cross-sectional city variation, how much is coming from time series variation, and how much is coming from night vs. day.

In Stata I would use Fixed effects and look at the R^2. Hopefully my question makes sense.

Basically, what I want to do, is find the ANOVA breakdown of "Degrees" by three other columns.

wolfsatthedoor
  • 7,163
  • 18
  • 46
  • 90
  • 2
    You'll want to look into scipy or statsmodels (I just added those tags, pending approval) – JohnE Aug 27 '14 at 23:28
  • 1
    In a nutshell, statsmodels is analogous to the statistical parts of stata (whereas pandas is the data management part). – JohnE Aug 27 '14 at 23:34
  • Anything more specific :) ? – wolfsatthedoor Aug 27 '14 at 23:43
  • The main ANOVA extension to the linear model (OLS) are explained here http://statsmodels.sourceforge.net/devel/examples/notebooks/generated/interactions_anova.html OLS Results have rsquared, anova_lm calculates the sum of squares or F tests for the effect of categorical variables. – Josef Aug 28 '14 at 00:24
  • 5
    By coincidence just came across the o'reilley book "think stats" which uses pandas and statsmodels. Free online version here: http://www.greenteapress.com/thinkstats2/html/index.html – JohnE Aug 28 '14 at 01:35
  • 1
    There's a complete code example finishing with an ANOVA table and residuals at http://statsmodels.sourceforge.net/devel/anova.html. – cphlewis Feb 21 '15 at 06:10

1 Answers1

32

I set up a direct comparison to test them, found that their assumptions can differ slightly , got a hint from a statistician, and here is an example of ANOVA on a pandas dataframe matching R's results:

import pandas as pd
import statsmodels.api as sm
from statsmodels.formula.api import ols


# R code on R sample dataset

#> anova(with(ChickWeight, lm(weight ~ Time + Diet)))
#Analysis of Variance Table
#
#Response: weight
#           Df  Sum Sq Mean Sq  F value    Pr(>F)
#Time        1 2042344 2042344 1576.460 < 2.2e-16 ***
#Diet        3  129876   43292   33.417 < 2.2e-16 ***
#Residuals 573  742336    1296
#write.csv(file='ChickWeight.csv', x=ChickWeight, row.names=F)

cw = pd.read_csv('ChickWeight.csv')

cw_lm=ols('weight ~ Time + C(Diet)', data=cw).fit() #Specify C for Categorical
print(sm.stats.anova_lm(cw_lm, typ=2))
#                  sum_sq   df            F         PR(>F)
#C(Diet)    129876.056995    3    33.416570   6.473189e-20
#Time      2016357.148493    1  1556.400956  1.803038e-165
#Residual   742336.119560  573          NaN            NaN
Community
  • 1
  • 1
cphlewis
  • 15,759
  • 4
  • 46
  • 55
  • 2
    But this is not ANOVA test. This is a linear model coefficients analysis. – aghd Apr 09 '19 at 23:53
  • @cphlewis, actually, an ANOVA test is nothing more than a linear model in which we have a continuous dependent variable and a categorical input. – Luis Jan 28 '22 at 17:57