I'm trying to automate a regression usually performed in Eviews in python's statsmodels. Unfortunately it won't be possible to give reproducible example in this case, but the model is a multivariate regression model with ARMA errors. The number of observations is in the ten thousands and I'm using about 50 independent variables (actually 5 variables with 10 lags. The model itself is fine, so leave this aside.).
In Eviews (a commercial econometrics package for time series), the model will converge after about 20 seconds having performed something like 17 iteration using BFGS as optimisation method. Using statsmodels however, convergence takes forever even with a much smaller number of observations, and retrieving the beta standard errors takes even longer. I've tried supplying starting values, different optimisation methods and tolerance levels, but python remains extremely slow in comparison.
I was expecting that the performance would be similar, as statsmodels seems able to use the same optimisation method that Eviews uses (BFGS). The fast convergence in Eviews seems dependent on selecting "Marquardt" as "step method" in the optimisation. If instead opting for "line search", the model takes a very long time to converge. Could it be that statsmodels uses the latter?
My question is really this: Should I simply give up on scipy's (which I believe statsmodels relies on) optimiser, as it cannot be expected to have nearly the same performance as a commercial econometric software? Or is it reasonable to expect similar performance, and I should hence try other ways of "optimising the optimisation"?
By the way, I've started looking at this pyeviews, but there is very little documentation for it. Anyone with experience?
I've wasted a lot of time trying to get this thing to work, so any guidance would be very much appreciated.
UPDATE: With my version of Eviews and python, pyeviews throws an error whenever trying to push more than five hundred or something rows into Eviews. Seems to be a bug. I've posted on Github.
Björn
