Nested for loop with multiple operations using List Comprehension

Question

Just understanding List Comprehensions, as in a recent interview the tech guy asked me this question and being a self learner I answered lambda which is NOT list comprehension.

let say we have a time series data "Shiller", http://us.spindices.com/indices/real-estate/sp-corelogic-case-shiller-us-national-home-price-nsa-index

I calculated aic/bic using following loop:

shiller = [please use some random data or use the link above]

import matplotlib.pyplot as plt
import statsmodels.api as sm
import pandas as pd

def aicbic(shiller):
  arimaijk = []
  aicijk = []
  bicijk = []
  index = []
  for i in range(1,3):
      for j in range(1,2):
          for k in range(0,5):
              arimaijk.append(sm.tsa.ARIMA(shiller,(i,j,k)).fit())
              index.append([i,j,k])
              aicijk.append(arimaijk[k].aic)            
              bicijk.append(arimaijk[k].bic)
  return aicijk, bicijk

aicbic(shiller)
Out[9]: 
([-235.77314152121426,-233.9375761653174,-233.3841011331017,-241.65994870973782,-240.2975620564456,-235.77314152121426,-233.9375761653174,-233.3841011331017,-241.65994870973782,-240.2975620564456],
 [-227.98778197081049,-223.55709676477906,-220.40850188242874,-226.08922960893028,-222.13172310550345,-227.98778197081049,-223.55709676477906,-220.40850188242874,-226.08922960893028,-222.13172310550345])

Now, I want this result using List Comprehension, so I wrote following lines, which are returning error:

def aicbic(data):    
  arimaijk = []
  aicijk = []
  bicijk = []
  index = []
  [(sm.tsa.ARIMA(data,(i,j,k)).fit(),index.append([i,j,k]),\
  aicijk.append(arimaijk[k].aic),bicijk.append(arimaijk[k].bic)) \
  for i in range(1,3) for j in range(1,2) for k in range(0,5)]

IndexError: list index out of range

You can use list comprehensions here. You will just need to cut it up a bit differently. And if you had constructed a [Minimal, Complete, and Verifiable](http://stackoverflow.com/help/mcve) example, I would have done it for you. And likely so would have the other people who voted to close this question. — Stephen Rauch, Mar 19 '17 at 03:46
Added some data and libraries, if you mean that for Minimal, Complete, and Verifiable example. — Bhushan, Mar 19 '17 at 03:58
To get the best help here, we want to be able to cut and paste right from the post into an editor, and then run the code. With verifiable results. Then we can change the code and make sure the results are still good. If we know something, it is generally the tools, not *your* problem domain. So, you just pasted some data named `shiller`, but your code does not reference it. Also there are no expected results. — Stephen Rauch, Mar 19 '17 at 04:07
OK, done. Sorry for the trouble! I was not sure if I can reproduce S&P data here, but anyway I did it partially. I will take it down in sometime. — Bhushan, Mar 19 '17 at 04:15
1) use fake data. It should not matter for the example. 2) have you cut and paste from this post and run it? It does not run. Sigh.... — Stephen Rauch, Mar 19 '17 at 04:24
I think its OK for now to use the data as its freely available. It's not full. I did copy paste now and its working. — Bhushan, Mar 19 '17 at 04:29

score 2 · Answer 1 · answered Mar 19 '17 at 05:06

2

The error does not relate to list comprehenstion itself:

[(sm.tsa.ARIMA(data,(i,j,k)).fit(),index.append([i,j,k]),\
  aicijk.append(arimaijk[k].aic),bicijk.append(arimaijk[k].bic)) \
  for i in range(1,3) for j in range(1,2) for k in range(0,5)]

The error IndexError: list index out of range is raised because you want to access arimaijk[k] while arimaijk is an empty list (the first line of aicbic(data) function is arimaijk=[]).

answered Mar 19 '17 at 05:06

Hossein

2,041
1
16
29

This does not address the problem. List comprehensions are intended to construct a list. This code is using a side effect of the list comprehension to append to two lists and thus create the list inside of a list comprehension. In this case a for loop would be a better solution. See: http://stackoverflow.com/questions/5753597/is-it-pythonic-to-use-list-comprehensions-for-just-side-effects#5753614 – Stephen Rauch Mar 19 '17 at 05:42
I just clarified what is the reason for the `IndexError`. Thank you anyway, Stephen. – Hossein Mar 19 '17 at 07:37

score 1 · Accepted Answer · answered Mar 19 '17 at 05:07

Your function recast using itertools and list comprehensions:

Code:

import itertools as it

def aicbic(shiller):
    loop = list(it.product(range(1, 3), range(1, 2), range(0, 5)))
    arimaijk = [sm.tsa.ARIMA(shiller, (i, j, k)).fit() for i, j, k in loop]
    aicijk = [arimaijk[k].aic for i, j, k in loop]
    bicijk = [arimaijk[k].bic for i, j, k in loop]
    return aicijk, bicijk

Test Code:

result = aicbic(shiller)

import numpy as np
assert np.all(np.isclose(result, (
    [-235.77314152121426, -233.9375761653174, -233.3841011331017,
     -241.65994870973782, -240.2975620564456, -235.77314152121426,
     -233.9375761653174, -233.3841011331017, -241.65994870973782,
     -240.2975620564456],
    [-227.98778197081049, -223.55709676477906, -220.40850188242874,
     -226.08922960893028, -222.13172310550345, -227.98778197081049,
     -223.55709676477906, -220.40850188242874, -226.08922960893028,
     -222.13172310550345]
)))

Note:

You likely have a bug of some sort, as aicijk and bicijk depend only on k.

Thanks Stephen. Interesting solution. – Bhushan Mar 19 '17 at 05:17 — Bhushan, Mar 19 '17 at 05:17

score 0 · Answer 3 · answered Mar 19 '17 at 05:24

0

Thanks to Hossein, I guess now I know how to use List Comprehensions!

def aicbic(data):
  arimaijk = []
  aicijk = []
  bicijk = []
  index = []
  [(arimaijk.append(sm.tsa.ARIMA(data,(i,j,k)).fit()),index.append([i,j,k]),\
  aicijk.append(arimaijk[k].aic),bicijk.append(arimaijk[k].bic)) \
  for i in range(1,3) for j in range(1,2) for k in range(0,5)]
  return aicijk, bicijk


result = aicbic(shiller)

answered Mar 19 '17 at 05:24

Bhushan

87
1
9

What? No. You are entirely missing the point.... If your list comprehension solution is given on an interview you will not get the job. For your sake, please study what I wrote. – Stephen Rauch Mar 19 '17 at 05:32
Hi Stephen, this is only the part of the code where I wanted to understand how can I reimplement it using List Comprehension. You're right about the code that it depends only on k. I need to think another code for p in ARIMA(p,1,q) i.e. 'i'. If you mean that. Also, this question was not on interview. The tech guy only asked explain 'List Comprehension' and answered 'Lambda'. But apparently its not correct. – Bhushan Mar 19 '17 at 05:43
Read my response to Hossein's answer. To think that his answer is correct will confuse you going forward. – Stephen Rauch Mar 19 '17 at 05:47
OK. It doesn't make sense to use 'append' within List Comprehension. You're absolutely correct. Thanks Stephen for defending your point. – Bhushan Mar 19 '17 at 07:02
In my answer, I just clarified what is the reason for `IndexError`. About the list comprehension, I agree with Stephen that when you don't want to explicitly use the output of a list comprehension, you should avoid using the list comprehension. – Hossein Mar 19 '17 at 07:59

Nested for loop with multiple operations using List Comprehension

3 Answers3