Create subset of multiple pandas dataframes using for statement

Question

I am working with multiple dataframes in pandas, and am looking to only select certain columns from each of them.

AUD = [AUD2yr,AUD5yr,AUD10yr,AUD30yr]

for df in AUD:
    df = df[['Date','Open']]

I would ideally like to keep the names of the dataframes, but i receive the following error:

'NoneType' object is not subscriptable

Any help would be much appreciated.

Answer 1

'NoneType' object is not subscriptable

This means in your program somewhere None[...] is attempted. From the snippet you shared, that happens with df[...], i.e., somehow df became None.

Apart from that, doing df = df[["Date", "Open"]] inside the loop body will not alter the frames in the list; rather, it makes df now "look" at a new frame; the reference to the frame in the list is lost.

To fix that, you can traverse the list along with the index and modify with indexing:

for idx, frame in enumerate(AUD):
    AUD[idx] = frame[["Date", "Open"]]

or perhaps better, with a list comprehension:

AUD = [frame[["Date", "Open"]] for frame in AUD]

Lastly, to avoid the infamous SettingWithCopyWarning in case you further modify the frames, it's better you put .copy() after subsetting, e.g.,

AUD = [frame[["Date", "Open"]].copy() for frame in AUD]

---

Additionally, to affect the individual variables inside the list, i.e., AUD*yr vars in AUD = [AUD2yr, AUD5yr, AUD10yr, AUD30yr], there are some ways:

• We can go for inplace=True option in operations we do. Note that this is not always possible; but in your specific case, we'll use .drop method which supports this, and we will drop all but "Date" and "Open" columns, so here we go:

for idx, frame in AUD:
    cols_to_drop = frame.columns.difference(["Date", "Open"])
    AUD[idx].drop(columns=cols_to_drop, inplace=True)

^this approach is infeasible with a list comprehension. But that brings us to 2nd option:

• If there is handful of frames in the list, like you have 4 in AUD, we can unpack the new list again to those variables:

AUD2yr, AUD5yr, AUD10yr, AUD30yr = AUD = [frame[["Date", "Open"]].copy()
                                          for frame in AUD]

This will now not only alter AUD but also the individual variables. First option also does that, so your choice :)

• There's a 3rd alternative... But that requires you to change AUD to be a dictionary rather than a list. This way you won't refer to the frames as bareword variables like AUD10yr but rather as dictionary keys, e.g., AUD["AUD10yr"]. For what it's worth, this might be a better practice if you got a lot frames especially. So,

# initialize as dictionary
AUD = {"AUD2yr": AUD2yr, "AUD5yr": AUD5yr, "AUD10yr": AUD10yr, "AUD30yr": AUD30yr}

# modify via dict comprehension, akin to list comprehension in the very above
AUD = {frame_name: frame[["Date", "Open"]].copy()
       for frame_name, frame in AUD.items()}

# now, can refer to individuals as:
print(AUD["AUD5yr"])

---

Some references:

• How "names" work in Python, regarding to df = df[...] not altering the originals: https://nedbatchelder.com/text/names1/names1.html
• On the SettingWithCopyWarning: How to deal with SettingWithCopyWarning in Pandas
• List comprehensions: https://docs.python.org/3/tutorial/datastructures.html#list-comprehensions
• Variable variables: How do I create variable variables?