Loops over lists of Pandas objects exhibit weird behavior

Question

I have encountered a slight head scratcher when it comes to lists of Pandas objects and their loops. In some code I was working on, there were a few pandas dataframes which were placed into a list, so operations could be performed on all of them.

However, I noticed that certain operations, such as creating new columns, work in "naive" Python for loops, whereas other operations, like reversing the orders of the dataframes,

1. require explicit indexing, and
2. do not effect the original dataframes (only their copies residing within the list).

I am seeking help in getting the second part of my MWE below working as easily as the first part, and also to gain insight into understanding what underlying logic is causing this discrepancy in the first place.

## Creating data
import pandas as pd
from io import StringIO

data = StringIO(
"""
date;time;random
2019-06-12;19:59:59+00:00;99
2019-06-12;19:59:54+00:00;200
2019-06-12;19:59:52+00:00;65
2019-06-12;19:59:34+00:00;140
"""
               )

df = pd.read_csv(data, sep=";")

print(df)

## Creating list; there is only one dataframe in this list to make the
## code easier to work with, but in actuality I am working with >20 dataframes
df_list = [df]

## First operation - successfully adds new column to both original df and df_list[0]
for dataframe in df_list:
    dataframe['date_time'] = pd.to_datetime(dataframe['date']+' '+dataframe['time'], utc=True)
print(df)
print(df_list[0])

## Second operation - successful only if using explicit indexing over list, first commented segment does nothing;
## using second segment works, but does not effect original df, only df_list[0].

# for dataframe in df_list:
#     dataframe = dataframe.iloc[::-1]
#     dataframe.reset_index(drop=True, inplace=True)

for i in range(len(df_list)):
    df_list[i] = df_list[i].iloc[::-1]
    df_list[i].reset_index(drop=True, inplace=True)

print(df)
print(df_list[0])

Answer 1

The first operation, dataframe['date_time']= suggests it's an in-place operation, which is not an assignment.

The reason why in the second operation, the second approach works, it is because when you loop through a list not using the index, you created a new variable that not related to the list, and assign it to a new value.

a = [1,2,3]
for i in a:
    i = 0
print(a)
print(i)

The output is:

[1, 2, 3]
0

So in your case, when you for dataframe in df_list:, you create a new variable dataframe, that refer or point to the address of each element in the df_list. Then when you assign them to the reversed data frame, dataframe refers or points to a new variable.

The problem here is you (or we) confused in-place operation vs assignment.

Answer 2

I found that the main point in your question is the premise that all kinds of operations over the shallow copied element(df_list[0]) will be reflected in the original mutable instance(df), but it does not include the assignment as explained here: Python: Assignment vs Shallow Copy vs Deep Copy.

Let's see this normal example:

In [29]: df_list = [df]

In [30]: df_list[0]['date_time'] = pd.to_datetime(df_list[0]['date']+' '+df_list[0]['time'], utc=True)

In [31]: df_list
Out[31]:
[         date            time  random                 date_time
 0  2019-06-12  19:59:59+00:00      99 2019-06-12 19:59:59+00:00
 1  2019-06-12  19:59:54+00:00     200 2019-06-12 19:59:54+00:00
 2  2019-06-12  19:59:52+00:00      65 2019-06-12 19:59:52+00:00
 3  2019-06-12  19:59:34+00:00     140 2019-06-12 19:59:34+00:00]

In [32]: df
Out[32]:
         date            time  random                 date_time
0  2019-06-12  19:59:59+00:00      99 2019-06-12 19:59:59+00:00
1  2019-06-12  19:59:54+00:00     200 2019-06-12 19:59:54+00:00
2  2019-06-12  19:59:52+00:00      65 2019-06-12 19:59:52+00:00
3  2019-06-12  19:59:34+00:00     140 2019-06-12 19:59:34+00:00

It works as expected. That is df_list has its own pointer but df_list[0] and df share the same pointer, then df changes when df_list[0] changes.

In [35]: hex(id(df))
Out[35]: '0x7f2c90e8d978'

In [36]: hex(id(df_list[0]))
Out[36]: '0x7f2c90e8d978'

In [37]: hex(id(df_list))
Out[37]: '0x7f2c90d68188'

The method to check the memory address of Python variable: answer to "print memory address of Python variable [duplicate]"

But in the following example, we are facing a different scenario.

In [22]: df_list = [df]
In [23]: df_list[0] = df_list[0].iloc[::-1]

In [24]: df_list
Out[24]:
[         date            time  random
 3  2019-06-12  19:59:34+00:00     140
 2  2019-06-12  19:59:52+00:00      65
 1  2019-06-12  19:59:54+00:00     200
 0  2019-06-12  19:59:59+00:00      99]

In [25]: df
Out[25]:
         date            time  random
0  2019-06-12  19:59:59+00:00      99
1  2019-06-12  19:59:54+00:00     200
2  2019-06-12  19:59:52+00:00      65
3  2019-06-12  19:59:34+00:00     140

In [26]: df_list[0]['date_time'] = pd.to_datetime(df_list[0]['date']+' '+df_list[0]['time'], utc=True)
/usr/bin/ipython3:1: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  #! /bin/sh
In [27]: df_list
Out[27]:
[         date            time  random                 date_time
 3  2019-06-12  19:59:34+00:00     140 2019-06-12 19:59:34+00:00
 2  2019-06-12  19:59:52+00:00      65 2019-06-12 19:59:52+00:00
 1  2019-06-12  19:59:54+00:00     200 2019-06-12 19:59:54+00:00
 0  2019-06-12  19:59:59+00:00      99 2019-06-12 19:59:59+00:00]

In [28]: df
Out[28]:
         date            time  random
0  2019-06-12  19:59:59+00:00      99
1  2019-06-12  19:59:54+00:00     200
2  2019-06-12  19:59:52+00:00      65
3  2019-06-12  19:59:34+00:00     140

The reason is that we have made operations like adding and removing items, meaning that we have first removed the df_list[0] and then adding a new df_list[0](or replacing) which will both not reflected in the original mutable object instance.

In [40]: hex(id(df_list[0]))
Out[40]: '0x7f2c90d6ea58'

In [41]: hex(id(df))
Out[41]: '0x7f2c90e8d978'

As we can see the pointer of df_list[0] has changed.

Let's see the following simple illustration:

In [44]: a = [[1, 2, 3], [4, 5]]

In [45]: b = a[:]

In [46]: a[0] = [0, 0, 0]

In [47]: b
Out[47]: [[1, 2, 3], [4, 5]]

In [48]: a
Out[48]: [[0, 0, 0], [4, 5]]

It may be not caused by the for loop as you suspected but the discrepancy between the assignment and shallow copy. HTH :)