Why do we lose data when assigned to transpose of dataframe?

Question

Lets say I have a dataframe like

df = pd.DataFrame({'A':[1,2,3,4],'B':[1,3,4,7]})

   A  B
0  1  1
1  2  3
2  3  4
3  4  7

When I assign some data to transpose of a dataframe, there is no error i.e

df.T['C'] = 3

There is no change in the dataframe after running this.

But the question is where is the data being stored ? Why did't it give any error? I was expecting an error for this kind of assignment or an output like

   A  B
0  1  1
1  2  3
2  3  4
3  4  7
C  3  3

Neither is happening when I did df.T['C'] = 3

Edit: as @Zero mention we might have to do

df = df.T.assign(C=3).T # Which is like df.loc['C',:] = 3

Answer 1

df.T is a different object. The changes you make will not be reflected in the original df. Where is it? Since there is no variable pointing to it, either it has already been collected by the garbage collector or it is waiting to be collected. You cannot access it.

What you can do is to create a new variable

transposed = df.T

transposed['C'] = 3

transposed
Out: 
   0  1  2  3  C
A  1  2  3  4  3
B  1  3  4  7  3   

The same thing happens when you call any method that returns a new DataFrame. df.drop(0)['C'] = 2, df.reset_index()['C'] = 3 or df.drop_duplicates()['C'] = 3. The original DataFrame always stays the same. There is another DataFrame created with that exact row assigned to it but it becomes inaccessible as soon as you execute that statement because you don't have any variables pointing to it. For CPython's garbage collection, there is some useful information here.

---

Edit from @Bharath:

(an explanation given by one of my teachers)

T returns a copy. That means new memory is allocated to store the new object. If you look up python garbage collection you’ll find that each object in memory keeps a counter of how many pointers are pointing to it.

When the garbage collection is run, it will find this object in memory and see that it has zero pointers. Because it has zero pointers the garbage collection will reclaim the memory and the object is gone forever.

So it is recommended to keep a single pointer pointing to the object by assigning to a name (or variable).

Answer 2

Method T does return super(DataFrame, self).transpose(1, 0, **kwargs).
It will create another DataFrame.

Answer 3

Adding to the existing answers, I'd like to draw your attention to the canny similarity between -

df

   A  B
0  1  1
1  2  3
2  3  4
3  4  7

df.T['C'] = 3

df

   A  B
0  1  1
1  2  3
2  3  4
3  4  7

And, a similar case with python lists -

l = [1, 2, 3, 4, 5]
l[:].append(6)

l
[1, 2, 3, 4, 5]

What happens in both cases is that a new object is created! The operation is then applied to that newly created object, following which, that object is garbage collected since there are no active references pointing to it. You see that with this -

import sys

sys.getrefcount(df.T)
1

There's only one reference to that object (the reference at that point of time, which is subsequently lost). This becomes easy to understand once you accept the fact that df.T returns a completely new object (I've said this already, but I'm trying to drive home the point) -

id(df.T)
4612098928

id(df.T)
4612098872

id(df.T)
4612098592

In summary, you are attempting to modify a fresh object to which you have no reference, and you do not see any changes to the original because you did not make any.