1

I like to store metadata about a dataframe by simply setting an attribute and its corresponding value, like this: df.foo = "bar" However, I've found that attributes stored like this are gone once I slice the dataframe:

df.foo = "bar"
df[:100].foo
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "C:\Users\admin\PycharmProjects\project\venv\lib\site-packages\pandas\core\generic.py", line 5465, in __getattr__
    return object.__getattribute__(self, name)
AttributeError: 'DataFrame' object has no attribute 'foo'

I wonder if this behavior can be changed, similar to how drop=True or inplace=True change the way attributes like df.set_index(args) work. I didn't find anything helpful in the pandas docs.

humungous aligator
  • 13
  • 2
  • The reason why your slice doesn't persist is probably because it creates a new dataframe without the metadata. There isn't a way to propagate that metadata though. See https://stackoverflow.com/questions/14688306/adding-meta-information-metadata-to-pandas-dataframe. – ifly6 Jun 21 '21 at 15:14

1 Answers1

0

For many operations, pandas returns a new object so any attributes you have defined, which aren't natively supported in the pd.DataFrame class will not persist.

A simple alternative is to subclass the DataFrame. You need to be sure to add the attribute to the _metadata else it wont persist

import pandas as pd

class MyDataFrame(pd.DataFrame):

    # temporary properties
    _internal_names = pd.DataFrame._internal_names
    _internal_names_set = set(_internal_names)

    # normal properties
    _metadata = ["foo"]

    @property
    def _constructor(self):
        return MyDataFrame

df = MyDataFrame({'data': range(10)})
df.foo = 'bar'

df[:100].foo
#'bar'
ALollz
  • 57,915
  • 7
  • 66
  • 89
  • 2
    Thanks. I suspected that I'll have to use subclassing at some point which would also prove more scalable. For now, however I've found [this](https://stackoverflow.com/a/60063557/16060809) answer from another thread which explains the use of df. attrs (which is a way of inheriting attributes). – humungous aligator Jun 21 '21 at 15:25