Is it possible to add attributes to built in python objects dynamically in Python?

Question

I need to add an attribute (holding a tuple or object) to python objects dynamically. This works for Python classes written by me, but not for built in classes.

Consider the following program:

import numpy as np

class My_Class():
    pass

my_obj = My_Class()
my_obj2 = My_Class()

my_obj.__my_hidden_field = (1,1)
my_obj2.__my_hidden_field = (2,1)

print(my_obj.__my_hidden_field, my_obj2.__my_hidden_field)

This correctly prints (1, 1) (2, 1). However the following program doesnt work.

X  = np.random.random(size=(2,3))


X.__my_hidden_field = (3,1) 
setattr(X, '__my_hidden_field', (3,1))

Both of the above line throws the following error # AttributeError: 'numpy.ndarray' object has no attribute '__my_hidden_field'

Now, the reason found from these questions (i.e., Attribute assignment to built-in object, Can't set attributes of object class, python: dynamically adding attributes to a built-in class) is Python does not allow dynamically adding attributes to built_in objects.

Excerpt from the answer: https://stackoverflow.com/a/22103924/8413477

This is prohibited intentionally to prevent accidental fatal changes to built-in types (fatal to parts of the code that you never though of). Also, it is done to prevent the changes to affect different interpreters residing in the address space, since built-in types (unlike user-defined classes) are shared between all such interpreters.

However, all the answers are quite old, and I am badly in need of doing this for my research project.

There is a module that allows to add methods to built in Class though: https://pypi.org/project/forbiddenfruit/

However,it doesnt allow adding objects/attributes to each object.

Any help ?

Answer 1

You probably want weakref.WeakKeyDictionary. From the doc,

This can be used to associate additional data with an object owned by other parts of an application without adding attributes to those objects.

Like an attribute, and unlike a plain dict, this allows the objects to get garbage collected when there are no other references to it.

You'd look up the field with

my_hidden_field[X]

instead of

X._my_hidden_field

Two caveats: First, since a weak key may be deleted at any time without warning, you shouldn't iterate over a WeakKeyDictionary. Looking up an object you have a reference to is fine though. And second, you can't make a weakref to an object type written in C that doesn't have a slot for it (true for many builtins), or a type written in Python that doesn't allow a __weakref__ attribute (usually due to __slots__).

If this is a problem, you can just use a normal dict for those types, but you'll have to clean it up yourself.

Answer 2

Quick answer

Is it possible to add attributes to built in python objects dynamically in Python?

No, the reasons your read about in the links you posted, are the same now days. But I came out with a recipe I think might be the starting point of your tracer.

Instrumenting using subclassing combined with AST

After reading a lot about this, I came out with a recipe that might not be the complete solution, but it sure looks like you can start from here.

The good thing about this recipe is that it doesn't use third-party libraries, all is achieved with the standard (Python 3.5, 3.6, 3.7) libraries.

The target code.

This recipe will make code like this be instrumented (simple instrumentation is performed here, this is just a poof of concept) and executed.

# target/target.py

d = {1: 2}
d.update({3: 4})
print(d)                 # Should print "{1: 2, 3: 4}"
print(d.hidden_field)    # Should print "(0, 0)"

Subclassing

Fist we have to add the hidden_field to anything we want to (this recipe have been tested only with dictionaries).

The following code receives a value, finds out its type/class and subclass it in order to add the mentioned hidden_field.

def instrument_node(value):
    VarType = type(value)
    class AnalyserHelper(VarType):
        def __init__(self, *args, **kwargs):
            self.hidden_field = (0, 0)
            super(AnalyserHelper, self).__init__(*args, **kwargs)
    return AnalyserHelper(value)

with that in place you are able to:

d = {1: 2}
d = instrument_node(d) 
d.update({3: 4})
print(d)                 # Do print "{1: 2, 3: 4}"
print(d.hidden_field)    # Do print "(0, 0)"

At this point, we know already a way to "add instrumentation to a built-in dictionary" but there is no transparency here.

Modify the AST.

The next step is to "hide" the instrument_node call and we will do that using the ast Python module.

The following is an AST node transformer that will take any dictionary it finds and wrap it in an instrument_node call:

class AnalyserNodeTransformer(ast.NodeTransformer):
    """Wraps all dicts in a call to instrument_node()"""
    def visit_Dict(self, node):
        return ast.Call(func=ast.Name(id='instrument_node', ctx=ast.Load()),
                        args=[node], keywords=[])
        return node

Putting all together.

With thats tools you can the write a script that:

1. Read the target code.
2. Parse the program.
3. Apply AST changes.
4. Compile it.
5. And execute it.

import ast
import os
from ast_transformer import AnalyserNodeTransformer

# instrument_node need to be in the namespace here.
from ast_transformer import instrument_node

if __name__ == "__main__":

    target_path = os.path.join(os.path.dirname(__file__), 'target/target.py')

    with open(target_path, 'r') as program:
        # Read and parse the target script.
        tree = ast.parse(program.read())
        # Make transformations.
        tree = AnalyserNodeTransformer().visit(tree)
        # Fix locations.
        ast.fix_missing_locations(tree)
        # Compile and execute.
        compiled = compile(tree, filename='target.py', mode='exec')
        exec(compiled)

This will take our target code, and wraps every dictionary with an instrument_node() and execute the result of such change.

The output of running this against our target code,

# target/target.py

d = {1: 2}
d.update({3: 4})
print(d)                 # Will print "{1: 2, 3: 4}"
print(d.hidden_field)    # Will print "(0, 0)"

is:

>>> {1: 2, 3: 4} 
>>> (0, 0)

Working example

You can clone a working example here.

Answer 3

Yes, it is possible, it is one of the coolest things of python, in Python, all the classes are created by the typeclass

You can read in detail here, but what you need to do is this

In [58]: My_Class = type("My_Class", (My_Class,), {"__my_hidden_field__": X})

In [59]: My_Class.__my_hidden_field__
Out[59]:
array([[0.73998002, 0.68213825, 0.41621582],
       [0.05936479, 0.14348496, 0.61119082]])

*Edited because inheritance was missing, you need to pass the original class as a second argument (in tuple) so that it updates, otherwise it simply re-writes the class)