Python dynamic function building and closure rule

Question

While interactively executing the following code in http://pythontutor.com/visualize.html, the frame of each call to build_match_and_apply_functions appears gray in the graphical view:

This program is for getting a plural of a word , which is quoted from the chapter in DIVE in Python 3

CODE:

import re

def build_match_and_apply_functions(pattern, search, replace):
    def matches_rule(word):                                   
        return re.search(pattern, word)
    def apply_rule(word):                                     
        return re.sub(search, replace, word)
    return (matches_rule, apply_rule)        

patterns =  \
    (
    ('[sxz]$',           '$',  'es'),
    ('[^aeioudgkprt]h$', '$',  'es'),
    ('(qu|[^aeiou])y$',  'y$', 'ies'),
    ('$',                '$',  's')
  )
rules = [build_match_and_apply_functions(pattern, search, replace)  
         for (pattern, search, replace) in patterns]


def plural(noun):
    for matches_rule, apply_rule in rules: 
        if matches_rule(noun):
            return apply_rule(noun)


plural('vacancy')

question:

1) what does gray frame mean? it is a closure that still takes memory?

2) can I go in memory block. so i can figure out in the objects area , are all matches_rule functions the same or not? if they are the same , f2/f3/f4/f5 should be there to offer pattern/search/replace values.

if not, if all matches_rules functions have already be changed to different functions , f2 3 4 5 could be end of live and dispear. they are useless.

i dont know , that is how dynamic language works and being built.

pythontutor.com ANALYZE DIGRAM really surprised me, tutor did the good job

if u dont expirence it, please copy the link below and paste my code in it. i bet you are having fun with it.

http://pythontutor.com/visualize.html

score 1 · Answer 1 · edited May 23 '17 at 12:33

The gray frames are the execution frames, which contain the closure. They can theoretically be read from memory.

For example, I managed to read the data from the frames with the following code (CPython v3.5):

Some utilities first:

import gc, inspect, ctypes, random

def all_live_ids():
    """Get all valid object IDs"""
    for obj in gc.get_objects():
        yield id(obj)

def getobj(obj_id):
    """Get the object from object ID"""
    if obj_id in all_live_ids():
        return ctypes.cast(obj_id, ctypes.py_object).value
    else:
        return None

frame_ids = []

def print_frames():
    for fid in frame_ids:
        frame = getobj(fid)
        if frame:
            print('frame', hex(fid), 'is alive and has pattern:',
                   frame.f_locals['pattern'])
        else:
            print('frame', hex(fid), 'is dead')

Then a simplified version of the code from the question:

def build_match_and_apply_functions(pattern, search, replace):
    frame = inspect.currentframe()
    frame_ids.append(id(frame))
    print('executing frame', frame, 'pattern:', pattern)
    def matches_rule(word):                                   
        return re.search(pattern, word)
    return matches_rule

rule1 = build_match_and_apply_functions('[sxz]$', '$', 'es')
rule2 = build_match_and_apply_functions('[^aeioudgkprt]h$', '$', 'es')

print('\nbefore gc.collect():')
print_frames()

gc.collect()

print('\nafter gc.collect():')
print_frames()

In my case, it printed:

executing frame <frame object at 0x0071EDB0> pattern: [sxz]$
executing frame <frame object at 0x00C0E4B0> pattern: [^aeioudgkprt]h$

before gc.collect():
frame 0x71edb0 is alive and has pattern: [sxz]$
frame 0xc0e4b0 is alive and has pattern: [^aeioudgkprt]h$

after gc.collect():
frame 0x71edb0 is dead
frame 0xc0e4b0 is dead

What did I do?

I remembered IDs of all frames in frame_ids. In CPython, id returns the memory address of the object.

Then I used this solution to convert IDs back to objects.

I had to do that in order to be able to get the frame without ever storing a reference to the frame (Usualy weakref does that, but not with frame objects`)

There is a catch

Notice that the frames disappeared after gc.collect. The frames are not needed once the function finishes, but there were some circular references which prevented their immediate deletion. gc.collect is smart enough to detect those and delete the unneeded frames. With different code, that may happen even without calling gc.collect and I believe it should happen in http://pythontutor.com/ as well, but their internals are in some sort of conflict with the garbace collector.

Python dynamic function building and closure rule

1 Answers1

What did I do?

There is a catch