1

I am trying to create an app which will launch many processes and monitor their outputs. For this purpose I am creating a dictionary of callbacks as so:

{
  'alpha':  {
     'stdout': <function <lambda> at 0x7f0aa5cbaee0>, 
     'stderr': <function <lambda> at 0x7f0aa5cbaf70>
  }, 
  'beta': {
     'stdout': <function <lambda> at 0x7f0aa5cc6040>, 
     'stderr': <function <lambda> at 0x7f0aa5cc60d0>
  }, 
  'gamma': {
     'stdout': <function <lambda> at 0x7f0aa5cc6160>, 
     'stderr': <function <lambda> at 0x7f0aa5cc61f0>
  }
}

I am using this code to generate these callbacks

def callback_func(arg, job):
    print("job: " + job + ", arg: " + arg + ", job address: " + hex(id(job)))

callbacks = {}
important_var = "hello"

for job in ["alpha", "beta", "gamma"]:
    callbacks[job] = {}
    callbacks[job]["stdout"] = lambda x: callback_func(x, job)
    callbacks[job]["stderr"] = lambda x: callback_func(x, job)

print(callbacks)

for jobname in callbacks:
    print("Testing callbacks for " + jobname)
    callbacks[jobname]["stdout"]("STDOUT test")
    callbacks[jobname]["stderr"]("STDERR test")
    print("")

However this code results in:

Testing callbacks for alpha
job: gamma, arg: STDOUT test, job address: 0x7f21b5a74c70
job: gamma, arg: STDERR test, job address: 0x7f21b5a74c70

Testing callbacks for beta
job: gamma, arg: STDOUT test, job address: 0x7f21b5a74c70
job: gamma, arg: STDERR test, job address: 0x7f21b5a74c70

Testing callbacks for gamma
job: gamma, arg: STDOUT test, job address: 0x7f21b5a74c70
job: gamma, arg: STDERR test, job address: 0x7f21b5a74c70

Even though the callbacks were generated with different job arguments they still insist on using the last job argument.

I ended up using intermediary objects as described here. So my code looks like this:

def callback_func(arg, job):
    print("job: " + job + ", arg: " + arg + ", job address: " + hex(id(job)))

class Callback(object):
    def __init__(self, job):
        self.job = job

    def __call__(self, arg):
        callback_func(self.job, arg)

callbacks = {}
important_var = "hello"

for job in ["alpha", "beta", "gamma"]:
    callbacks[job] = {}
    callbacks[job]["stdout"] = Callback(job)
    callbacks[job]["stderr"] = Callback(job)

print(callbacks)

for jobname in callbacks:
    print("Testing callbacks for " + jobname)
    callbacks[jobname]["stdout"]("STDOUT test")
    callbacks[jobname]["stderr"]("STDERR test")
    print("")

But why was this even necessary? Even if I included job = copy.deepcopy(job) it still refused to work correctly.

Meowxiik
  • 83
  • 5
  • 1
    `lambda x, job=job: callback_func(x, job)` would have been the simple fix in your case. Look into Python closures. – Paul M. Jul 16 '21 at 18:34
  • 1
    `job` is a free variable the body of the lambda expression, just as it would be in the body of a `def` statement. Its lookup occurs when the function is called, not when the function is defined, and you change the value of the variable `job` between definition and call. – chepner Jul 16 '21 at 18:36
  • Thank you for the feedback! I will definitely look up Python closures, I had a feeling something like that was gonna be it but didnt know how to search for it – Meowxiik Jul 16 '21 at 18:38
  • 1
    Another solution is to use `partial(callback_func, job)` (requires swapping the parameters in the definition of `callback_func` so that `job` is the first positional parameter) or `partial(callback_func, job=job)` (passing the job as a keyword argument instead of a positional argument) instead of lambda expressions. – chepner Jul 16 '21 at 18:38
  • 1
    Relevant article on late-binding closures: https://docs.python-guide.org/writing/gotchas/#late-binding-closures – Kyle Parsons Jul 16 '21 at 18:44

0 Answers0