Passing *args or **kwargs into threading.Thread()

Question

Using the threading library, I am attempting to pass down a variety of args/keyword args into the threading function. There are three different situations. Each one consists of a function ran prior to the threading function, which passes down the options to the threading function to tell it what function to call for the threading, and the arguments. In some situations, there will be keyword arguments, in some arguments, and in some, none. Here is the sample code of the threading function:

def create_some_threads(my_target, *my_args, **my_keyword_args):
        for DeviceName in Devices:
            device_ip = Devices[DeviceName]['mgmt_ip']
            my_thread = threading.Thread(target=my_target, args=(device_ip, DeviceName, *my_args, **my_keyword_args))
            my_thread.start()

Unfortunately, I get the error:

    my_thread = threading.Thread(target=my_target, args=(device_ip, DeviceName, *args, **kwargs))
                                                                                ^                                                                
SyntaxError: invalid syntax

Here is the function where I call in the *my_args:

pingable_devices = {}
unpingable_devices = {}
with open(os.devnull, "wb") as limbo:
    print("\n[+] Progress:\n")
    pbar = tqdm(total=100)
    my_args = (pingable_devices, unpingable_devices, pbar, limbo)
    my_target = ping_em_all
    create_some_threads(my_target, *my_args)

The problem is, for each function I use prior to calling the threading function, there are a different set of arguments/keyword arguments that I will call. I attempted to make this a function on its own, but if I can't run it something like this, then I may have to explore other avenues.

Edit: This first problem was solved by unutbu, but now I have encountered a second problem. It seems the function called by the threading function is not recognizing the variables passed into threading, my_args. Error is below:

Exception in thread Thread-4:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "Automatelab.py", line 157, in ping_em_all
    ping_reply = subprocess.Popen(['ping', '-c', '2', '-w', '2', '-q', device_ip.rstrip('\n')],stdout=limbo, stderr=limbo).wait()
NameError: global name 'device_ip' is not defined

Here is the ping_em_all function:

def ping_em_all(*my_args, **my_keyword_args):
    """.rstrip is needed for the ip as .readline adds a \n to
    the lines' text"""
    if "Linux" in platform.system():
        ping_reply = subprocess.Popen(['ping', '-c', '2', '-w', '2', '-q', device_ip.rstrip('\n')],stdout=limbo, stderr=limbo).wait()
    #Darwin is Mac OSX
    elif "Darwin" in platform.system():
        ping_reply = subprocess.Popen(['ping', '-c', '2', '-t', '2', '-q', '-n', device_ip.rstrip('\n')],stdout=limbo, stderr=limbo).wait()
        """Subprocess for Cygwin still not supported"""
    else:
    #Only other would be Windows
        ping_reply = subprocess.Popen(['ping', '-n', '2', '-w', '2', device_ip.rstrip('\n')],stdout=limbo, stderr=limbo).wait()

score 2 · Answer 1 · edited Oct 25 '19 at 11:23

The target is called with:

self._target(*self._args, **self._kwargs)

Therefore, you could use:

def create_some_threads(my_target, *my_args, **my_keyword_args):
    for DeviceName in Devices:
        device_ip = Devices[DeviceName]['mgmt_ip']
        my_args = (device_ip, DeviceName,) + my_args
        my_thread = threading.Thread(target=my_target, 
                                     args=my_args, kwargs=my_keyword_args))
        my_thread.start()

When you define you define a function with *my_args or **my_keyword_args in its call signature, note that my_args becomes a tuple inside the body of that function, and my_keyword_args becomes a dict.

my_args = (device_ip, DeviceName,) + my_args

simply concatenates two more values to the front of the my_args tuple.

The error message

NameError: global name 'device_ip' is not defined

is saying that the variable name device_ip has not been defined in a scope that is accessible at the point where the NameError occurred. A look at the ping_em_all function shows that that's true:

def ping_em_all(*my_args, **my_keyword_args):
    """.rstrip is needed for the ip as .readline adds a \n to
    the lines' text"""
    if "Linux" in platform.system():
        ping_reply = subprocess.Popen(['ping', '-c', '2', '-w', '2', '-q', device_ip.rstrip('\n')],stdout=limbo, stderr=limbo).wait()
    ...

device_ip is the first value in my_args:

my_args = (device_ip, DeviceName,) + my_args

Therefore, change ping_em_all to

def ping_em_all(*my_args, **my_keyword_args):
    """.rstrip is needed for the ip as .readline adds a \n to
    the lines' text"""
    device_ip, DeviceName, my_args = my_args    #<-- This defines device_ip
    if "Linux" in platform.system():
        ping_reply = subprocess.Popen(['ping', '-c', '2', '-w', '2', '-q', device_ip.rstrip('\n')],stdout=limbo, stderr=limbo).wait()
    ...

to unpack values from the my_args tuple and give them variable names.

So that did work, but now I have a separate problem. All variables in my_args are not recognized by the target function. See above. — Winner1235813213455, Dec 10 '17 at 12:43

score 0 · Answer 2 · answered Nov 09 '19 at 21:28

There are a few possible issues I see.

In this line:

my_thread = threading.Thread(target=my_target, args=(device_ip, DeviceName, *my_args, **my_keyword_args))

You don't need the asterisks in front of *my_args and **my_keyword_args The asterisk goes in the function parameters but inside of the function you can just refer to them as my_args or my_keyword_args. Secondly, for the args parameter, you can remove the my_keyword_args from that and instead put it at the end. Here's a basic format to help clarify:

import threading

def start_thread(function_name, *args, **kwargs):
    t = threading.Thread(target=function_name, args=args, kwargs=kwargs)
    t.daemon = True
    t.start()

def this_and_that(input_data, started=False, finished=False):
    for i in names_list:
        if started is True and finished is False:
            print(i, started, finished)
        elif started is True and finished is True:
            print(i, started, finished)
        else:
            print(i, started, finished)

n = ['Steve','Smith']

start_thread(this_and_that, n, started=True, finished=True)

Output looks like this:

Steve True True
Smith True True

Passing *args or **kwargs into threading.Thread()

2 Answers2