Python - combine two lists into a single tuple - for multiprocessing pool.map

Question

I am trying to pass multiple arguments to a function using a single tuple. The reason for this odd approach is that my actual program is following this example to pass multiple arguments to a multiprocessing pool.map.

The problem is that the example uses only two arguments, but I need more. I end up passing two lists of arguments, when I really just want a single tuple of multiple entries.

I end up having to redefine the individual variables in the function (e.g. rowTest = data_to_check[0]). But this is ugly and cumbersome. What is the better way to do this, please?

Here is a reduced example:

import itertools

def worker(data_to_check, support_data):
    # UGLY -- do not want to have to reassign internally
    rowTest = data_to_check[0]
    rowProd = data_to_check[1]
    countRow = support_data[0]
    precision = support_data[1]    
    print 'data:', rowTest, rowProd    
    print 'params:', countRow, precision

def worker_star(input_tuple):
    """Convert `f([1,2])` to `f(1,2)` call."""
    print 'input_tuple:', input_tuple
    return worker(*input_tuple)

def main():
    # data arguments to pass - iterate through all pairs
    groups_test = [ 1, 2, 3]
    groups_prod = [-1,-2,-3]
    groups_combined = zip(groups_test,groups_prod)

    # supporting arguments to pass (repeat on each call)
    countRow = 0
    precision = 4

    worker_params = [countRow, precision]

    # producing a tuple of lists (just need a simple tuple?)        
    map(worker_star, itertools.izip( [ [x[0],x[1]] for x in groups_combined ], itertools.repeat(worker_params) ))

    print 'done'

if __name__ == '__main__':
        main()

score 0 · Answer 1 · answered Jun 23 '14 at 01:10

One incremental improvement:

def worker(data_to_check, support_data):
    rowTest, rowProd = data_to_check
    countRow, precision = support_data

Or

groups_test = [ 1, 2, 3]
groups_prod = [-1,-2,-3]
countRows = [0]*4
precisions = [4]*4
groups_combined = zip(groups_test, groups_prod, countRows, precisions)
# [(1, -1, 0, 4), (2, -2, 0, 4), ...]

map(worker_star, groups_combined)  # redefine worker_star to accept 4 parameters

score 0 · Accepted Answer · edited May 23 '17 at 12:29

I found this related example showing how to pass more than 2 arguments. Basically, I modified the helper function worker_star to flatten out the tuple before passing it on:

import itertools

#def worker(data_to_check, support_data):
def worker(rowTest, rowProd, countRow, precision):
# UGLY -- do not want to have to reassign internally
#     rowTest = data_to_check[0]
#     rowProd = data_to_check[1]
#     countRow = support_data[0]
#     precision = support_data[1]    
    print 'data:', rowTest, rowProd    
    print 'params:', countRow, precision

def worker_star(input_tuple):
    """Convert `f([1,2])` to `f(1,2)` call."""
    # flatten out the tuple before passing it on
    input_tuple_flat = input_tuple[0] + input_tuple[1]
    #return worker(*input_tuple)
    return worker(*input_tuple_flat)

def main():
    # data arguments to pass - iterate through all pairs
    groups_test = [ [1], [2], [3]]
    groups_prod = [[-1],[-2],[-3]]
    groups_combined = zip(groups_test,groups_prod)

    # supporting arguments to pass (repeat on each call)
    countRow = 0
    precision = 4

    worker_params = (countRow, precision)

    # producing a tuple of lists (just need a tuple?)        
    map(worker_star, itertools.izip( ( (x[0],x[1]) for x in groups_combined ), itertools.repeat(worker_params) ))

    print 'done'

if __name__ == '__main__':
    main()

Python - combine two lists into a single tuple - for multiprocessing pool.map

2 Answers2