How to use nested loops in joblib library in python

Question

Actual code looks like:

def compute_score(row_list,column_list): 

    for i in range(len(row_list)):
            for j in range(len(column_list)):
                tf_score = self.compute_tf(column_list[j],row_list[i])

I am tying to achieve multi-processing i.e. at every iteration of j I want to pool column_list. Since compute_tf function is slow I want to multi-process it.

I've found have to do it using joblib in Python, But I am unable to workaround with nested loops.

Parallel(n_jobs=2)(delayed(self.compute_tf)<some_way_to_use_nested_loops>)

This is what is to be achieved. It would be a great help if any solution on this is provided or any-other solution.

score 14 · Accepted Answer · answered Aug 03 '16 at 12:05

Another solution without having to implement a generator function, is to use the nested list comprehension for the generator:

Parallel(n_jobs=2)(delayed(self.compute_tf)(i, j) for j in column_list for i in row_list)

The order will be given as:

[(i, j) for j in range(10) for i in range(10)]

score 3 · Answer 2 · answered Apr 05 '16 at 10:53

I'm not understanding your part about Parallel. However, your code can be simplified to this:

def compute_score(row_list,column_list): 

    for i in row_list:
            for j in column_list:
                tf_score = self.compute_tf(j,i)

It may be useful to create a generator that yields each j,i so that you can process them outside of the generator

def compute_pairs(row_list,column_list): 

    for i in row_list:
            for j in column_list:
                yield j, i

then use:

 scored = [compute_tf(j,i) for j, i in compute_pairs(row_list, column_list)]

`compute_pairs` doesn't even need to be spelled out - `itertools.product` does the exact same thing. — user4815162342, Feb 02 '17 at 23:03

How to use nested loops in joblib library in python

2 Answers2

Linked