-1

I am using multiprocessing as given below:

        with ProcessPoolExecutor(max_workers=3) as exe:
            result = exe.map(self.extraction, tokens)

tokens is a list. Problem is its not getting into extraction function. Tried with print statements within function, not printing at all.

Anjaly Vijayan
  • 237
  • 2
  • 9

2 Answers2

1

I see two potential reasons why with the code you provide, your function isn't being called, but no error is occurring either:

1 - self.extraction does not have the right signature. If the supplied function isn't one that takes exactly one argument, it won't be called. In the case of a instance function (method), this does not include self. So your method's signature should look like this:

def extraction(self, token):
    ...

I ran into this case myself a few weeks ago and it cost me a few hours. It's strange that the framework does not complain about the function's signature, and yet doesn't call it either.

2 - tokens is an empty iterable. Since the function will be called once for each item in the iterable, if it is empty, it will never be called. If tokens isn't an iterable at all, it seems that you get an error stating this.

CryptoFool
  • 21,719
  • 5
  • 26
  • 44
  • thanks for the reply! tokens is not an empty list. and my function is like this only def extraction(self, token): – Anjaly Vijayan Oct 06 '20 at 02:23
  • wow, bummer. If you can't yet be 100% sure that this code is being run, you might want to put a print statement just before this code just to make sure. But if you have print statements before this code, and a print statement as the first line of your function, then I'm truly baffled. The only other thing I can think of is that something about the objects in the iterable does make a difference, and causes the behavior to change. This is truly strange. I've worked with this function a lot. If what you say is true, your function should be getting called regardless of the rest of your code. – CryptoFool Oct 06 '20 at 02:38
0

I found the issue and resolved it referring this link Multiprocessing: How to use Pool.map on a function defined in a class?

My code is working like this :

        with ProcessPoolExecutor(max_workers=3) as exe:
            result = exe.map(ExtractSuggestions.extract_suggestion, tokens)

ExtractSuggestions is my class

Anjaly Vijayan
  • 237
  • 2
  • 9
  • Can you tell us what was wrong with the code that was causing it to fail? Telling us why one version works and another does not would make this question much more valuable. It's the non-working case that is the interesting one here. – CryptoFool Oct 06 '20 at 02:40
  • I looked at the link you provide. I don't see how it applies to your situation. Maybe it caused you to make a change that got your code working, but I don't see it explaining what was wrong with your first version. – CryptoFool Oct 06 '20 at 02:43