0

This is a sniplet of my code:

data = [currentAccount.login,currentAccount.password,campaign.titlesFile,campaign.licLocFile,campaign.subCity,campaign.bodiesMainFile,campaign.bodiesKeywordsFile,campaign.bodiesIntroFile]
results = multiprocessing.Pool(5).map(partial(self.postAd,data),range(3))
...
def postAd (self,login,password,titlesFile,licLocFile,subCity,bodiesMainFile,bodiesKeywordsFile,bodiesIntroFile):
...

(Just so you know what's going on: currentAccount and campaign are classes, those are variables within those classes. Using self b/c this is all being run in a class. I'm trying to run self.postAd 3x passing it all the variables I have in data)

When I run that it says " postAd() missing 6 required positional arguments: 'titlesFile', 'licLocFile', 'subCity', 'bodiesMainFile', 'bodiesKeywordsFile', and 'bodiesIntroFile'"

What am I doing wrong? Why does it only accept 2 variables?

If I cant use Pool map, how should I be doing this?

I also tried this with no success:

results = multiprocessing.Pool(5).map(lambda args: self.postAd(currentAccount.login,currentAccount.password,campaign.titlesFile,campaign.licLocFile,campaign.subCity,campaign.bodiesMainFile,campaign.bodiesKeywordsFile,campaign.bodiesIntroFile), range(3))

Error: Can't pickle <function NewPostService.processNewAds.<locals>.<lambda> at 0x0000000002F3CBF8>: attribute lookup <lambda> on functions failed
Daniel Rusu
  • 115
  • 1
  • 11
  • I also tried this with no success: results = multiprocessing.Pool(5).map(lambda args: self.postAd(currentAccount.login,currentAccount.password,campaign.titlesFile,campaign.licLocFile,campaign.subCity,campaign.bodiesMainFile,campaign.bodiesKeywordsFile,campaign.bodiesIntroFile), range(3)) Error: Can't pickle . at 0x0000000002F3CBF8>: attribute lookup on functions failed – Daniel Rusu Sep 28 '16 at 23:22
  • Do you really want all three calls to use *exactly* the same arguments? I'm not sure that works with `map`, since it will always want to be providing an extra argument from the sequence it loops over (the `range` in this case). You can probably make it work by writing an extra function that ignores its last argument (and using `partial(self.postadd, *data)`), but it might be easier to use your own loop creating `Process`es. – Blckknght Sep 28 '16 at 23:41
  • Yes I want it to use the same arguments. I want the same process to happen 3x simultaneously is the reason why. How would I do this " but it might be easier to use your own loop creating Processes." Can you give me an example based on my code if you'd be so kind – Daniel Rusu Sep 29 '16 at 00:26

1 Answers1

0

Your first attempt is a misuse of partial. data is a single argument: it being a list doesn't automatically unpack its contents. partial simply takes variable arguments and so you should pass those arguments 'normally', either 

partial(self.postAd, currentAccount.login,currentAccount.password, ...

or

partial(self.postAd, *data)

The reason it says that postAd received two arguments instead of just one (data) is that it also implicitly received the self argument.

Your second attempt is the right idea and very close, but it so happens that in older versions of python pickle (which is essential for multiprocessing) can't handle lambdas. Replace the lambda with a named function defined using def:

def postNow(arg):  # Accepts an argument from map which it ignores
    self.postAd(currentAccount.login, ..., campaign.bodiesIntroFile)

Side notes:

It's a bit odd that your argument to the lambda is called args when it's just one argument. If your intention is to make it flexible and accept variable arguments, use lambda *args: ... or even lambda *args, **kwargs: ... to accept keyword arguments. The names are not important, they are just convention. The important part is the * and **. Note that partial has a signature like this, as an example.

I have never seen this issue with the second attempt before. I learnt it by Googling the error message. Always Google things.

Alex Hall
  • 34,833
  • 5
  • 57
  • 89
  • Thanks for the response Alex. I'm still a beginner python programmer so some of this is over my head. I tried doing both options of 1. partial(self.postAd, *data) and 2. multiprocessing.Pool(5).map(partial(self.postAd,currentAccount.login,currentAccount.password,campaign.titlesFile,campaign.licLocFile,campaign.subCity,campaign.bodiesMainFile,campaign.bodiesKeywordsFile,campaign.bodiesIntroFile),range(3) ----------------- . and got this error for both of them: Error: postAd() takes 9 positional arguments but 10 were given. I'm not sure where the 10th argument is as I'm only passing 8. – Daniel Rusu Sep 29 '16 at 02:49
  • @DanielRusu it's getting the 8 normal arguments, `self`, and the 10th is the value from `range(3)` passed in by `map`. It wouldn't make much sense to modify `postAd` for this case so actually just forget about trying to use `partial`, it's not intended for this. – Alex Hall Sep 29 '16 at 10:53