6

I had a for-loop that would take a URL from a set of URLs and reach out to that URL and do some other stuff, but it was taking forever, so I figured I'd speed it up with some multiprocessing, but am struggling to do so.

Thank you for any assistance.

def accessAndSaveFiles(urlSet, user, verboseFlag):
    with multiprocessing.Pool(os.cpu_count()) as pool:
        pool.starmap(processURL, zip(itertools.repeat(urlSet), user, verboseFlag))

def processURL(url, user, verboseFlag):
    filePath = some_path

    img_data = requests.get(url, allow_redirects=True)
    open(filePath, 'wb').write(img_data.content)


def main():
    ...
    accessAndSaveFiles(urlSet, user, verboseFlag)
    ...

I get an error on the "pool.starmap(processURL, zip(itertools.repeat(urlSet), user, verboseFlag))" line saying "TypeError: zip argument #3 must support iteration". I don't want to iterate over this item, I just want to send the same value every time.

Karl Knechtel
  • 62,466
  • 11
  • 102
  • 153
Ryan
  • 103
  • 1
  • 1
  • 6
  • do all the processURL have the same user and verboseflag? – Ankur S Jan 04 '19 at 17:46
  • Both are specified on the command line. So, yes, but I can't hard code it – Ryan Jan 04 '19 at 18:00
  • Does this answer your question? [How to use multiprocessing pool.map with multiple arguments](https://stackoverflow.com/questions/5442910/how-to-use-multiprocessing-pool-map-with-multiple-arguments) – Karl Knechtel Feb 17 '23 at 00:49

1 Answers1

7

Assuming that urlset is an iterable, you should use

pool.starmap(processURL, zip(urlSet, repeat(user), repeat(verboseFlag)))

This is because you want to iterate over the urlset but have the same user and verboseFlag for each processURL instance(thus,repeat)

For reference you should take a look at Python multiprocessing pool.map for multiple arguments

The output of zip when iterated over, should look something like

[('www.google.com','user1',True),('www.goodle.uk','user1',True),]

for pool.starmap to make sense of it.

Ankur S
  • 548
  • 3
  • 18