How to make a loop add a function to a pool of multiprocessing tasks in PYTHON?

Question

I need some help with multiprocessing in Python. I want my code to extract a link from each line of a CSV file and then feed each into an instance of function. I need these functions to be running at the same time. Currently my code pulls the link from the CSV and feeds it to the function one by one - meaning no more link functions can be started until the one before it has finished.

I think threading or multiprocessing may be needed however I haven't managed to implement it properly using a loop.

Any help would be greatly appreciated! Thanks

links = csv.reader(open('links.csv','r'))
row = list(csv.reader(open('links.csv','r')))
row_count = sum(1 for row in links)
for i in range(0,row_count):
    link = ((str(row[i])).replace("'","")).replace("[","").replace("]","")
    print(link)
    LINKFUNCTIONPROCESS(link)

Please don't take this as bragging, this is a well intended advice: I really recommend getting familiar with the basics before diving into concurrency. There is a lot of things wrong with the code you presented here. — shmee, Sep 12 '18 at 13:39

score 0 · Answer 1 · answered Sep 12 '18 at 13:31

From this answer

import multiprocessing
p = multiprocessing.Pool()
links = csv.reader(open('links.csv','r'))
row = list(csv.reader(open('links.csv','r')))
row_count = sum(1 for row in links)
for i in range(0,row_count):
    link = ((str(row[i])).replace("'","")).replace("[","").replace("]","")
    print(link)
    p.apply_async(LINKFUNCTIONPROCESS, [link]) 


p.close()
p.join()

How to make a loop add a function to a pool of multiprocessing tasks in PYTHON?

1 Answers1