I have a function that i want to process for thousands of objects in parallel.
EDIT:
Using Pool from: https://docs.python.org/3/library/multiprocessing.html
Using Client object from: https://github.com/ping/instagram_private_api
from multiprocessing import Pool
from instagram_private_api import Client
from database.models import Account # SQLAlchemy model
# MAIN FUNCTION
accounts = get_all_accounts() # get accounts from DB
p = Pool(len(accounts)) # len of accounts is ~100
results = p.map(login, accounts)
p.terminate()
p.join()
# LOGIN FUNCTION
def login(acc: Account):
client = Client(proxy=acc.proxy)
client = Client(username=acc.username, password=acc.password)
client.login() # request is sent here
I am using the Pool
object from the multiprocessing
library and inside the function, i have two GET requests to external API ( note: I'm not using requests
or urllib
, i'm using a third-party library for sending the requests).
However for a lot of the requests I get the URLError <urlopen error timed out>
error when i use Pool(100)
( 100 parallel processes ) , if i use less ( i.e. 10 ) i dont get any errors.
Any idea on how to overcome this issue if it is possible?