I've got a list of links in my program, and I want to visit these links and collect some data from them, also I've got three other empty lists which I will append to based on the result. I don't know how to use multiprocessing on this task.
All processes should read from the main list and will write(append) to one of other three lists.
How to distribute links between processes.
Is multiprocessing even suitable for this task? If yes witch method should be used?
def remove_redirect(): main_list = get_data_from_db() empty_list_1 = [] empty_list_2 = [] empty_list_3 = [] for link in main_list: req_result = manage_requests(link) if isinstance(req_result, bool): pass else: try: if req_result.find('div', class_="errorName"): empty_list_1.append(link) elif req_result.find('section', class_='filter'): empty_list_2.append(link) elif req_result.find('div', class_="proStatus"): empty_list_3.append(link) except Exception as error: print(error) pass if __name__ == "__main__": processes = [] for i in range(os.cpu_count()): print('Registering process %d' % i) processes.append(Process(target=remove_redirect)) for process in processes: process.start() for process in processes: process.join()