0

I have a list with more than 500 elements, and I want to run a function for each element in the list at the same time, how can I do that?

That solution will speed up the process, and also, my code will be shorter.

Im using one lib for Instagram, and getting the values from that lib requires a lot of time. I have managed the speed it with threads but im dealing with large number of values in list. Each element is an username, so I want to go to profile of 500 users at the same time and collect some data from it (post picture, number of comments, number of likes...).

For each user it takes around 1-2 seconds for collecting data. That means that collecting data for 1000 users will take 1000-2000 seconds. So Im trying to make this faster by running the same function at the same time for every item in list.

I was reading a little about threading and I have made something like this, but what if I have 500 elements in list:

import threading

list_of_results = []
list_of_values = [1,2,3]

def one(list_of_values, list_of_results):

    result = 0
    for i in range(0,5000):
        result = list_of_values[0] + i

    list_of_results.append(result)

def two(list_of_values, list_of_results):

    result = 0
    for i in range(0,5000):
        result = list_of_values[1] + i

    list_of_results.append(result)

def three(list_of_values, list_of_results):

    result = 0
    for i in range(0,5000):
        result = list_of_values[2] + i

    list_of_results.append(result)

t1 = threading.Thread(target = one, args = (list_of_values, list_of_results))
t2 = threading.Thread(target = two, args = (list_of_values, list_of_results))
t3 = threading.Thread(target = three, args = (list_of_values, list_of_results))

t1.start()
t2.start()
t3.start()

t1.join()
t2.join()
t3.join()

print(list_of_results)
taga
  • 3,537
  • 13
  • 53
  • 119
  • 1
    You're looking for `multiprocessing.Pool`, and specifically `pool.imap(my_function, my_list)`. However, you should make clear what takes a lot of times: from what you're saying, it seems it is instagram or your netwrok, and making 500 queries at a time may not be faster than the sequential method. – Demi-Lune Oct 21 '19 at 13:08
  • 1
    Your question is too broad. For any speed improvement, a "bottleneck" should be indentified first. The most common types of limits are summarized here: https://stackoverflow.com/a/868664/5378816 – VPfB Oct 21 '19 at 13:11
  • I have edited the question. – taga Oct 21 '19 at 13:17

1 Answers1

-1

You can't do 500 operations at once. Your PC is sequencial. The Thread might help you with more complex operations => It will make a process on each of your PC's cores.

It looks like you need to write a for loop inside another for loop.

What exactly do you want to do with these 500 elements?

Maybe you need something like this?:

for j in range(0, 500)
    for i in range(0,5000):
        result = list_of_values[j] + i
Ivan
  • 1,352
  • 2
  • 13
  • 31
  • thats not what I need. Im using one lib for Instagram, and getting the values from that lib requires a lot of time. I have managed the speed it with `threads` but im dealing with large number of values in list. Each elemet is an username, so I want to go to profile of 500 users at the same time – taga Oct 21 '19 at 12:59