0

Preface -- I'm new to Python and programming in general. I am inheriting code and am attempting to modify existing code to make it more efficient.

Current issue is an existing "for server in serverlist" loop that appends the output to a list. It takes too long because it is processing the list one-by-one. It takes about 5seconds per machine lasting around 4minutes.

What do I need to do if I want to execute this single call on all 50 servers at once thereby getting back data for all 50 servers within a total of around 5 seconds? Or rather -- is there anything that can be done to speed this up in general to call on multiple machines at once instead of just one?

Current code example:

def dosomething(server):
    stuff = []
    try:
        # function here
        stuff = # function stuff here
    except Exception:
        pass
    return stuff

def getdata(self):
    serverlist = [servera, serverb, serverc]
    data = []
    for server in serverlist:
        results = dosomething(server)
        data.append(results)
    return data
mjfjones
  • 1
  • 1
  • 3
    You may need to take a look at `threading` https://docs.python.org/3/library/threading.html – Bahrom Mar 25 '16 at 23:34
  • You may want to take a look at this answer: https://stackoverflow.com/questions/3329361/python-something-like-map-that-works-on-threads - your loop is essentially equivalent to a map (`data = map(dosomething, serverlist)`) - that question explains how you can do a map using threads to execute multiple iterations simultaneously. – Gareth Latty Mar 25 '16 at 23:36
  • @Latty -- I must have looked all over stackoverflow and my queries never brought me to that post. THANK YOU! I made the following changes and I'm hitting all 50 servers at once: def getdata(self): serverlist = [servera, serverb, serverc] data = [] p = multiprocessing.dummy.Pool(50) data = p.map(dosomething,serverlist) return data – mjfjones Mar 25 '16 at 23:47

0 Answers0