Python threading did not execute all threads which created

Question

I have a python script, so I use threading module in python to execute concurrently.

class check_bl(threading.Thread):
    def __init__(self, ip_client, reverse_ip, key, value):
        threading.Thread.__init__(self)
        self.ip_client = ip_client
        self.reverse_ip = reverse_ip
        self.key = key
        self.value = value

def run(self): db = MySQLdb.connect('localhost', 'mytable', 'user', 'mytable') cursor = db.cursor() query = "dig +short " + str(reverse_ip) + "." + key try: output = subprocess.check_output(query, shell=True) output_edited = output.strip() except: output_edited = "Timeout occured"

    if output_edited in value:
        try:
            cursor.execute() # execute INSERT command
            db.commit()
        except:
            db.rollback()
        db.close()
    else:
        try:
            cursor.execute() # execute INSERT command
            db.commit()
        except:
            db.rollback()
        db.close()

for ip_client in IPNetwork(range_ip_client): ip_client = .... reverse_ip = ...

for key, value in my_dictionary.items():
    myThread = check_bl(ip_client, reverse_ip, key, value)
    myThread.start()
    threadList.append(myThread)

for t in threadList:
t.join()

The problem is each some key, value in my_dictionary did not execute(not INSERT to database, although some key, value execute more than one time). My dictionary has 30 items, but just 10 - 20 of them execute. If I insert join() like 2 example below, all my dictionary item() execute but at the same time, just one thread execute. Is there something wrong for every threads created, it create an database connection. So database can't handle it.

for key, value in my_dictionary.items():
    myThread = check_bl(ip_client, reverse_ip, key, value)
    myThread.start()
    threadList.append(myThread)

for t in threadList:
    t.join()

or

for key, value in my_dictionary.items():
    myThread = check_bl(ip_client, reverse_ip, key, value)
    myThread.start()
    t.join()

Thanks Dksj, my problem is solved. Maybe something wrong with class method, just need to define a function.

def check_bl(ip_client, reverse_ip, key, value):
    db = MySQLdb.connect('localhost', 'mytable', 'user', 'mytable')
    cursor = db.cursor()
    query = "dig +short " + str(reverse_ip) + "." + key
    try:
        output = subprocess.check_output(query, shell=True)
        output_edited = output.strip()
    except:
        output_edited = "Timeout occured"

    if output_edited in value:
        try:
            cursor.execute() # execute INSERT command
            db.commit()
        except:
            db.rollback()
        db.close()
    else:
        try:
            cursor.execute() # execute INSERT command
            db.commit()
        except:
            db.rollback()
        db.close()

for ip_client in IPNetwork(range_ip_client):
    ip_client = ....
    reverse_ip = ...
    for key, value in my_dictionary.items():
        myThread = check_bl(ip_client, reverse_ip, key, value)
        threadList.append(myThread)

[t.start() for t in threadList]

[t.join() for t in threadList]

score 0 · Answer 1 · edited May 23 '17 at 12:08

0

My problem is slight different from your's and I didn't use class method I just modified your code based on mine you may give a try. This link helped me to solve my problem : How to use threading in Python?

def check_bl(ip_client, reverse_ip, key, value): ....

threads []
 for key, value in my_dictionary.items():
    # Instantiates the thread
    myThread = Thread( target=check_bl, args=(ip_client, reverse_ip, key, value) )
    #add thread to list 
    threads.append(myThread)

[th.start() for th in threads]
#Join all the threads to make sure the parent waits for all of them to finish
[th.join() for th in threads]

edited May 23 '17 at 12:08

Community

1
1

answered Jul 14 '16 at 14:37

Dksj

1
2

Although this code may be help to solve the problem, providing additional context regarding _why_ and/or _how_ it answers the question would significantly improve its long-term value. Please [edit] your answer to add some explanation. – Toby Speight Jul 14 '16 at 14:54
Thank you so much, I solved my problem by follow your code. Exactly, just define a function, don't need to use class method. I will post my edited code above. – ThOong Ku Jul 15 '16 at 01:48

Python threading did not execute all threads which created

1 Answers1