Shared variables in Python

Question

I am trying to create multiple threads of bot and they share some variables, but I am failing miserably in getingt the shared variables to work.

Here is the code:

import requests
import sys
import threading
import signal
import time

class bot(threading.Thread):
    terminate = False
    #def __init__(self):
    #   threading.Thread.__init__(self)
    #   self.terminate = False

    def getCode():
        code_lock.acquire()
        work_code = code
        try:
            code += 1
        finally:
            code_lock.release()
        return work_code

    def checkCode(code):
        try:
            #if(code % 1000000 == 0):
            print("Code "+str(code)+" is being checked...\n")
            html = requests.get(url+str(code))
            html.encoding = 'utf-8'
            return not 'Page Not Found' in html.text
        except requests.exceptions.ConnectionError:
            print("Connection Error! Retrying...\n")
            time.sleep(0.5)
        except KeyboardInterrupt:
            logCode(code)
            sys.exit()

    def storeCode(code):
        file_lock.acquire()
        try:
            file.write(code+'\n')
        finally:
            file_lock.release()

    def logCode(code):
        log_lock.acquire()
        try:
            log.write(code+'\n')
        finally:
            log_lock.release()

    #def run(self):
    #   global bots
    #   global url
    #   global file
    #   global log
    #   global code_lock
    #   global file_lock
    #   global log_lock
    while(not terminate):
        code = getCode()
        if(checkCode(code)):
            storeCode(code)
    logCode(code)


def main(code = 0, threads = 16):

    #bots = [threading.Thread(target=bot) for bot in range(threads)]
    bots = []
    url = 'https://test.ing/codes/'
    file = open("valid-codes.txt", "a")
    log = open("log.txt", "a")
    code_lock = threading.Lock()
    file_lock = threading.Lock()
    log_lock = threading.Lock()

    def signal_handler(signal, frame):
        print('Exiting...\n')
        log_lock.acquire()
        try:
            log.write("\n\n"+str(time.strftime("%Y-%m-%d %H:%M:%S", time.gmtime()))+"\n")
        finally:
            log_lock.release()
        for bot in bots:
            bot.terminate = True
        for bot in bots:
            bot.join()
        sys.exit(0)

    #for bot in bots:
    #    bot.start()

    for i in range(threads):
        t = bot()
        bots.append(t)
        t.start()

    signal.signal(signal.SIGINT, signal_handler)
    while True:
        signal.pause()

main(736479509787350, 1)

With this code I get this error:

Traceback (most recent call last): File "bot.py", line 7, in

class bot(threading.Thread):   File "bot.py", line 59, in bot

code = getCode()   File "bot.py", line 14, in getCode

code_lock.acquire() NameError: name 'code_lock' is not defined

I don't know if I should override the run(self) method of bot, but when I tried that it never actually ran the method run and I also receive the same error from all the threads created: that int is not callable (and I can't see where I can possibly be using an int as object).

Additionaly I don't know if I am handling correctly the exit signal from keyboard, as you can see I am trying to deal with that using a terminate variable, but I don't think that this is the problem...

One last thing, the ConnectionError exception is not being appropriately handled, as it's saying "Retrying...", but in fact it will not retry, but I am aware of that and it should be ok, I'll fix it latter.

Worth mentioning that I'm not very used to deal with multi-threading and when I do deal with it, it is in C or C++.

Edit

I can make the code work by using global variables, but I do not want to do that, I prefer to avoid using globals. My attempts of passing the variables directly to the instances of the class bot or by passing an data-object to it weren't successful so far, whenever I pass the variables or the auxiliar object to bot I am unable to access them as attributes using self. and without self. Python claims that the variable was not defined.

Here is the updated code, without success yet:

import requests
import sys
import threading
import signal
import time

class Shared:
    def __init__(self, code, url, file, log, code_lock, file_lock, log_lock):
        self.code = code
        self.url = url
        self.file = file
        self.log = log
        self.code_lock = code_lock
        self.file_lock = file_lock
        self.log_lock = log_lock

class bot(threading.Thread):
    def __init__(self, data):
        threading.Thread.__init__(self)
        self.terminate = False
        self.data = data

    @classmethod
    def getCode(self):
        self.data.code_lock.acquire()
        work_code = self.data.code
        try:
            self.data.code += 1
        finally:
            self.data.code_lock.release()
        return work_code

    @classmethod
    def checkCode(self, work_code):
        try:
            #if(code % 1000000 == 0):
            print("Code "+str(work_code)+" is being checked...\n")
            html = requests.get(self.data.url+str(work_code))
            html.encoding = 'utf-8'
            return not 'Page Not Found' in html.text
        except requests.exceptions.ConnectionError:
            print("Connection Error! Retrying...\n")
            time.sleep(0.5)
        except KeyboardInterrupt:
            self.logCode(work_code)
            sys.exit()

    @classmethod
    def storeCode(self, work_code):
        self.data.file_lock.acquire()
        try:
            self.data.file.write(work_code+'\n')
        finally:
            self.data.file_lock.release()

    @classmethod
    def logCode(self, work_code):
        self.data.log_lock.acquire()
        try:
            self.data.log.write(work_code+'\n')
        finally:
            self.data.log_lock.release()

    @classmethod
    def run(self):
        while(not self.terminate):
            work_code = self.getCode()
            if(self.checkCode(work_code)):
                self.storeCode(work_code)
        self.logCode(work_code)


def main(code = 0, threads = 16):

    #bots = [threading.Thread(target=bot) for bot in range(threads)]
    bots = []
    url = 'https://www.test.ing/codes/'
    file = open("valid-codes.txt", "a")
    log = open("log.txt", "a")
    code_lock = threading.Lock()
    file_lock = threading.Lock()
    log_lock = threading.Lock()

    data = Shared(code, url, file, log, code_lock, file_lock, log_lock)

    def signal_handler(signal, frame):
        print('Exiting...\n')
        log_lock.acquire()
        try:
            log.write("\n\n"+str(time.strftime("%Y-%m-%d %H:%M:%S", time.gmtime()))+"\n")
        finally:
            log_lock.release()
        for bot in bots:
            bot.terminate = True
        for bot in bots:
            bot.join()
        sys.exit(0)

    #for bot in bots:
    #    bot.start()

    for i in range(threads):
        t = bot(data)
        bots.append(t)
        t.start()

    signal.signal(signal.SIGINT, signal_handler)
    while True:
        signal.pause()

main(736479509787350, 4)

Yet, the working code with global variables:

import requests
import sys
import threading
import signal
import time

code = 736479509787350
url = 'https://www.test.ing/codes/'
file = open("valid-codes.txt", "a")
log = open("log.txt", "a")
code_lock = threading.Lock()
file_lock = threading.Lock()
log_lock = threading.Lock()

terminate = False

class bot(threading.Thread):
    def __init__(self):
        threading.Thread.__init__(self)

    @classmethod
    def getCode(self):
        global code
        code_lock.acquire()
        work_code = code
        try:
            code += 1
        finally:
            code_lock.release()
        return work_code

    @classmethod
    def checkCode(self, work_code):
        try:
            if(code % 1000000 == 0):
                print("Code "+str(work_code)+" is being checked...\n")
            html = requests.get(url+str(work_code))
            html.encoding = 'utf-8'
            if(not 'Page Not Found' in html.text):
                time.sleep(0.5)
                html = requests.get(url+str(work_code)+":999999999")
                html.encoding = 'utf-8'
                return 'Page Not Found' in html.text
        except requests.exceptions.ConnectionError:
            #print("Connection Error! Retrying...\n")
            time.sleep(1)
            return self.checkCode(work_code)
        except KeyboardInterrupt:
            self.logCode(work_code)
            sys.exit()

    @classmethod
    def storeCode(self, work_code):
        global file_lock
        global file
        file_lock.acquire()
        try:
            file.write(str(work_code)+'\n')
        finally:
            file_lock.release()

    @classmethod
    def logCode(self, work_code):
        global log_lock
        global log
        log_lock.acquire()
        try:
            log.write(str(work_code)+'\n')
        finally:
            log_lock.release()

    @classmethod
    def run(self):
        global terminate
        while(not terminate):
            work_code = self.getCode()
            if(self.checkCode(work_code)):
                print("Code "+str(work_code)+" is a valid code!\n")
                self.storeCode(work_code)
        self.logCode(work_code)


def main(threads = 16):

    #bots = [threading.Thread(target=bot) for bot in range(threads)]
    bots = []
    #url = 'https://www.facebook.com/leticia.m.demenezes/posts/'
    #file = open("valid-codes.txt", "a")
    #log = open("log.txt", "a")
    #code_lock = threading.Lock()
    #file_lock = threading.Lock()
    #log_lock = threading.Lock()

    def signal_handler(signal, frame):
        global terminate
        print('Exiting...\n')
        log_lock.acquire()
        try:
            log.write("\n\n"+str(time.strftime("%Y-%m-%d %H:%M:%S", time.gmtime()))+"\n")
        finally:
            log_lock.release()
        terminate = True
        for bot in bots:
            bot.join()
        sys.exit(0)

    #for bot in bots:
    #    bot.start()

    for i in range(threads):
        t = bot()
        bots.append(t)
        t.start()

    signal.signal(signal.SIGINT, signal_handler)
    while True:
        signal.pause()

main()

Side-note: _Always_ use `with` statements with locks. It's much more obvious what is protected, much harder to get wrong and much simpler. `with code_lock:` followed by a block containing the protected accesses is much simpler than an explicit `.acquire()`, followed by a `try` block with the protected accesses and a `finally` with an explicit `.release()`. — ShadowRanger, Jan 28 '16 at 13:03
@ShadowRanger Thanks! I wasn't aware of that, great to know! — Rodrigo Oliveira, Jan 28 '16 at 13:07

score 1 · Answer 1 · edited May 23 '17 at 11:51

1

You could make the code_lock global as you're trying to do, but why not just pass it into each bot class?

t = bot(code_lock)

Next create a constructor for your class:

class bot(threading.Thread):
    def __init__(self, code_lock):
       threading.Thread.__init__(self)
       self.code_lock = code_lock

Now, whenever you try to use code_lock inside your bot class, always prefix it with self (self.code_lock).

If you really insist on using global variables, then look into the global keyword.

edited May 23 '17 at 11:51

Community

1
1

answered Jan 27 '16 at 16:48

Martin Konecny

57,827
19
139
159

Yes, I made them global once and the code worked, but I am not for using globals. I tried to pass each variable to the class (I think that a problem would appear when passing `code` as it would be passed by value) and I also tried creating an object to hold the necessary arguments, but I seem to be unable to access the passed object (or variables when tried then individually) within the class methods. I will update my question to reflect that. Thanks for your attention. – Rodrigo Oliveira Jan 28 '16 at 12:39
And one more thing, when you make `self.code_lock = code_lock`, will changes in the instance's `code_lock` affect the other instances' `code_lock`s? I fear you are making `code_lock` local, thus independent, to each bot instance. I want `code_lock` to be a shared variable. – Rodrigo Oliveira Jan 30 '16 at 12:49
Nope, `code_local` is not becoming local. There is only one copy and that it when you do `code_lock = threading.Lock()`. After that you are only passing around *references* to this single lock object. – Martin Konecny Jan 30 '16 at 17:01
For your updated question, use `self.code_lock.acquire()` instead of `self.data.code_lock.acquire()` – Martin Konecny Jan 30 '16 at 17:02

score 0 · Answer 2 · answered Jan 27 '16 at 17:18

0

It's clear that you are trying to access code_lock out of it's scope, may be you can follow #MartinKonecny suggestion to fix that.

I could see that even after fixing code_lock problem, your code has lot of problems. as soon as you fix code_lock problem you'll face similar issue with the variable code in the same function getCode.

After fixing all those compiled time issues, you'll face issues with your whole class implementation, this is not the way to implement Python classes.

It's better that you go through this to know more about python name spaces and classes.

answered Jan 27 '16 at 17:18

gsb-eng

1,211
1
9
16

Well, I think you are right, I am not doing such a pythonic class implementation, I think I am failing dealing with that because I'm just seeing the class as a thread, as if the inside of the class was just a plain new file to write my Python code into it. I think I should not be creating methods and using them within the class, maybe I am carrying habits of C++ namespaces. But what else should I do? – Rodrigo Oliveira Jan 28 '16 at 12:44

Shared variables in Python

Edit

2 Answers2