Maintaining log file from multiple threads in Python

Question

I have my python baseHTTPServer server, which handles post requests. I used ThreadingMixIn and its now opens a thread for each connection. I wish to do several multithreaded actions, such as: 1. Monitoring successful/failed connections activities, by adding 1 to a counter for each. I need a lock for that. My counter is in global scope of the same file. How can I do that? 2. I wish to handle some sort of queue and write it to a file, where the content of the queue is a set of strings, written from my different threads, that simply sends some information for logging issues. How can it be done? I fail to accomplish that since my threading is done "behind the scenes", as each time Im in do_POST(..) method, Im already in a different thread.

Succcessful_Logins = 0
Failed_Logins = 0
LogsFile = open(logfile)

class httpHandler(BaseHTTPRequestHandler):

    def do_POST(self):
       ..

class ThreadingHTTPServer(ThreadingMixIn, HTTPServer):
    pass

server = ThreadingHTTPServer(('localhost', PORT_NUMBER), httpHandler)
server.serve_forever()

this is a small fragile of my server. Another thing that bothers my is the face I want to first send the post response back to the client, and only then possibly get delayed due to locking mechanism or whatever.

score 0 · Accepted Answer · answered Aug 16 '13 at 19:00

0

From your code, it looks like a new httpHandler is constructed in each thread? If that's the case you can use a class variable for the count and a mutex to protect the count like:

class httpHandler(...):
    # Note that these are class variables and are therefore accessable
    # to all instances
    numSuccess = 0
    numSuccessLock = new threading.Lock()

    def do_POST(self):
        self.numSuccessLock.aquire()
        self.numSuccess += 1
        self.numSuccessLock.release()

As for writing to a file from different threads, there are a few options:

Use the logging module, "The logging module is intended to be thread-safe without any special work needing to be done by its clients." from http://docs.python.org/2/library/logging.html#thread-safety
Use a Lock object like above to serialize writes to the file
Use a thread safe queue to queue up writes and then read from the queue and write to the file from a separate thread. See http://docs.python.org/2/library/queue.html#module-Queue for examples.

answered Aug 16 '13 at 19:00

Oliver Dain

9,617
3
35
48

But if the numSuccess is a class variable, it is instantiated again and again, for each new connection(=thread), am I wrong? About mutex lock, Can I handle it after I return from my do_POST, to not delay the response? – buddy123 Aug 16 '13 at 19:56
*Instances* variables are per instance, while *class* variables are per class. Thus numSuccess is instantiated only once for the entire program. [this blog post](http://timothyawiseman.wordpress.com/2012/10/06/class-and-instance-variables-in-python-2-7/) has some more details. I don't know if you can delay it until after do_POST because I don't know what the rest of your code looks like. However, the time to do the locking and incrementing should be minimal so I don't think you'll notice the delay. – Oliver Dain Aug 16 '13 at 23:23
Ok. Thank you Oliver. I will read about the link you shared with logging module. anyway, there's a slight fix you need to do. In the answer you supplied me, the variables should be accessed by using class name `httpHandler`, not `self` – buddy123 Aug 17 '13 at 05:48
@e-r-a-n You can reference class variables through `self`. See this excellent explanation: http://stackoverflow.com/a/8701644/1431244 – Oliver Dain Aug 19 '13 at 16:57

Maintaining log file from multiple threads in Python

1 Answers1