Event listening and handling in Python

Question

I will start out by saying I have never done events and triggers in anything other than javascript.

I have a thread that runs and processes all the files in a folder. I want this thread to run everytime a new file writes to that folder. I thought events would be the way to do that, but I have no idea how to do that in python.

How do you tell a thread to throw or trigger an event? How does another thread pick this up?

sample thread:

def list_files_thread(dir):
    for filename in os.listdir(dir):
        print filename

thread.start_new_thread(list_files_thread, ('output',))

Edit: I am new to mutlithreading, so maybe I should use this instead?

class list_files_thread(threading.Thread):
    def __init__(self, directory):
        self.directory = directory
    def run(self):
        list_files(self.directory)

def list_files(directory_path):
    for filename in os.listdir(directory_path):
        print filename

not the answer to your question, but you should really use `threading` and not `thread`. The former is the high-level API, the latter is rather low-level. — roippi, Mar 27 '14 at 14:58
I think you might be better suited with multiprocessing than threading. Take a look at multiprocessing.Queue — FlorianLudwig, Mar 27 '14 at 15:00
What you want is that your code be able to detect when a new file is created in that folder. Is that correct? — Raydel Miranda, Mar 27 '14 at 15:03
@RaydelMiranda I am trying to make a platform independent program, but I am currently on Windows. Yes, the idea is that this thread would list the files in the dir (or whatever action), maybe move them to a "processed" folder, and anytime a new file is written, it would rerun the thread. — Jeff, Mar 27 '14 at 15:05
@FlorianLudwig that's not really sound advice. Threading is ideal for i/o operations. `multiprocessing` introduces a lot of complications like interprocess transfer of non-picklable information that you just don't have to deal with when using threads. — roippi, Mar 27 '14 at 15:05

score 1 · Answer 1 · edited May 23 '17 at 12:05

1

First of all, this is not a problem of python events it's more about Operative System Signals. You will need some way of monitoring the changes in the target folder. Since you are planning your code to be multiplatform I recommend you work with Qt specifically QtCore.QFileSystemWatcher.

Here are you have an aswer that can help you: How do I watch a file for changes?

edited May 23 '17 at 12:05

Community

1
1

answered Mar 27 '14 at 15:11

Raydel Miranda

13,825
3
38
60

Thanks! If I have access to the script that would write files to that folder, would this be a different situation? – Jeff Mar 27 '14 at 15:12
Well, of course. Then you only need to implement and `obserber pattern` and combine it with a `Provider consumer pattern` I'll update the answer to show what I mean. – Raydel Miranda Mar 27 '14 at 15:16

score 1 · Accepted Answer · answered Mar 27 '14 at 15:12

1

Well, the typical way to trigger asynchronous events is to use the signal module. As the documentation notes, signals cannot be used for inter-thread communication, so threads would be inappropriate in this case. Of course, you can also use threads using threading.Condition objects.

That being said, asynchronous events, threads, and signals are confusing and hard to implement correctly. Are you sure you want to reinvent the wheel? (You very well might be; I just like to check.) If not, watchdog provides the exact capabilities you require.

answered Mar 27 '14 at 15:12

ubomb

9,438
2
20
26

Couldn't I just create a watchdog thread that monitors. In that thread, if it detects a change, it spawns a "list_dir" thread? – Jeff Mar 27 '14 at 15:16
2

Sure - there's many, many ways to accomplish the same thing (although there is usually one preferred way in python). I'm not sure of your desired end goal, and I'm not sure what a list_dir thread would do. You asked about events specifically, so beyond that, you should really post a separate question. – ubomb Mar 27 '14 at 15:22
Does my answer implement what you were talking about? – Jeff Mar 27 '14 at 16:09
1

Well, the best way is to find out is to just go ahead and test it! (However, I also won't just leave you hanging - yes, it seems to do what you originally set out to accomplish from the brief description you provided.) – ubomb Mar 27 '14 at 16:28

score 0 · Answer 3 · answered Mar 27 '14 at 16:08

Thanks for the great answers.

This is the code I ended up using:

def list_files(directory_path):
    print directory_path
    parent_dir = os.path.abspath(os.path.join(directory_path, os.pardir))
    print parent_dir
    for filename in os.listdir(parent_dir):
         print filename

class trigger_event_handler(FileSystemEventHandler):
    def on_created(self, event):
        super(trigger_event_handler, self).on_created(event)
        list_files(event.src_path)


new_event_handler = trigger_event_handler()
observer = Observer()
observer.schedule(new_event_handler, output_dir, recursive=False)
observer.start()
try:
    while True:
        time.sleep(1)
except KeyboardInterrupt:
    observer.stop()
observer.join()

Event listening and handling in Python

3 Answers3