How to log messages from different threads to different files?

Question

I have a Driver.py scripts where it calls multiple threads based on the given inputs. Threads are basically runs a module of a selected object. So Driver.py may call thread_1.run(), thread_2.run(), thread_3.run(), and continue its process.

Driver.py logs its output into a main.log folder, where I want threads to log their output into unique filenames for each. Driver.py and Threads also uses common modules that are defined on differnt files, where they also logs information.

I call setup_load("main.log") first on Driver, afterwards in each Thread, called setup_load(f"thread_{jobid}.log") as well. I realize that when Thread is called, now Driver.py writes into thread's log file. I may use a differnt logger inside Thread but when the Thread calls another modules, since those common modules are using import logging they write into the root logger's defined filename.

---

=> Is it possible to log messages from different threads to different files? I found multiple answers on SO (for example), but non of them covers when another module is called on a different file, how would they can find out that which logger they can use.

=> So the problem I am facing is since every thread is using the same underlying logger, when I change the file path of the logging.basicConfig in one thread, it affects the class across all threads and the driver, since they're all using it.

=> How would be functions from different modules called from the thread or driver would understand which logger should it choose?

---

Comment section on How to change filehandle with Python logging on the fly with different classes and imports has a discussion and recommended solution.

@Martijn Pieters:

next option: create per-thread handlers, give each of them a filter that filters on the logrecord thread attribute. Attach a filter to any other handlers that returns False for logrecords with thread set

Answer 1

Yes, you can direct log entries from different threads to different files. You'll need to:

• Create a log filter that can filter records by their LogRecord.thread or LogRecord.threadName attribute
• Create a filter that does not accept records with specific or all thread ids.
• Create a log handler per thread, giving it a log filter that only accepts logrecords for their specific thread.
• Attach the filter that ignores log records for your threads to any other handlers.

When filtering, you have the choice between filtering on thread id (the value returned by threading.get_ident()) or thread name (whatever you passed in as the name argument to the Thread() object). If you have a pattern for your thread names, this is where you'd use it.

Creating a custom filter is easy enough:

import threading
from logging import Filter

class ThreadFilter(Filter):
    """Only accept log records from a specific thread or thread name"""

    def __init__(self, threadid=None, threadname=None):
        if threadid is None and threadname is None:
            raise ValueError("Must set at a threadid and/or threadname to filter on")
        self._threadid = threadid
        self._threadname = threadname

    def filter(self, record):
        if self._threadid is not None and record.thread != self._threadid:
            return False
        if self._threadname is not None and record.threadName != self._threadname:
            return False
        return True

class IgnoreThreadsFilter(Filter):
    """Only accepts log records that originated from the main thread"""

    def __init__(self):
        self._main_thread_id = threading.main_thread().ident

    def filter(self, record):
        return record.thread == self._main_thread_id

If you want to match a specific pattern, adjust the code accordingly.