How to use multiprocessing to create threads in python?

Question

I was trying to use Process to create threads in python.

But it keeps showing errors.

And here is my code:

from multiprocessing import Process
import threading

class OneProcess(Process):
    def __init__(self):
        super().__init__()
        self.workers = []
        for i in range(5):
            worker = OneThread()
            self.workers.append(worker)

    def run(self):
        for worker in self.workers:
            worker.start()
        for worker in self.workers:
            worker.join()


class OneThread(threading.Thread):
    def __init__(self):
        super().__init__()

    def run(self):
        print("do somethings")

class Shell():
    def __init__(self):
        self.first_process = OneProcess()

    def start(self):
        self.first_process.start()


if __name__ == "__main__":
    shell = Shell()
    shell.start()

And here is the error:

    ForkingPickler(file, protocol).dump(obj)
TypeError: can't pickle _thread.lock objects

I guess it just can't serilize the thread object? So is there any method can achieve this? Or using Process to create threads is unavaliable in python?

Create your thread objects inside the `run()` method instead of in the `__init__` — rdas, Jan 26 '21 at 10:21
@rdas You should write your comment up as an answer with an explanation (if you have one) -- it would be useful for others. — Booboo, Jan 26 '21 at 11:09

Booboo · Accepted Answer · 2021-01-26T12:33:26.060

I was hoping that @rdas would turn his comment into an actual answer. Since nothing has been forthcoming, I will answer as best I can:

Your code actually works under Linux. So I am guessing you are running under Windows (I don't know enough about other environments, such as the Mac). The threads are being created within the __init__ method of class OneProcess, which is executed under the same process as the main process. This is demonstrated in the modified code below. But when the statement self.first_process.start() is executed causing the OneProcess.run() to be called, we are now running is the subprocess and therefore the internal state of the OnceProcess object has to be transferred over to the new subprocess using pickle, and this is where the error occurs. By moving the creation of the threads to the run method, there is no thread-related state to be pickled.

from multiprocessing import Process
import threading
import os

class OneProcess(Process):
    def __init__(self):
        super().__init__()
        print('OneProcess __init__:', os.getpid())


    def run(self):
        print('OneProcess run:', os.getpid())
        self.workers = []
        for i in range(5):
            worker = OneThread()
            self.workers.append(worker)
        for worker in self.workers:
            worker.start()
        for worker in self.workers:
            worker.join()


class OneThread(threading.Thread):
    def __init__(self):
        super().__init__()

    def run(self):
        print("do somethings")

class Shell():
    def __init__(self):
        self.first_process = OneProcess()

    def start(self):
        self.first_process.start()


if __name__ == "__main__":
    print('main pid:', os.getpid())
    shell = Shell()
    shell.start()

Prints:

main pid: 25208
OneProcess __init__: 25208
OneProcess run: 18912
do somethings
do somethings
do somethings
do somethings
do somethings

Why does your original code works in Linux? This has to do with Linux supporting the OS fork() call, which copies the entire process when creating a new subprocess and therefore does not have to rely on using pickle to transfer the state. See What is being pickled when I call multiprocessing.Process?

How to use multiprocessing to create threads in python?

1 Answers1