Python two concurrent threads not printing simultaneously

Question

I'm trying to write a small server wrapper in Python in which I can start multiple servers and make them print in the same terminal. Each process would be started in a thread, and print stdout to the terminal. Below is my code:

class Server():
    def __init__(self,name,args):
        self.name = name
        self.args = shlex.split(args)
        self.started = False
    def start(self):
        threading.Thread(target=self.__start).start()
    def __start(self):
        print(bcolors.HEADER+"{}: INFO: Server started.".format(self.name)+bcolors.ENDC)
        self.started = True
        self.p = subprocess.Popen(self.args, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE) 
        stdout_list = []
        while True:
            stdout = self.p.stdout.readline()
            if  stdout == '' and self.p.poll() != None:
                break
            else:
                if not stdout == '':
                    stdout_list.append(stdout)
                    stdout = stdout.rstrip()
                    print("{}: STDOUT: {}".format(self.name,stdout))
        stderr = self.p.stdout.readline()
        if stderr != '':
            print(bcolors.WARNING+"{}: STDERR: {}".format(self.name,stderr)+bcolors.ENDC)
        print(bcolors.HEADER+"{}: INFO: Server closed.".format(self.name)+bcolors.ENDC)
        self.start = False
        return ''.join(stdout)
    def interrupt(self):
        if self.started == True:
            self.p.send_signal(signal.SIGINT)
        else:
            print("Server not started.")
    def terminate(self):
        if self.started == True:
            self.p.kill()
        else:
            print("Server not started.")

t=Server("Test","python testserver.py")
t2=Server("Test2","python testserver.py")

t.start()
t2.start()

testserver.py is a simple script that prints 1-3, one each second, and each time flushing the stdout. It's just here to test if the program can host and print stuff concurrently.

The main script works fine if only t.start() is executed, it would print 1-3 one by one, and it also runs from a thread, so any code after it would work as well. Yet when I add t2.start(), I get mixed results.

user@desktop:~/Documents/home automation$ python serverhost.py
Test: INFO: Server started.

Test2: INFO: Server started.
Test: STDOUT: 1
Test2: STDOUT: 1
Test: STDOUT: 2
Test2: STDOUT: 2
Test: STDOUT: 3
Test2: STDOUT: 3
Test: INFO: Server closed.
Test2: INFO: Server closed.

and sometimes

user@desktop:~/Documents/home automation$ python serverhost.py
Test: INFO: Server started.
Test2: INFO: Server started.
Test: STDOUT: 1
Test: STDOUT: 2
Test: STDOUT: 3
Test: INFO: Server closed.
Test2: STDOUT: 1
Test2: STDOUT: 2
Test2: STDOUT: 3
Test2: INFO: Server closed.

with the STDOUT from Test2 all showing at once (not realtime) when the host quits.

Anyone knows what's going on? I thought if it doesn't work, it should at least be consistent..

They aren't supposed to be printing simultaneously. The threads are asynchronous. — Debanik Dawn, Sep 09 '17 at 07:05
Could you please simplify __start() function like def __start(): for i in range(3): print(i) sleep(1) ? — RedEyed, Sep 09 '17 at 07:13
@Trevor, read up on how threads are actually executed. And specifically, read up on Python's threading model. The processor actually switches between the threads very rapidly to get the job done. So, no guarantee when each part of a task will be executed w.r.t the other threads — Debanik Dawn, Sep 09 '17 at 07:19
@VadymStupakov , that would be different because I think it is part of calling the subprocesses that are causing the problems. I just tested that if I replaced it with the code then it would work as planned. — Trevor, Sep 09 '17 at 07:23
@Trevor, I suppose that the problem is not in the multithreading, the problem is in the Client side. Could you share the Client code? — RedEyed, Sep 09 '17 at 07:28
@VadymStupakov Here: https://pastebin.com/nwRmnezs, basically what you wrote, but the difference is one would be directly running it, and the other is running it as a process and communicating using stdout. — Trevor, Sep 09 '17 at 07:37
@Trevor, you need interprocess communication (IPC). Why not to use sockets for example? — RedEyed, Sep 09 '17 at 08:18
@VadymStupakov I'm using Pyro actually, it handles IPC. But I might use this for non python programs. Eitherway, now I'm going to use systemd instead of this. — Trevor, Sep 09 '17 at 08:22
@Trevor, I'm sorry, but what does "Pyro"? "Eitherway, now I'm going to use systemd instead of this" Does it mean that question is outdated? — RedEyed, Sep 09 '17 at 08:42
For the time being, I'm going to use systemd as an alternative, but I still have no idea why the above is not working. Pyro (https://pythonhosted.org/Pyro4/) is a magical python library that does low-level processing for you. — Trevor, Sep 09 '17 at 08:45
@Trevor,I run your code with some minor changes and always get 1 1 2 2 3 3 sequences. I do that on linux. Check it out please. https://pastebin.com/hsnJyg5r https://pastebin.com/CXAW2FEh — RedEyed, Sep 09 '17 at 09:39

score 0 · Answer 1 · answered Sep 09 '17 at 07:12

0

You may want to synchronize your threads using semaphors. But I'd suggest you get some basic knowledge about Concurrency.

answered Sep 09 '17 at 07:12

Monkey Supersonic

1,165
1
10
19

score 0 · Answer 2 · answered Sep 09 '17 at 07:19

Threading is Concurrent for parellel programing you need to use Multiprocessing

Concurrency is when two or more tasks can start, run, and complete in overlapping time periods. It doesn't necessarily mean they'll ever both be running at the same instant. Eg. multitasking on a single-core machine.

Parallelism is when tasks literally run at the same time, eg. on a multicore processor.

Python on the CPython interpreter does not support true multi-core execution via multithreading

for detailed: check this blog

check this answer

Python two concurrent threads not printing simultaneously

2 Answers2