subprocess.Popen not thread-safe?

Question

I wrote this little test class, based upon a Python issue - closed/fixed, and it seems to be occurring in Python 2.7.1 on Fedora 15.

from subprocess import Popen, PIPE
from threading import Thread

OUTPUT = "asl;dkfhqwiouethnjzxvnhsldfhuyasdhofuiweqamchuisemfawepfhuaipwemhfuaehfclkuehnflw ehfcoiuwehfiuwmhefuiwehfoiuhSfcl hfulh fuiqhuif huiwafh uiahf iUH fhh flkJH fl HASLFuhSAIULFhSUA HFulSAHfOI SUFChiuwqhncriouweycriuowanbyoUIWCryu iWyruawyrouiWYRcoiu YCRoiuNr uyr oUIAWryocIUWRNyc owuroiuNr cuWyrnawueitcnoy U IuiR yiuowaYnorc oWIUAr coiury iuoAW rnuoi asdfsdfd\n"


class X(Thread):
    def __init__(self):
        Thread.__init__(self)

    def run(self):
        print("Running")
        for i in xrange(10):
            s = Popen(
                "cat /tmp/junk",
                shell=True,
                stdout=PIPE,
                universal_newlines=True
            )
            output = s.communicate()[0]
            if not output == OUTPUT:
                print("Error: %r" % output)


XThreads = set([])

for i in xrange(1000):
    XThreads.add(X())

for x in XThreads:
    x.start()

Just create a file, /tmp/junk in this case, that has the contents of OUTPUT, less the last newline.

Running this, you would expect to see "Running" on every line. However, sometimes it shows " Running" or "RunningRunning\n\nRunning".

(Removed citation to actual problem, as this was a false-symptom, thanks to @phihag's answer).

The actual issue: https://stackoverflow.com/questions/9338409/python-subprocess-popen-corrupts-binary-streams

score 2 · Accepted Answer · edited May 23 '17 at 11:48

2

The behavior you're seeing has nothing to do with subprocess; I can reproduce it with:

import threading
def run(): print("Running")
for x in [threading.Thread(target=run) for i in range(1000)]:
    x.start()

This is because Python's print is not thread-safe. To avoid the race condition between printing out text and newline, directly write to stdout, like this:

import threading,sys
def run(): sys.stdout.write("Running\n")
for x in [threading.Thread(target=run) for i in range(1000)]:
    x.start()

This assumes the underlying write calls to stdout are thread-safe. That depends on the platform.

edited May 23 '17 at 11:48

Community

1
1

answered Feb 18 '12 at 01:42

phihag

278,196
72
453
469

thanks, I think I've found my issue, but require uploading multiple files to provide a working example - not sure how to do that here...can you help? – CrackerJack9 Feb 18 '12 at 03:35
more interestingly, the real issue - http://stackoverflow.com/questions/9338409/python-subprocess-popen-corrupts-binary-streams – CrackerJack9 Feb 18 '12 at 04:18
@CrackerJack9 With a few modifications, you should be able to concatenate any multi-file Python program into one. If that's not possible (or the program is really long), upload the files to [gist](https://gist.github.com/) or [pastebin](http://pastebin.com) – phihag Feb 18 '12 at 09:50

subprocess.Popen not thread-safe?

1 Answers1