1

I have run a command line program that can take any text input that ends with "Enter" key from stdin and give a text response promptly to stdout. Now I have a file that contains thousands of sentences and one sentence per line. Can I use two threads, one for reading this file line by line and send it to stdin when I run the command line program, and the other thread for catching the response and write to another file?

For "sending to stdin" thread:

def readSentence(inpipe, senlines):
    for sen in senlines:
        inpipe.write(sen.strip()+'\n')
    inpipe.close()

For "getting from stdout" thread:

def queueResult(outpipe, queue):
    for res in iter(outpipe.readlines()):
        queue.put(res)
    outpipe.close()

The main thread in which the command line program is called:

def testSubprocess():
    ee = open('sentences.txt', 'r')
    ff = open('result.txt', 'w')
    lines = ee.readlines()
    cmd = ['java', 
           '-cp', 'someUsefulTools.jar', 
           'fooClassThatReadSentenceAndOutputResponse', 
           '-stdin',]   # take input from stdin
    proc = Popen(cmd, stdout=PIPE, stdin=PIPE)
    q = Queue()
    readThread = Thread(target=readSentence, args=(proc.stdin, lines))
    queueThread = Thread(target=queueResult, args=(proc.stdout, q))
    readThread.daemon = True
    queueThread.daemon = True
    readThread.start()
    queueThread.start()

    result = []

    try:
        while not q.empty(): 
            result = result.append(q.get_nowait())
    except Empty:
        print 'No results!'

I print input and output in the for loop in readSentence() and queueResult()(not shown in above code). I found at the end, the input sentences are not completely read, and the output is nothing. What could go wrong in my code? How can achieve synchronization between "stdin"-thread and "stdout"-thread so they can work in pair? i.e. "stdin"-thread put one sentence to the pipe, and then "stdout"-thread get the result from the pipe.

p.s. I reference this article for non-blocking read: Non-blocking read on a subprocess.PIPE in python

Community
  • 1
  • 1
Frank Yang
  • 49
  • 1
  • 3
  • 10
  • why do you need threads here at all? It looks like [XY problem](http://meta.stackexchange.com/a/66378/137096). You could use the files directly: `subprocess.check_call(cmd, stdin=ee, stdout=ff)` – jfs Apr 01 '15 at 22:25
  • @J.F.Sebastian The post might look irrelevant to my real problem. Thanks for the correct. What I really want is to do some pre-processing to the input file, for example, read every third sentence and check if some character appears and if yes then send to stdin. I'll try if I can do the pre-processing and then use `proc.stdin.write(input_sentence)` to send to stdin. What would you advise? – Frank Yang Apr 02 '15 at 03:23
  • It does not explain why do you need **non-blocking** read. To send a pre-processed input: `subprocess.Popen(cmd, stdin=subprocess.PIPE, stdout=ff, universal_newlines=True).communicate("\n".join(sentences))` (you can send the input incrementally without threads too). Don't ask new questions in a comment, update your question or ask a new one. If you need pre-,post- etc process the input/output; mention it. – jfs Apr 03 '15 at 06:34

0 Answers0