0

I have a weird problem to read from STDIN in a python script.

Here is my use case. I have rsyslog configured with an output module so rsyslog can pipe log messages to my Python script.

My Python script is really trivial :

#! /usr/bin/env python
# -*- coding: utf-8 -*-
import sys

fd = open('/tmp/testrsyslogomoutput.txt', 'a')
fd.write("Receiving log message : \n%s\n" % ('-'.join(sys.stdin.readlines())))
fd.close()

If I run echo "foo" | mypythonscript.py I can get "foo" in the target file /tmp/testrsyslogomoutput.txt. However when I run it within rsyslog, messages seems to be sent only when I stop/restart rsyslog (I believe some buffer is flushed at some point).

I first thought it was a problem with Rsyslog. So I replaced my python program with a shell one, without changing anything to the rsyslog configuration. The shell script works perfectly with rsyslog and as you can see in the code below, the script is really trivial:

#! /bin/sh
cat /dev/stdin >> /tmp/testrsyslogomoutput.txt

Since my shell script works but my Python one does not, I believe I made a mistake somewhere in my Python code but I can not find where. If you could point me to my mistake(s) that would be great.

Thanks in advance :)

Jérôme R
  • 1,227
  • 2
  • 13
  • 23

3 Answers3

3

readlines will not return until it has finished reading the file. Since the pipe feeding stdin never finishes, readlines never finishes either. Stopping rsyslog closes the pipe and lets it finish.

Mark Ransom
  • 299,747
  • 42
  • 398
  • 622
0

If you use readline() instead, it will return on \n, though this will only write one line then quit.

If you want to keep writing lines as long they are there, you can use a simple for:

for line in fd:
  fd.write("Receiving log message : \n%s\n" % (line)
fd.close()
Karl Barker
  • 11,095
  • 3
  • 21
  • 26
  • I tried this. The problem remains the same. I tried to loop over sys.stdin and /dev/stdin and the input is only written when \n is sent. Using an alternative to readline as suggested above did not work either. – Jérôme R Feb 25 '12 at 04:39
0

I'd also suspect the reason is that rsyslog does not terminate. readlines() should not return until it reaches a real EOF. But why would the shell script act differently? Perhaps the use of /dev/stdin is the reason. Try this version and see if it still runs without hanging:

#!/bin/sh
cat >> /tmp/testrsyslogomoutput.txt

If this makes a difference, you'll also have a fix: open and read /dev/stdin from python, instead of sys.stdin.

Edit: So cat somehow reads whatever is waiting at stdin and returns, but python blocks and waits until stdin is exhausted. Strange. You can also try replacing readlines() with a single read() followed by split("\n"), but at this point I doubt that will help.

So, forget the diagnosis and let's try a work-around: Force stdin to do non-blocking i/o. The following is supposed to do that:

import fcntl, os

# Add O_NONBLOCK to the stdin descriptor flags 
flags = fcntl.fcntl(0, fcntl.F_GETFL)
fcntl.fcntl(0, fcntl.F_SETFL, fl | os.O_NONBLOCK)

message = sys.stdin.read().split("\n")  # Read what's waiting, in one go
fd = open('/tmp/testrsyslogomoutput.txt', 'a')
fd.write("Receiving log message : \n%s\n" % ('-'.join(message)))
fd.close()

You probably want to use that in combination with python -u. Hope it works!

alexis
  • 48,685
  • 16
  • 101
  • 161
  • I tried what you proposed it does not make any difference. It still works perfectly with this trivial shell script. I also tried to read from /dev/stdin in my python script (without readlines()) and it still hangs. – Jérôme R Feb 25 '12 at 04:32
  • Oh well. Next idea: take python's i/o buffering out of the equation by invoking python with the -u flag. – alexis Feb 25 '12 at 12:08
  • Thanks for your support Alexis, I also tried with python -u and it did not change anything. – Jérôme R Feb 25 '12 at 14:39
  • It seems @abalus got kind of the same problem with Apache : http://stackoverflow.com/questions/7056306/python-wait-until-data-is-in-sys-stdin – Jérôme R Feb 25 '12 at 15:33
  • I think @abalus had the opposite problem: In his set-up the reads return immediately without blocking, but your program blocks when cat somehow returns without blocking. – alexis Feb 25 '12 at 16:58
  • I owe you a pint. I would have never thought about adding 0_NONBLOCK to sys.stdin. It works, thanks a lot ! – Jérôme R Feb 25 '12 at 17:53
  • :-) Glad it worked! I'm still wondering why cat didn't block, though... tk – alexis Feb 25 '12 at 18:08