2

My subprocess command is not working with quotes.

tail = subprocess.Popen('tail -f -n 1 /mnt/syslog/**/*.log | egrep -v \'^$|mnt\'',\
            shell=True,stdout=subprocess.PIPE,stderr=subprocess.PIPE)

When I execute python file.py I get a empty line:

# python main.py 
^CTraceback (most recent call last):
  File "main.py", line 18, in <module>
    main()
  File "main.py", line 12, in main
    line = tail.stdout.readline()
KeyboardInterrupt

It works fine in bash as you can see:

# tail -f -n 1 /mnt/syslog/**/*.log | egrep -v '^$|mnt'
Sep  9 22:44:07 x.x.x.x : %LINK-3-UPDOWN: Interface GigabitEthernet1/0/19, changed state to down
Sep  9 18:32:56 x.x.x.x : %LINK-5-CHANGED: Interface GigabitEthernet1/0/24, changed state to administratively down

What is wrong?

superloopnetworks
  • 504
  • 6
  • 17
  • 1
    What problem are you trying to solve? Is it that your script is hanging with no output at all (that's caused by the `-f` option, which is tricky to use with subprocess)? Or is it that you are getting an empty string in the `line` variable? – FMc Sep 10 '17 at 02:03
  • The problem is that there is no output.. I suspect it's the quotes that is causing this.. If I were to remove the egret statement and input as tail -f -n 1 /mnt/syslog/**/*.log, it works just fine.. I see output when running it through Python.. – superloopnetworks Sep 10 '17 at 02:14
  • I don't think it's a quoting issue. I think it's the combination of `-f` with the pipe with subprocess. For example, drop the `-f` and it will also work for you (at least it does for me). "Work" in the sense of giving some output. – FMc Sep 10 '17 at 02:19
  • Hummm.. But it already works fine with the -f without the egrep? I get the expected output.. – superloopnetworks Sep 10 '17 at 02:22
  • 1
    One question is why do you think you need `egrep` at all. Let Python do that filtering. – FMc Sep 10 '17 at 02:24
  • I'm using the egrep so I don't have to do any output adjustments in Python which would be more lines of code. – superloopnetworks Sep 10 '17 at 02:27
  • Can you try on your side to see if you get the same issue? – superloopnetworks Sep 10 '17 at 02:27
  • As noted in my comments above, both of these worked for me: (a) remove `-f`, or (b) drop the pipe to `egrep`. The logic to do the filtering in Python is trivial (at least for the regex you are using in your example). Also see this for tailing a file directly in Python: https://stackoverflow.com/a/12523416/55857 – FMc Sep 10 '17 at 02:29
  • You're right.. I do see output, but I require the -f for what I'm trying to achieve.. I need to tail live logs.. So any updates to the log file I can see.. Which I can, if I remove the egrep portion and leave the -f.. If I make an update to the log file, Python output the updated log file as expected.. I'm clueless what to do.. :( – superloopnetworks Sep 10 '17 at 02:30
  • As I noted, the combination of `-f` (which never ends), the pipe to another bash command, and Python `subprocess` creates a complicated buffering situation that can be hard to sort out -- it might be solvable, but I've had trouble in the past with these issues. You're probably better off doing it all in Python. See the link above for various approaches. – FMc Sep 10 '17 at 02:32

2 Answers2

1

You need to run communicate() on Popen instance. It should look like

tail = subprocess.Popen('tail -f -n 1 /mnt/syslog/**/*.log | egrep -v \'^$|mnt\'',
                        shell=True,stdout=subprocess.PIPE,stderr=subprocess.PIPE)
stdout, stderr = tail.communicate()
print(stdout)

If you need unicode string instead of bytes use decode():

print(stdout.decode())

Update: Because of -f flag to tail you should get output live:

tail = subprocess.Popen('tail -f -n 1 /mnt/syslog/**/*.log | egrep -v \'^$|mnt\'',
                        shell=True,stdout=subprocess.PIPE,stderr=subprocess.PIPE)
for line in tail.stdout:
    print(line)

It's not best method, more you can find in this topic.

stachel
  • 86
  • 6
  • Any error message? I don't have your logs, but simple **ls** working perfectly for me with python 3.6. – stachel Sep 10 '17 at 02:18
  • No error messages.. Just no output.. Now if I drop the egrep portion, it works.. I suspect it's the quotes within the egrep that is not making it work.. – superloopnetworks Sep 10 '17 at 02:25
  • I've updated my answer and added link to another question about live output. – stachel Sep 10 '17 at 02:29
1

I don't think the problem is the quotes at all.

The command is doing tail -f, which by definition never terminates (it keeps tailing the file(s)). When you call it at the shell, you immediately see some output (maybe, depending on whether the grep matches). But it probably doesn't return to the prompt, because the tail is still running.

If you really want to be following the file, then you shouldn't be using communicate() as that expects a terminating process. You have to keep reading from stdout (and stderr, to be safe!) until the process dies.

But I suspect that all you need to do is to remove the -f on the tail:

tail = subprocess.Popen('tail -n 1 /mnt/syslog/**/*.log | egrep -v \'^$|mnt\'',\
        shell=True,stdout=subprocess.PIPE,stderr=subprocess.PIPE)

If you remove the -f, then communicate() is exactly the right call to use.

Or, you can just use the check_output helper:

subprocess.check_output('tail...', shell=True)
Sean Suchter
  • 328
  • 1
  • 6