python shell command pipes grep no results

Question

I am trying to get a domain name from a cpanel user in python.

Here is my code:

import subprocess
user = "testuser"
getuserdata = 'cat /var/cpanel/users/' + user
getdnsline = 'grep "DNS="'
test = 'wc -l'

userdataprocess = subprocess.Popen(getuserdata.split(), stdout=subprocess.PIPE)

testprocess = subprocess.Popen(test.split(), stdin=userdataprocess.stdout, stdout=subprocess.PIPE)

test, error = testprocess.communicate()
print(test)

dnslineprocess = subprocess.Popen(getdnsline.split(), stdin=userdataprocess.stdout, stdout=subprocess.PIPE)

website, error = dnslineprocess.communicate()
print(website.decode('utf-8').splitlines())

my output is:

b'60\n'
[]

So this means, that the wc -l command gives back 60 lines. So passing the output of the first getuserdata command to the wc -l command works.

However, the grep command always return blank. No matter it I put in "DNS=" or "=" or even "a". The file is the normal cpanel user file, and I have verified that DNS is in the file.

When I just output the data from the first process userdataprocess I can manually check for the DNS entry.

Do I have to do anything different when using the grep command in this fashion?

`userdataprocess.stdout` is empty when `dnslineprocess` runs, because `testprocess` consumed all of it already. I'm not aware of a simple workaround for that - but your use of `subprocess` is entirely unnecessary anyway: everything you're doing with it (reading a file, counting lines, checking for specific text in those lines) is trivially doable in Python directly. — jasonharper, Jul 01 '18 at 17:32

blhsing · Answer 1 · 2018-07-01T17:48:59.473

This is because your testprocess.communicate() for the wc -l command already consumes all of the output of userdataprocess.stdout and closes it in fact, so there's nothing left for dnslineprocess.communicate() to read.

You should instead read the output of userdataprocess.stdout into a variable and then use it as an input to both testprocess.communicate() and dnslineprocess.communicate().

Also, as @pyb pointed out, you are unnecessarily quoting DNS= in your grep command, which, without a shell, will be passed to grep with double quotes included as part of the string to filter with. You should simply remove them as there are no special characters in your filter string.

import subprocess
user = "testuser"
getuserdata = 'cat /var/cpanel/users/' + user
getdnsline = 'grep DNS='
test = 'wc -l'

userdataprocess = subprocess.Popen(getuserdata.split(), stdout=subprocess.PIPE)

userdata = userdataprocess.stdout.read()

testprocess = subprocess.Popen(test.split(), stdin=subprocess.PIPE, stdout=subprocess.PIPE)

test, error = testprocess.communicate(userdata)
print(test)

dnslineprocess = subprocess.Popen(getdnsline.split(), stdin=subprocess.PIPE, stdout=subprocess.PIPE)

website, error = dnslineprocess.communicate(userdata)
print(website.decode('utf-8').splitlines())

That doesn't work. I thought of it too and commented the parts that call `wc -l` and got the same results. — pyb, Jul 01 '18 at 17:44
It actually does work, but the secondary problem is as pyb pointed out, that the OP unnecessarily quoted `DNS=`. I've edited the answer accordingly. — blhsing, Jul 01 '18 at 17:50

score 1 · Accepted Answer · answered Jul 01 '18 at 17:43

Your script fails because of the quotes around DNS=.

You can use shell=True to make the script work:

dnslineprocess = subprocess.Popen(getdnsline, stdin=userdataprocess.stdout, stdout=subprocess.PIPE, shell=True)

Source: Passing double quote shell commands in python to subprocess.Popen()?

python shell command pipes grep no results

2 Answers2