0

I'm trying to get the output of the following shell command in my python script,

hadoop fs -ls /projectpath/ | grep ^d | grep -v done | head -1 | awk {'print $8'}

I can successfully get the output through os.popen as follows:

import os
cmd = "hadoop fs -ls /projectpath/ | grep ^d | grep -v done | head -1 | awk {'print $8'}"
p = os.popen(cmd,"r")
while 1:
    line = p.readline()
    if not line: break
    print line

But os.popen() is deprecated since python 2.6 so I wanted to replace the above snippet with the subprocess.Popen() function.

But the code snippet for subprocess.Popen() below gives a different result than the code snippet above.

import subprocess as sub
import shlex
cmd = "hadoop fs -ls /projectpath/ | grep ^d | grep -v done | head -1 | awk {'print $8'}"
args = shlex.split(cmd)
p = sub.Popen(args,stdout=sub.PIPE,stderr=sub.PIPE)
output, errors = p.communicate()
print output

The above command just gives output for 'hadoop fs -ls /projectpath/' part of the command. I have tried consulting several references (http://docs.python.org/2/library/subprocess.html#popen-objects, Python, os.system for command-line call (linux) not returning what it should?) for subpocess.Popen() but cannot get it to execute the command in the string cmd. Can anyone point out what I'm doing wrong?

luzik
  • 673
  • 2
  • 7
  • 15
rohanag
  • 913
  • 3
  • 11
  • 15

1 Answers1

2

try this:

cmd = "hadoop fs -ls /projectpath/ | grep ^d | grep -v done | head -1 | awk {'print $8'}"
p = sub.Popen(cmd,stdout=sub.PIPE,stderr=sub.PIPE, shell=True)
David.Zheng
  • 903
  • 8
  • 6