1

I am using Python 2.6.6 and failed to re-direct the Beeline(Hive) SQL query output returning multiple rows to a file on Unix using ">". For simplicity's sake, I replaced the SQL query with simple "ls" command on current directory and outputting to a text file.

Please ignore syntax of function sendfile. I want help to tweak the function "callcmd" to pipe the stdout onto the text file.

def callcmd(cmd, shl):
    logging.info('> '+' '.join(map(str,cmd)))
    #return 0;
    start_time = time.time()
    command_process = subprocess.Popen(cmd, shell=shl, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, universal_newlines=True)
    command_output = command_process.communicate()[0]
    logging.info(command_output)
    elapsed_time = time.time() - start_time
    logging.info(time.strftime("%H:%M:%S",time.gmtime(elapsed_time))+' = time to complete (hh:mm:ss)')
    if (command_process.returncode != 0):
        logging.error('ERROR ON COMMAND: '+' '.join(map(str,cmd)))
        logging.error('ERROR CODE: '+str(ret_code))
    return command_process.returncode

cmd=['ls', ' >', '/home/input/xyz.txt']
ret_code = callcmd(cmd, False)
nnnmmm
  • 7,964
  • 4
  • 22
  • 41
cheapcoder
  • 183
  • 1
  • 3
  • 12
  • please fix your code, it is not syntactically correct. – MAQ Feb 07 '18 at 00:17
  • popen refers to opening the process for reading so you do not want to redirect with 'ls' '>' '/home/input' but rather just do a listing 'ls' '/home/input' – Snohdo Feb 07 '18 at 00:48
  • My intention is not to do a 'ls' on /home/input but to list all the contents of my current directory and pipe it to a text file .let me know if this makes sense – cheapcoder Feb 07 '18 at 01:52
  • A redirection is not a pipe. There are many ways to do this, and the most appropriate is to open the file in python and write the data to it. What you are trying to do could be done by passing the arguments `sh`, `-c`, `ls > /home/input/xyz.txt`, but there are many better ways to do it. – William Pursell Feb 07 '18 at 03:35
  • where exactly should I add the arguments "sh -c" ? and what does this do ? – cheapcoder Feb 07 '18 at 04:18
  • Possible duplicate of https://stackoverflow.com/questions/3172470/actual-meaning-of-shell-true-in-subprocess – tripleee Feb 07 '18 at 07:15

3 Answers3

1

Your command (i.e. cmd) could be ['sh', '-c', 'ls > ~/xyz.txt']. That would mean that the output of ls is never passed to Python, it happens entirely in the spawned shell – so you can't log the output. In that case, I'd have used return_code = subprocess.call(cmd), no need for Popen and communicate.

Equivalently, assuming you use bash or similar, you can simply use 

subprocess.call('ls > ~/test.txt', shell=True)

If you want to access the output, e.g. for logging, you could use

s = subprocess.check_output(['ls'])

and then write that to a file like you would regularly in Python. To check for a non-zero exit code, handle the CalledProcessError that is raised in such cases.

nnnmmm
  • 7,964
  • 4
  • 22
  • 41
0

Here the stdout in command_output is written to a file. You don't need to use any redirection although an alternative might be to have the python print to stdout, and then you would redirect that in your shell to a file.

#!/usr/bin/python

import subprocess

cmd=['ls']  

command_process = subprocess.Popen(
cmd,
shell='/bin/bash',
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
universal_newlines=True
)

command_output = command_process.communicate()[0]

if (command_process.returncode != 0):
  logging.error('ERROR ON COMMAND: '+' '.join(map(str,cmd)))
  logging.error('ERROR CODE: '+str(ret_code))

f = open('listing.txt','w')
f.write(command_output)
f.close()
Snohdo
  • 156
  • 5
0

I added this piece of code to my code and It works fine.Thanks to @Snohdo

f = open('listing.txt','w')
f.write(command_output)
f.close()
cheapcoder
  • 183
  • 1
  • 3
  • 12