1

I want to ask the shell to display a csv file in a nice format in my python script. So I wrote the following:

printout = "column -s, -t < output.csv | less -#2 -N -S "
subprocess.call(printout.split(), shell = False)

The error I get is:

column: invalid option -- '#'

I have a rough idea that it is something to do with shell=False; however when I set it to True and run in cmd line, it puts me into another line and I have to ctrl+C to get out.

tipanverella
  • 3,477
  • 3
  • 25
  • 41
abwxl
  • 13
  • 4
  • `shell=False` means you don't have a shell, so there's nothing that knows how to process redirections (`<`) or pipes (`|`). It's actually *better* to use `shell=False`, but it means you need to do more of your setup in Python. – Charles Duffy Dec 07 '17 at 18:00

1 Answers1

3

Your original code was equivalent to the shell command:

# this can be used to reproduce your bug at a shell
column -s, -t '<' output.csv '|' less -#2 -N -S

...passing <, |, less, etc. as arguments to column, not treating them as shell directives.

See the section Replacing Shell Pipelines in the subprocess module documentation.

p1 = subprocess.Popen(['column', '-s,', '-t'],
                      stdin=open('output.csv', 'r'),
                      stdout=subprocess.PIPE)
p2 = subprocess.Popen(['less', '-#2', '-N', '-S'],
                      stdin=p1.stdout)
p1.stdout.close() # ensure that p2 has the only remaining handle on p1.stdout
p2.communicate()  # let less run
Charles Duffy
  • 280,126
  • 43
  • 390
  • 441
  • Why we need separate list entry like 'column','-s,'...can we use it instead like 'column -s -t'? Or what is the advantage of former approach – pankaj mishra Dec 07 '17 at 18:06
  • 1
    @pankajmishra, the advantage of this approach is that you're not relying on a shell to do the splitting for you. This is very much better practice, because something like `split()` or even `shlex.split()` can be intentionally misled or broken by unexpected or malicious filenames -- think about what happens if your `output.csv` is from a variable and the filename has a space in it. – Charles Duffy Dec 07 '17 at 18:09
  • OMG! Thank you SOOOOOO MUUUCH – abwxl Dec 07 '17 at 18:09
  • @pankajmishra, ...some folks will somewhat naively use quoting like `"'%s'" % filename` with `shlex.split()` or `shell=True`, but even that doesn't work if the filename contains literal `'`s that can then escape the syntactical ones. – Charles Duffy Dec 07 '17 at 18:10
  • @pankajmishra, ...and the other point is that a list of strings is actually the native format for an argv list at the operating system level -- the way other programs are started on UNIX is with the `execve` syscall, and it takes a list of C strings. Anything you do with a *single* string instead of a list is going to need to be broken down / transformed into that list before it can actually be used; passing a list yourself means you have absolutely control, instead of relying on whatever does that transformation to do it the way you intend. – Charles Duffy Dec 07 '17 at 18:12
  • Thanks Charles,nice explanation..it cleared my doubt. – pankaj mishra Dec 07 '17 at 18:16