6

I am creating a program that will pull in a list of account numbers and then run an ls -lh command to find a file for each one. When I run my command on our Linux server without Python it pulls up the files no problem, but then when I do it through Python it says it can't find them. 

import subprocess as sp
sp.call(['cd', input_dir])
for i, e in enumerate(piv_id_list):
    proc_out = sp.Popen(['ls', '-lh', '*CSV*APP*{0}.zip'.format(e)])
    proc_out_list.append(proc_out)
    print(proc_out)

Here is some sample output when I run the commands through the Python interpreter:

>>> ls: cannot access *CSV1000*APP*: No such file or directory

But through Linux the same command:

ls -lh *CSV*APP*

It returns the output like it should.

das-g
  • 9,718
  • 4
  • 38
  • 80
flybonzai
  • 3,763
  • 11
  • 38
  • 72

4 Answers4

5

This is because the shell does the replacement of wildcards with existing files that match the pattern. For example, if you have a.txt and b.txt, then ls *.txt will be expanded from the shell to ls a.txt b.txt. With your command you actually ask for ls to return info about a file containing an asterisk in its filename. Use the following if you want to verify:

sp.Popen(['bash', '-c', 'ls', '-lh', '*CSV*APP*{0}.zip'.format(e)])

Also you should use os.chdir to change the directory, since sp.call(['cd', input_dir]) changes the current directory for the new process you created and not the parent one.

JuniorCompressor
  • 19,631
  • 4
  • 30
  • 57
  • Should I just the os module for everything? Seems much simpler as I can just type in the commands as a string like I would do in Linux. – flybonzai Jul 31 '15 at 21:03
  • 2
    Yes of course. That's what I would do. You won't have any external dependencies like `ls`, it would work on other os, and it's not that difficult... – JuniorCompressor Jul 31 '15 at 21:13
3

ls, as run through Python, is probably right: I guess there is no file called *CSV*APP* in the current directory. There is probably a file with a name that matches that glob pattern. But ls doesn't care about globs. What happens when you run the command on the shell, is that the shell expands the glob to matching filenames it can see in the current directory and these expanded names is what the shell passes to ls.

To get the same result in the shell as in Python (for demonstration, not because you'd want that), protect the arguments from glob expansion by single quotes:

ls -lh '*CVS*APP*'${e}'.zip'

But how can you get the shell's behavior in Python? You can use shell=True as some other answers suggest, but that is a slippery slope, as invoking an actual shell on your dynamically generated strings (maybe dependent on user input in a more complex application) can make you vulnerable to command injections and other nastiness.

Here, you only need one specific behavior of the shell, filename globbing. And Python happens to be able to do that all by itself:

import subprocess as sp
from glob import glob
sp.call(['cd', input_dir])
for i, e in enumerate(piv_id_list):
    proc_out = sp.Popen(['ls', '-lh', glob('*CSV*APP*{0}.zip'.format(e))])
    proc_out_list.append(proc_out)
    print(proc_out)

As JuniorCompressor has pointed out, this would still look in the wrong directory, because the cd would only affect the subprocess of the cd invocation, so let's fix that, too:

import subprocess as sp
from glob import glob

os.chdir(input_dir)
for i, e in enumerate(piv_id_list):
    proc_out = sp.Popen(['ls', '-lh', glob('*CSV*APP*{0}.zip'.format(e))])
    proc_out_list.append(proc_out)
    print(proc_out)
Note

You could probably use the slightly higher-level sp.check_output instead of the underlying sp.Popen directly.

Community
  • 1
  • 1
das-g
  • 9,718
  • 4
  • 38
  • 80
  • I would if I could, unfortunately we're running vs. 2.6 here at work:/ – flybonzai Jul 31 '15 at 21:41
  • 1
    @flybonzai: 1. passing `cwd` parameter instead of `os.chdir()` is less global (better). 2. It is easy to [emulate `check_output()` on Python 2.6](http://stackoverflow.com/a/2924457/4279). 3. You don't need to call `ls` at all: you could use `glob` module to get filename and `os.path.getsize()` to get the file size or `os.stat()` for any other info that you might need. – jfs Aug 02 '15 at 00:05
2

You should use cwd argument to Popen and shell=True, then communicate to get the output.

Your code would look like:

import subprocess as sp
for i, e in enumerate(piv_id_list):
    proc = sp.Popen(['ls', '-lh', '*CSV*APP*{0}.zip'.format(e)], cwd=input_dir, stdout=sp.PIPE, shell=True)
    proc_out_list.append(proc.communicate()[0])
    print(proc_out_list[-1])

But why are you making a subprocess instead of using the standard lib?

Edit

Like @tripleee said, it replace a few functions only. I think it's better to use builtins/stdlib when it's possible; in your case you "only" want to list files for a given pattern (glob) and show ordered (sorted) informations about their size (stat).

Using the stdlib make your code more portable; even if you don't care about Microsoft Windows portability, you might want to avoid running into surprises running your code on computer without GNU binutils (ie: Mac OS, BSD, ...).

You want to use the subprocess module for things that cannot (easily) be implemented in pure Python (ie.: encode videos with ffmpeg, change the user passowrd with passwd, escalating privileges with sudo, ...).

bufh
  • 3,153
  • 31
  • 36
  • 1
    I thought that subprocess was the replacement for the os module? – flybonzai Jul 31 '15 at 20:53
  • `subprocess` only supersedes a small portion of `os`, namely `os.popen()` and its immediate family (`popen2`, `popen3`, etc). See e.g. https://docs.python.org/2/library/os.html#os.popen – tripleee Aug 01 '15 at 09:07
1

I believe you need to add shell=True as a parameter to Popen and replacing the list with one string:

proc_out = sp.Popen('ls -lh *CSV*APP*{0}.zip'.format(e), shell=True)

See here for more information and possible usage of glob: Python subprocess wildcard usage

Community
  • 1
  • 1
awmo
  • 329
  • 1
  • 2