python: trying to understand "subprocess" O/S calls

Question

Please help with a lot of frustration at below dialogs:

-) why does "subprocess.check_output(["ls","-rt","."])" produce no output, though at least is accepted?

-) why is "subprocess.check_output(["ls -rt","."]) " not accepted at all?

-) most of all: how can I, in python, get the name of the most recent file that matches some regex? My idea was to feed something like "ls -rt $REGEX | head -1" to python, but python seems to strongly dislike such an approach..?

karel@suske:~/home_shared/develop/airnav_db$ python --version
Python 2.7.6
karel@suske:~/home_shared/develop/airnav_db$ python -c 'import subprocess ; subprocess.check_output(["ls -rt","."])'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python2.7/subprocess.py", line 566, in check_output
    process = Popen(stdout=PIPE, *popenargs, **kwargs)
  File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1327, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory
karel@suske:~/home_shared/develop/airnav_db$ python -c 'import subprocess ; subprocess.check_output(["ls","-rt","."])'
karel@suske:~/home_shared/develop/airnav_db$ ls -l
total 52
drwxrwxrwx 2 karel users 4096 Oct 11  2009 auxdata
...

Why would you use an external process to list files? This is http://mywiki.wooledge.org/ParsingLs in spades. — tripleee, Dec 02 '15 at 18:16
@stark: I was trying to get it to execute "ls -rt" which ought to be one and only one thing to it, given the quotes; but apparently it tries to play the smart ass. Oh well. — Karel Adams, Dec 02 '15 at 18:30
@tripleee: I was trying the only way I could think of. See my last point: I will be glad to consider any other python coding to get the name of the most recent file whose name matches some regex. Thanks for thinking along. — Karel Adams, Dec 02 '15 at 18:32
@KarelAdams, but there is no file named `/usr/bin/ls -rt`, so why would you think you could execute it? You can't run `"ls -rt"`, with the quotes, in a shell either and get anything but "command not found". — Charles Duffy, Dec 02 '15 at 18:38
@Charles Duffy: you seem to have well understood my quest, but what you say I cannot do I can: karel@suske:~/home_shared/develop/airnav_db$ bash -c "ls -rt" auxdata out in tekst source bin out3 in2 in3_eff etc tmp3 in3_raw bin3 — Karel Adams, Dec 02 '15 at 18:41
In general, though, any "trying to understand X" question is too broad. If your question were posed as "how do I get the name of the most newest file matching a pattern?", it would be much better. — Charles Duffy, Dec 02 '15 at 18:41
@KarelAdams, but those are quotes passed to the **outer** shell, not the **inner** shell. If you were to pass quotes to the inner shell, that would be `bash -c '"ls -rt"'`. — Charles Duffy, Dec 02 '15 at 18:42
@KarelAdams, ...when you use `subprocess.Popen()` or equivalents without `shell=True`, there **is** no inner shell at all; Python itself uses the `execve()` syscall to directly invoke the program you're running (in this case, `ls`), without any shell between the two. That's desirable -- it means your Python code has more direct control of exactly what the operating system is doing, eliminating both performance impact and security issues such as Shellshock. — Charles Duffy, Dec 02 '15 at 18:42
@KarelAdams, ...also, if you didn't read the link tripleee gave about ParsingLs, you really should. The output of `ls` is not reliably usable in programmatic contexts. — Charles Duffy, Dec 02 '15 at 18:46
@Charles Duffy: again, thank you for your effort in thinking along, but I am afraid you are missing my point. In C, I can do a syscall("execute_whatever") and it will execute what I tell it to execute. I could even do so in certain dialects of Basic, 20 or 30 years ago. Neither did C or Basic ever require me to know or care about inner vs. outer shells. Now this Python thingy thinks to outsmart me - why should it even try to? And no, do not try to tell me what I should do or what I should read - I'll decide for myself. Like dropping this Python thingy. — Karel Adams, Dec 02 '15 at 18:55
@KarelAdams, you're thinking of `system()`, not `syscall()`. `system()` -- which is a standard C library function, not a syscall -- **invokes a shell**, just like Python's `subprocess.call()` does when you pass `shell=True`. Using `system()` in C is bad practice there too; the better practice is `execve()` -- if the network processes that got caught by ShellShock were using `execve()` rather than `system()`, only those explicitly invoking something involving a shell would have been vulnerable. — Charles Duffy, Dec 02 '15 at 19:12
As for the "inner shell vs outer shell" thing, that's completely unrelated to Python. When you gave me an example of `bash -c "ls -rt"`, that was shell code, not Python code -- but it was shell code **starting another shell**. Outer shell == the shell you ran that code in; inner shell == the shell the code you wrote started. Nothing to do with Python at all. When I told you you couldn't run `"ls -rt"` at a shell either, I meant `"ls -rt"`, not `bash -c "ls -rt"`; the explanation was in going into why and how those differ. — Charles Duffy, Dec 02 '15 at 19:23

tripleee · Accepted Answer · 2020-11-26T18:41:43.307

6

"ls -rt" is not the name of a valid command, so that's why your second case is failing. "ls" with the arguments "-rt" and "." is working fine, just not producing output where you expect it. But anyway, you should be looping over os.listdir('.') instead.

when = 0
for name in os.listdir('.'):
    if not 'foo' in name:
        continue
    # name matches *foo*
    st = os.stat(name)
    if st.mtime > when:
        when = st.mtime
        newest = name
print newest

This should easily extend to regular expressions if you really need them; but for most pedestrian tasks, they are really overkill.

For the record, subprocess.check_output returns the shell command's output as a string. But you really want to avoid using external processes for tasks which are easily done in Python. Perhaps as a trivial fix you wanted print(subprocess.check_output(["ls", "-rt", "."]))

edited Nov 26 '20 at 18:41

answered Dec 02 '15 at 18:22

tripleee

175,061
34
275
318

OK, will look into that. When doing so, what would be the way to 1) match the names found against a regex? 2) of those that match, if any, get the most recently accessed/modified? – Karel Adams Dec 02 '15 at 18:28
1

@KarelAdams, a regex, or a glob pattern? (`*foo*` is a glob pattern, not a regex; if that's what you want, use the `fnmatch` Python module). – Charles Duffy Dec 02 '15 at 18:39
1

See update, with mtime code. Maybe google a bit before posting follow-up questions. – tripleee Dec 02 '15 at 18:42
Thanks to all, I think I ought to find a way. Those who think I didn't search the web may think again. And above all, it seems Python is really not my kind of language, you'll have to admit the suggested solution is anything but elegant, plus being cumbersome - I can do in one line of bash what takes 7 lines of python... But enough of this frustration, I shall manage. – Karel Adams Dec 02 '15 at 18:47
1

@KarelAdams, if your "one line of bash" is using `ls -t`, it's buggy. See http://mywiki.wooledge.org/BashFAQ/003 -- robust and reliable bash is longer than sloppy, buggy bash, just as Python is longer than sloppy, buggy bash too. :) – Charles Duffy Dec 02 '15 at 18:48
@KarelAdams, ...and if you haven't been bitten yet by sloppy, buggy bash, I submit that it's only a matter of time if you write enough mission-critical code. At one former employer, the cost of people learning to pay attention to the corner cases was the loss of TBs of billing data. – Charles Duffy Dec 02 '15 at 18:50
1

Unlike the (too) trivial Bash slution, this one handles file names with newlines out of the box. A Bash solution with that capability will be at least as long as this Python code, and a lot more obscure. Meanwhile, this one is easy to adapt for more complex scenarios, with sane exception handling and error reporting. – tripleee Dec 02 '15 at 19:05
1

... Overall, I think Python strikes an excellent balance between control, brevity, and expressiveness for tasks like this, and rivals the shell even in areas where the shell *should* be excellent. And if you think I don't know Bash, I invite you to explore how I earned my reputation on this site. – tripleee Dec 02 '15 at 19:05
2

`newest = max(os.listdir(os.curdir), key=os.path.getmtime)`. Related: [How do you get a directory listing sorted by creation date in python?](http://stackoverflow.com/q/168409/4279) – jfs Dec 02 '15 at 19:27

python: trying to understand "subprocess" O/S calls

1 Answers1