Python shell script cannot be executed with source command

Question

I want to call a bash script from a python script using subprocess.Popen(). Calling the shell script as an executable works, but sourceing it does not. Why?

File test_python.py:

import sys
import os
import subprocess

os.putenv("testvar", "testvalue")

test = subprocess.Popen("./test_shell.sh", shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()
print(test)
test = subprocess.Popen(". test_shell.sh", shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()
print(test)
test = subprocess.Popen("source test_shell.sh", shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()
print(test)
test = subprocess.Popen("/bin/bash test_shell.sh", shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()
print(test)
test = subprocess.Popen("/bin/sh test_shell.sh", shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()
print(test)

File test_shell.sh:

#!/bin/bash

echo "$testvar"

Output of python test_python.py:

('testvalue\n', '')
('', '/bin/sh: .: test_shell.sh: cannot open [No such file or directory]\n')
('', '/bin/sh: .: test_shell.sh: cannot open [No such file or directory]\n')
('testvalue\n', '')
('testvalue\n', '')

@Paolo: The larger Python script I'm using this for is a translation of a (main) shell script calling several shell subscripts via `source` such that any variables set in the main script were also available in the subscripts. Of course, as far as I understand, this is not the case for shell scripts called from the main python script anyways, so `./` should in principle be fine here as well. — bproxauf, Jul 17 '23 at 11:22
@Verpous: `subprocess.Popen(". test_shell.sh", shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()` appears to give the same stderr with `.` as with `source`. — bproxauf, Jul 17 '23 at 11:29
https://stackoverflow.com/questions/7040592/calling-the-source-command-from-subprocess-popen — JSN, Jul 17 '23 at 11:43
`source` is inherently useless from Python, since the entire point of `source` is to run a script in the current process rather than a subprocess, but when the current process is Python rather than a shell, that's not possible. To put it another way, `subprocess.Popen()` has already created a subprocess to run the script, so trying to avoid running the script in a subprocess is just not going to happen. On the other hand, if the data flow is entirely Python -> shell scripts, why not create environment variables, since subprocesses inherit copies of them? — Gordon Davisson, Jul 17 '23 at 22:45

score 1 · Answer 1 · answered Jul 17 '23 at 11:42

1

Calling the shell script as an executable works, sourceing it does not.

You are calling ./test_shell.sh. Yet, you source test_shell.sh.

Why?

Because there is no such script in PATH

If you want to source ./test_shell.sh then source ./test_shell.sh not source test_shell.sh.

answered Jul 17 '23 at 11:42

KamilCuk

120,984
8
59
111

The explanation is correct, of course, but I find confusing, that in the error message, we see that in both offending cases, the error is reported by `sh`. It is clear that `sh` is executed here (since no explicit shell has been provided in the Python code), but with POSIX shell, the `source` command should be rejected. AFIK, POSIX only defines the command `.`, but not `source`, which is a bash extension. – user1934428 Jul 17 '23 at 14:46
`the source command should be rejected` I disagree. A POSIX compatible shell _is not required_ to a have a source builtin. This does _not_ mean it is required to reject it. A POSIX compatible shell having a source builtin is still perfectly POSIX compatible. – KamilCuk Jul 17 '23 at 14:50
I find these inconsistencies between "standard"-implementations confusing. Do you know by chance, which POSIX shell accepts `source`? I tried `dash` and it rejects it. – user1934428 Jul 17 '23 at 14:53
It's hard to answer that, there are millions of projects on github and there are endless programs that state to be "POSIX compatible shell". There is no like "official" certification body. POSIX is a standard that states what has to be there _to be_ POSIX compatible, it is not a limitation of what exists. There exists what exists - most commonly bash, dash and busybox. If OP system has bash, and sh is a symlink to bash, then sh has source. Or OP has written his own sh that is not POSIX compatible, and still has source (we assume he didn't). I do not know how to answer. – KamilCuk Jul 17 '23 at 14:56
Bottom line, I do not know which shell to judge to be "POSIX". Bash shell states in the documentation to be POSIX compatible, I do not know if it has received any kind of certification, and there have been differences and bug reports. If we judge bash to be a POSIX shell, then bash has source, then the answer is bash. Busybox also has source. Sooo it is way more fragmented. Also, the question is tagged with bash, so we can assume it. – KamilCuk Jul 17 '23 at 15:05
I get the argument. We just see from the error message, that it does not come from bash, but from sh. Of course if this is a system where sh is a symlink to bash, it would be bash which effectively is executed, and if bash in POSIX mode (i.e. when called as sh) still supports `source`, this could explain the observation. – user1934428 Jul 17 '23 at 16:57
@user1934428 POSIX specifies a minimum set of features that a shell must support to be considered compliant; most POSIX shells *also* support extensions beyond the basics that POSIX requires. `source` is such an extension (see [this](https://wiki.ubuntu.com/DashAsBinSh), especially section 3, for some other common ones). I think `source` originated in ksh, and from there was copied by bash and zsh. BTW, zsh is *not* POSIX-compliant by default, since it disables several POSIX-mandated features that've turned out to be more trouble than they're worth. – Gordon Davisson Jul 17 '23 at 22:39

Python shell script cannot be executed with source command

1 Answers1