-1

I have a Python (3.9) script that exists in the following file structure:

L usr
  L src
    L project_root
      L folder_a
        L folder_b
          L foo.py
          L bar.sh

foo.py contains this line of code, which calls a subprocess bar.sh and passes some data to it:

subprocess.call(['/usr/src/folder_a/folder_b/bar.sh', f'{some_data}'])

This subprocess call finds the shell script just fine when run locally.

However, when I run this same code in a docker container, I get the following exception:

Exception: [Errno 2] No such file or directory: '/usr/src/folder_a/folder_b/bar.sh'

Troubleshooting steps I have tried:

  1. I confirmed that the path is correct, both spelling and location
  2. The docker container's top-level is the project_root folder
  3. I have tried both the relative and exact paths, neither work
  4. I have SSH'd into the container and checked the files, the shell script is present in the exact directory that I provided.
  5. I have tried using os.path.abspath() to generate the absolute path to the shell script, but it was still not found.
  6. I have checked os.cwd to confirm that the current working directory is usr/src
  7. I have used Path(__file__).parent('./bar.sh') to find the absolute path to the shell script, which just gave me the string /usr/src/folder_a/folder_b/bar.sh, the same as I've been using.

Any ideas?

Cameron Gould
  • 58
  • 7
  • You didn't actually run that line of code. The error message clearly shows a relative path and that is the source of your problem. – tdelaney Oct 01 '21 at 19:13
  • @tdelaney forgot to update the error output to show the same result for the absolute path. The error there was listed from when I *did* try using a relative path and it didn't work. I did run it with an absolute path, probably 50 times, nothing worked. Via documentation, the problem is with the lack of assigning kwarg `shell=True`. That fixed the issue immediately. – Cameron Gould Oct 01 '21 at 20:11
  • Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. – Community Oct 08 '21 at 11:16

1 Answers1

-2

After looking into the documentation for the subprocess module, it looks like there are a couple things wrong with my original approach.

  1. The documentation states that for Python 3.5 or higher, use subprocess.run() and not subprocess.call(), as the call function is only still there for backwards compatibility:

The recommended approach to invoking subprocesses is to use the run() function for all use cases it can handle. For more advanced use cases, the underlying Popen interface can be used directly.

The run() function was added in Python 3.5; if you need to retain compatibility with older versions, see the Older high-level API section.

  1. The key is to pass the value True to kwarg shell:
subprocess.run(['/usr/src/folder_a/folder_b/bar.sh', f'{some_data}'], shell=True)

The shell kwarg is, by default, set to False. When trying to execute a shell script (or possibly any script that is a file rather than a command line tool) you can to set the shell kwarg to True to have the list of arguments passed as a sting to run through shell. For some reason, not doing this will result in a FileNotFoundError, which in my opinion is a very confusing and misleading error output...

One thing to also note is that the subprocess will run asynchronously, so there is some error trapping to be done. I found some information about checking if a subprocess is still running in another StackOverflow question.

Cameron Gould
  • 58
  • 7
  • 1
    This not not what solved the problem. By default _both_ `call` and `run` take a list and do not run through the shell. They _both_ have a `shell=True` parameter which does run through the shell. In your case, the list is converted to a string and then passed to the shell. There really isn't a good reason to do this, the list without `shell=True` is fine. – tdelaney Oct 01 '21 at 19:15
  • @tdelaney without `shell=True` this doesn't work in the docker container. I suggest throwing some python code into a docker container and trying it yourself. I have tried finding the path in many different ways--it still cannot find the file. Might be a docker issue, might be a subprocess issue with docker. Regardless, it simply does not work in a docker container. I had to refactor code that worked locally in order to get it to work in the container. I would love if it *did* work as you say, but unfortunately it doesn't. – Cameron Gould Oct 01 '21 at 20:13
  • Is your docker on linux? I've assumed its not Windows based on the path. Lets hard code "some_data" for an example: `subprocess.run(['/usr/src/folder_a/folder_b/bar.sh', 'some_data'], shell=True)` becomes the command: `/bin/sh -c /usr/src/folder_a/folder_b/bar.sh some_data`. The -c option runs `bar.sh` in a subshell without additional arguments. No "some_data". "some_data" is just the `$)` param for the interim shell. This code should fail on non-windows systems because the parameter is lost. – tdelaney Oct 02 '21 at 04:28
  • @tdelaney I think I see where the miscommunication is. The path is generated from within the docker container, which is a Linux environment. However, the machine I am using is running Windows. The absolute path is different when running locally; the one I've provided is the absolute path from within the docker container, hence why it looks like a linux path and not a Windows one. But I've tested this running in the docker container and it ran the subprocess without any issues... very confusing. – Cameron Gould Oct 02 '21 at 05:32