0

My BASH scrit has following line:

video_id=$(/usr/local/bin/yt-dlp --no-warnings --get-id "$ChannelPath")

it succeeds when runs from command line under certain user.

When it runs from cron, under the same user, it results in:

/usr/bin/env: python3: No such file or directory

What could be different in these two cases?

  • CentOS 7
  • Python 3.11.0a4
Paul
  • 25,812
  • 38
  • 124
  • 247

3 Answers3

1

There are some discussions whether you should use #!/usr/bin/env python or #!/usr/bin/python. Proponents of the env variant say that it is better, because env uses the PATH to find the interpreter, and opponents say that the problem with the env variant is that it uses the PATH.

I dislike env, therefore I will give you a solution without it.

From the command line type

which python3

This will give you something like

/usr/bin/python3

In /usr/local/bin/yt-dlp, replace the first line with

#!/usr/bin/python3

(the output that you got from the which command.)

Or call /usr/local/bin/yt-dlp with explicitly the right interpreter:

video_id=$(/usr/bin/python3 /usr/local/bin/yt-dlp --no-warnings --get-id "$ChannelPath")
Ljm Dullaart
  • 4,273
  • 2
  • 14
  • 31
  • If I put `/usr/local/bin/python3` (the output of `which python3`) to the first lineof the script, instead of `/bin/bash`, it corrupts the script execution even in normal (not cron) mode. Secondly - all other bash scripts with yt-dlp calls are running fine in cron. – Paul Sep 05 '22 at 13:44
  • `/usr/local/bin/yt-dlp`, not your bash script, sorry. – Ljm Dullaart Sep 05 '22 at 13:47
  • How? it is a compiled binary. And it is running fine from other scripts. – Paul Sep 05 '22 at 13:52
  • Hm. that is unexpected. What happens if you put `PATH="$PATH:/usr/bin"` (the directory where your python3 lives) in the script before calling `yt-dlp`? – Ljm Dullaart Sep 05 '22 at 13:58
  • I moved the `python3` symlink from `/usr/local/bin` to `/usr/bin` and it works now. – Paul Sep 05 '22 at 14:24
  • If you install the script properly (using `setuptools`, etc) a hashbang containing `python` will be set to whatever Python is used to install it, without have to *guess* where the end user has the correct Python installed. – chepner Sep 05 '22 at 14:27
  • How can you override what the shebang points to? I have a Cygwin environment where I don't have Python3 installed so scripts with the `/usr/bin/env python3` fails. But the python interpreter is in the path. – not2qubit Aug 03 '23 at 19:04
  • If you have a question on how to access Python in Cygwin, ask a new question. Don't try to "revive" an old question with a comment that has little to do with the original question. – Ljm Dullaart Aug 05 '23 at 14:04
1

it succeeds when runs from command line under certain user.

When it runs from cron, under the same user, it results in:

/usr/bin/env: python3: No such file or directory

What could be different in these two cases?

The difference is the environment in which the script runs. In particular, the environment variables in it, and most particularly, the PATH.

When a user runs the script from the command line, it inherits the environment from which it was launched, which includes system-wide and possibly user-specific customizations that are engaged only for interactive shells. (For example, the contents of the user's ~/.bash_profile and / or ~/.bashrc files.) By default, when a shell is launched noninteractively (by cron, for example) it does not read or execute any environment configuration.

Evidently,

  1. Your /usr/local/bin/yt-dlp is or attempts to launch a Python script that has a shebang line using /usr/bin/env to choose and launch the python3 binary. That is, the affected script starts with

    #!/usr/bin/env python3

    These days, that form is widely used and recommended in the Python world. The purpose is to use the PATH to locate the python3 binary to use, as opposed to hard-coding that into the script. However,

  2. There is no system Python 3 installed on the machine. (That is, none installed in the default path.) The user who runs the script successfully is able to do so because they have an environment configured with some non-default directory in their PATH from which python3 can be launched.

One possible solution would be to install CentOS's Python 3:

sudo yum install python3

If you need a different version of Python 3 (CentOS 7's is version 3.6) then you can instead set an appropriate PATH in the relevant crontab file, maybe something like

PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
0 * * * * /path/to/my_script

Alternatively, you could modify your shell script to set the path there:

PATH=/usr/local/bin:$PATH
video_id=$(/usr/local/bin/yt-dlp --no-warnings --get-id "$ChannelPath")

or you could modify /usr/local/bin/yt-dip by altering its shebang (supposing that this is the affected Python script).

John Bollinger
  • 160,171
  • 8
  • 81
  • 157
  • I moved the `python3` symlink from `/usr/local/bin` to `/usr/bin` and it works now. – Paul Sep 05 '22 at 14:22
  • @Paul, I'm glad you have it working, but that particular solution is not one I would recommend. You may run into issues if later you install a CentOS package that depends on CentOS's python3 package (resulting in that one replacing the one you put in `/usr/bin`). If you can use CentOS's, then that's what you should do. If not, you should use a mechanism, such as one of those I suggested, to tell the script where to find the right Python. – John Bollinger Sep 05 '22 at 14:28
  • Last time I updated `yt-dlp` binary (it is not a script), it threw an error, that it won't work with Python 3.6 any more. That's why I had to remove 3.6 and started this complicated quest with installing Python 3.11. It is unlikely that I will be forced to install Python 3.6 again. Hopefully 3.11 is backwards compatible with 3.6. – Paul Sep 05 '22 at 14:52
  • cron seems to ignore crontab. there is `/sbin:/bin:/usr/sbin:/usr/bin` defined in `/etc/crontab`, but the real PATH is outputted like `/usr/bin:/bin`. The real crontab seems to be in `/var/spool/cron/$USERNAME` – Paul Sep 05 '22 at 15:01
0

I moved the python3 symlink from /usr/local/bin to /usr/bin and it works now.

/usr/local/bin was the suggested location in internet-manuals about how to compile and install Python 3.11 in CentOS7.

A better solution is to add

PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin

to the first line of crontab. Although it should be not /etc/crontab, but

crontab -l > crontab.txt

then edit crontab.txt

then

crontab crontab.txt
Paul
  • 25,812
  • 38
  • 124
  • 247
  • 2
    You should absolutey leave `/usr/bin` alone; this location is reserved for system binaries supplied by your OS on modern Linux distros. The simple and straightforward solution is to add `/usr/local/bin` to your `PATH`. – tripleee Sep 06 '22 at 13:29