I'm unable to build CPython from scratch on Linux because of inability to find encodings Module. What am I missing?

Question

I have an old Centos 7 system. It has one user [plus root] and has python2.7 and python3.6.8 installed. The latter has binary /bin/python3 Neither PYTHONHOME nor PYTHONPATH are defined. Python3 runs fine, for example, print(encodings.__file__) gives usr/lib64/python3.6/encodings/__init__.py but I need to upgrade it. There do not appear to be modern repositories for Centos 7 now, so I'll build by scratch. No problem....

So I git clone Python.3.11.2 (also tried 3.11.0a4) and tar xvf to create a directory and cd to that directory. The following fails

./configure
make

with an error

Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

The error is hit when trying to run _bootstrap_python. I've tried solving this a bunch of different ways. For example, trying a prefix gives this printout:

./_bootstrap_python ./Programs/_freeze_module.py abc ./Lib/abc.py Python/frozen_modules/abc.h
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = (not set)
  program name = './_bootstrap_python'
  isolated = 0
  environment = 0
  user site = 0
  safe_path = 0
  import site = 1
  is in build tree = 0
  stdlib dir = '/usr/local/lib/python3.11'
  sys._base_executable = '/bin/python3'
  sys.base_prefix = '/usr/local'
  sys.base_exec_prefix = '/usr/local'
  sys.platlibdir = 'lib'
  sys.executable = '/home/xuser/analysis/Python-3.11.2/_bootstrap_python'
  sys.prefix = '/usr/local'
  sys.exec_prefix = '/usr/local'
  sys.path = [
    '/usr/local/lib/python311.zip',
    '/usr/local/lib/python3.11',
    '/usr/local/lib/python3.11/lib-dynload',
  ]

I've tried running the configure inside a virtualenv. I've run it as xuser. I've run it as root.... It keeps stumbling on the same point. The download of 3.11 has perfectly fine encodings sitting inside the its Lib directory, but I think they are not getting copied over to the destination where makefile expects them.

Doubtless a dumb mistake of mine, but damned if I can find it. I did see some old discussions on stackoverflow to change the contents of .ini file (I dont have an .ini file) and/or to edit PYTHONPATH to something else. [No matter what I set that path to, it still shows up as "not set" in the configuration listed by bootstrap, so I dont think that is the solution.] My guess is a definition needs to be passed to .configure. But what?

Looking at that print out, I see that sys._base_executable is pointing at 3.6.8, which is suspicious (why does it care about old python?)

Just tried ```make clean``` before and/or after ./configure. It didn't help. — Tunneller, Mar 21 '23 at 13:41
Here's a related question with a Windows solution that does not seem to be particular well accepted by the community (change PYTHONPATH). Either way, I don't have windows and changing the Path environment variables does nothing when trying to build from scratch https://stackoverflow.com/questions/69550830/init-fs-encoding-failed-to-get-the-python-codec-of-the-filesystem-encoding — Tunneller, Mar 22 '23 at 03:04
Haha: don't downvote this comment: it's kind of a joke. But in the debugger, I see that I'm being passed "ANSI_X3..." so in config_get_codec_name() all you have to do is set ```*config_encoding = "ascii"; return 0;``` and all the problems of finding Encoding module go away. This [of course] is not actually going to help anything, but at least gets me closer. — Tunneller, Mar 22 '23 at 04:09
Update. If I run configure and then manually edit the Makefile to force PYTHONPATH to point to the Lib that came with the github download then it makes a lot of progress. Currently stuck on an inability to find ```_sysconfigdata__linux_x86_64-linux-gnu``` — Tunneller, Mar 22 '23 at 14:24
I copied sysconfig... over to Lib (why not) and also copied all the include files over to their wanted destination on /usr/local/... and *voila* python built... Woohoo. But it's all screwed up [as most of you could have probably told me to anticipated] , cannot initialize because its getting confused where the Modules should be. My current thoughts are to persuade configure to build a sys.path that includes the destination directories, but i can work out how to do that yet. — Tunneller, Mar 22 '23 at 19:50

score 0 · Answer 1 · answered Mar 23 '23 at 04:47

Ok, this is ugly. But it resulted in a working 3.11 environment on Centos 7 when I couldn't get anything else to work. I'll continue to sort out why .configure is screwing up, but here goes....

When building from a really old Python to 3.11, somehow the sys.path chosen by .configure is always wrong. Changing PYTHONHOME or PYTHONPATH doesn't help, I don't know why.

But you can fix it by hand by editing sysmodule.c (!!) Look for COPY_LIST("path"...) and immediately before there you can add the search paths you need.

   PyWideStringList_Append(&config->module_search_paths, L"/home/xuser/analysis/Python-3.11.0a4/Lib");
   PyWideStringList_Append(&config->module_search_paths,
                L"/home/xuser/analysis/Python-3.11.0a4/build/lib.linux-x86_64-3.11");
    
    if (config->module_search_paths_set) {
        COPY_LIST("path", config->module_search_paths);
    }

Now just press "make" to get a fully functioning, modern python. You should leave PYTHONHOME and PYTHONPATH blank because the link to the modules was hardcoded at compile time.

My guess is that any future upgrade will be by the book and not require this hack, but I'm posting in case anybody in the same boat. And yes, I'm sure there is a flag for .configure or Make on Centos7too. If nothing else, this hack should clarify what is needed.

The default configure will not compile ```_ctypes```, etc. It will try ```gcc -c _ctypes.c``` instead of ```gcc -c Modules/_ctypes.c```. This behaviour is documented in the Modules/Setup file and explains the fix. Namely that I also needed to run this command ```sed -n -E 's/^#([a-z_\*].*)$/\1/p' Modules/Setup > Modules/Setup.local``` before configure. — Tunneller, Mar 23 '23 at 14:48

score 0 · Answer 2 · answered Mar 23 '23 at 17:45

And now this works: so I can build 3.11.2 without any screwing around. [Thanks to a post by Lou King on Google Groups for advice on this].

./configure '--enable-ipv6' '--enable-shared' '--with-computed-gotos=yes' 
'--with-dbmliborder=gdbm:ndbm:bdb' '--with-system-expat' '--with-system-ffi' 
'--enable-loadable-sqlite-extensions' '--with-dtrace' 
'--with-ssl-default-suites=openssl' '--without-ensurepip' 
'--enable-optimizations' 'CFLAGS=-O2 -g -pipe -Wall 
-Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong 
--param=ssp-buffer-size=4 -grecord-gcc-switches   -m64 
-mtune=generic -D_GNU_SOURCE -fPIC -fwrapv  ' 
'LDFLAGS=-Wl,-z,relro  -g  ' 
'CPPFLAGS= ' 'PKG_CONFIG_PATH=:/usr/lib64/pkgconfig:/usr/share/pkgconfig'

I'm unable to build CPython from scratch on Linux because of inability to find encodings Module. What am I missing?

2 Answers2