Running Embedded Python in c++ using vcpkg to install

Question

I do not understand how to install python through vcpkg. The CMakesLists file I have is:

cmake_minimum_required(VERSION 3.8)

if(NOT DEFINED CMAKE_TOOLCHAIN_FILE)
set(CMAKE_TOOLCHAIN_FILE "./vcpkg/scripts/buildsystems/vcpkg.cmake"
    CACHE STRING "")
endif()

project(test)

find_package(Python3 COMPONENTS Development REQUIRED)

add_executable(test test.cpp)
target_link_libraries(test PRIVATE Python3::Python)

The c++ file was taken directly from https://docs.python.org/3/extending/embedding.html#very-high-level-embedding

everything compiles and links without error but when executing it gives this error:

Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = (not set)
  program name = 'C:\Users\smith\OneDrive\Desktop\OSU\vcpkg-python-test\build\Debug\test.exe'
  isolated = 0
  environment = 1
  user site = 1
  import site = 1
  sys._base_executable = 'C:\\Users\\smith\\OneDrive\\Desktop\\OSU\\vcpkg-python-test\\build\\Debug\\test.exe'
  sys.base_prefix = ''
  sys.base_exec_prefix = ''
  sys.platlibdir = 'lib'
  sys.executable = 'C:\\Users\\smith\\OneDrive\\Desktop\\OSU\\vcpkg-python-test\\build\\Debug\\test.exe'      
  sys.prefix = ''
  sys.exec_prefix = ''
  sys.path = [
    'C:\\Users\\smith\\OneDrive\\Desktop\\OSU\\vcpkg-python-test\\build\\Debug\\python310_d.zip',
    '.\\DLLs',
    '.\\lib',
    'C:\\Users\\smith\\OneDrive\\Desktop\\OSU\\vcpkg-python-test\\build\\Debug',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'
    
Current thread 0x000062bc (most recent call first):
  <no Python frame>

I don't understand what I am doing wrong or if there was some step I missed.

The full example code can be found at https://github.com/Drew-j-Smith/vcpkg-python-test

This was run on windows 10

edit:

Adding this to the CMakeLists.txt (after finding python):

get_filename_component(Python3_EXECUTABLE_DIR ${Python3_EXECUTABLE} DIRECTORY)
add_definitions(-DPYTHON_LIB=L"${Python3_EXECUTABLE_DIR}/Lib")

And calling

Py_SetPath(PYTHON_LIB);

before python is initialized does fix the issue. However, this seems like a hack to me. I am still looking for a robust solution.

I solved python cross-platform embedding in my company with Miniconda/CMake/CLion. It is not ideal, but it gets you there. — bobah, Mar 08 '22 at 16:35
https://stackoverflow.com/questions/65184937/fatal-python-error-init-fs-encoding-failed-to-get-the-python-codec-of-the-file https://stackoverflow.com/questions/5694706/py-initialize-fails-unable-to-load-the-file-system-codec — Alan Birtles, Mar 08 '22 at 16:43

Drew Smith · Answer 1 · 2022-03-10T02:11:53.717

After a lot of trial and error, the solution is to create a virtual environment in the parent directory to the executable.

This behavior seems odd to me but is documented here: https://docs.python.org/3/c-api/intro.html#embedding-python

Example solution to this is to add this to the end of the CMakeLists.txt

if(WIN32)
    add_custom_command(TARGET test POST_BUILD 
    COMMAND ${Python3_EXECUTABLE} -m venv $<TARGET_FILE_DIR:test>/..
    COMMAND $<TARGET_FILE_DIR:test>/../Scripts/activate.bat
    COMMAND python -m pip install -r ${CMAKE_SOURCE_DIR}/requirements.txt > pip_install.log)
else()
    add_custom_command(TARGET test POST_BUILD 
    COMMAND ${Python3_EXECUTABLE} -m venv $<TARGET_FILE_DIR:test>/..
    COMMAND $<TARGET_FILE_DIR:test>/../Scripts/activate
    COMMAND python -m pip install -r ${CMAKE_SOURCE_DIR}/requirements.txt > pip_install.log)
endif()

This will install any packages required and create a virtual environment for the executable so that it knows the path of python.

edit:

I had some issue running in debug mode. When trying to import numpy I had to use release mode due to a missing debug binary.

Running Embedded Python in c++ using vcpkg to install

1 Answers1