How do I package non-code data with python setup.py

Question

I have a python project with the below structure

my-project
   venv
   tests
   setup.py
   abc
      __init__.py
      my_pckg
         __init__.py
         foo.py
      static
         data_0.txt
         sub_dir1
             data_1.yaml
         sub_dir2
             data_2.csv
             another file with space in name.pdf
   

Suppose in abc/foo.py I have

#abc/foo.py
import os
import yaml
with open(os.path.join("abc", "static", "sub_dir1", "data_1.yaml"), "r") as f:
  content= yaml.safe_load(stream)

if __name__ == "__main__":
  print(content)

Method1: It works as I expect If I open python console in my-project directory

from abc.my_pckg.foo import content
...

Method2: it also works as I expect if I run

python abc/foo.py

Method3: Now I want to package the code using setup.py file as below

import setuptools
import os


def package_files(directory):
    paths = []
    for (path, directories, filenames) in os.walk(directory):
        for filename in filenames:
            paths.append(os.path.join('..', path, filename))
    return paths


extra_files = package_files('abc/static')

setuptools.setup(
    name="some-name",
    version="0.1.0",
    package_dir={"abc": "abc"},
    include_package_data=True,
    package_data={'': extra_files},
    packages=[
        "abc",
        "bma.my_pckg",
    ],
    python_requires=">=3.6",
    install_requires=[],
)

I then generate whl file by runnig python setup.py bdist_wheel --dist=dist

then move the whl file to where I expect to install and run pip install ... to install it

new-project
  venv

However, when I try to it I get this error

  File "abs-path-to-new-poject\venv\lib\site-packages\abc\my_pckg\foo.py", line , in <module>
    with open(os.path.join("abc", "static", "sub_dir1", "data_1.txt"), "r") as f:
  FileNotFoundError: [Errno 2] No such file or directory: 'abc\\static\\data_1.yaml'\

How should I fix that? how should I package static data with python?

I want to be able to use the three methods above to run the code

Note: After installation, I have all the file under venv/Lib/site_packages/static The problem is the code I have in foo.py

And how do I deal with another file with space in name.pdf ?

Note: I am using the method suggested here https://stackoverflow.com/a/36693250/11065874

Further read:

• should I use pkg_resources? https://kiwidamien.github.io/making-a-python-package-vi-including-data-files.html

• should I use import_lib? how? https://dev.to/bowmanjd/easily-load-non-python-data-files-from-a-python-package-2e8g

Answer 1

I solved my problem by changing the code to

#abc/foo.py
import os
import yaml


_path = os.path.join(Path(__file__).parent.parent, "static", "sub_dir1", "data_1.yaml")

with open(_path, "r") as stream:
    content= yaml.safe_load(stream)


if __name__ == "__main__":
  print(content)