0

I want to locally install a package from one project in another project. I am able to install the package and import the function I want to call. The function loads a pickle file. When I call the function it fails with a FileNoutFoundError.

I have followed this answer and use include_package_data=True and MANIFEST.in but that does not resolve my error.

I have a simple project structured like this:

.
├── example_pkg
│   ├── foobar.py
│   ├── __init__.py
│   └── pickle_files
│       └── test.p
├── MANIFEST.in
├── __pycache__
└── setup.py

MANIFEST.in content:

include example_pkg/pickle_files/*

foobar.py:

import os
import pickle


def out():
    with open(os.path.join("pickle_files", "test.p"), "wb") as f:
        pickle.dump({"hello": "world"}, f)


def f_in():
    with open(os.path.join("pickle_files", "test.p"), "rb") as f:
        d = pickle.load(f)
    print(d["hello"])

test.p is created by calling out(). I do not want to call it after installing the package in another project to create the file again, it is just here to demonstrate where the file comes from.

setup.py looks like this:

import setuptools

setuptools.setup(
    name="example-pkg-foo",
    version="0.0.1",
    packages=setuptools.find_packages(),
    include_package_data=True,
)

init.py is blank.

I then run python setup.py sdist which creates a folder dist with a file example-pkg-foo-0.0.1.tar.gz in it. When I manually inspect this tarball I can see that example_pkg/pickle_files/test.p exists.

--- Everything above this line is part of project A. Everything below is part of project B. ---

I then copy+paste the tarball to my other project (project B) and run pip install example*. The package is then installed in my project's (project B) virtual environment.

Then I run the following script using the virtual environment of project B:

from example_pkg.foobar import f_in

f_in()

I can import the package and call the function. However, it fails with 

Traceback (most recent call last):
  File "/home/madjura/PycharmProjects/package_experiment/foo.py", line 3, in <module>
    f_in()
  File "/home/madjura/PycharmProjects/package_experiment/venv/lib/python3.7/site-packages/example_pkg/foobar.py", line 11, in f_in
    with open(os.path.join("pickle_files", "test.p"), "rb") as f:
FileNotFoundError: [Errno 2] No such file or directory: 'pickle_files/test.p'

When I inspect the venv (of project B) I can see that the file was included in the installation:

.
└── site-packages
    ├── example_pkg
    │   ├── foobar.py
    │   ├── __init__.py
    │   ├── pickle_files
    │   │   └── test.p  <------ it is included in the venv
    │   └── __pycache__
    │       ├── foobar.cpython-37.pyc
    │       └── __init__.cpython-37.pyc
    └── example_pkg_foo-0.0.1-py3.7.egg-info
        ├── dependency_links.txt
        ├── installed-files.txt
        ├── PKG-INFO
        ├── SOURCES.txt
        └── top_level.txt

My Python version is 3.7.3. I want to install a package with a pickle file locally in another project, without creating a new pickle file. I want to use the one that already exists. How can I fix the FileNotFoundError when I call f_in()?

Lomtrur
  • 1,703
  • 2
  • 19
  • 35

1 Answers1

1

change foobar.py to:

import os
import pickle


def out():
    with open(os.path.join(os.path.dirname(__file__), "pickle_files", "test.p"), "wb") as f:
        pickle.dump({"hello": "world"}, f)


def f_in():
    with open(os.path.join(os.path.dirname(__file__), "pickle_files", "test.p"), "rb") as f:
        d = pickle.load(f)
    print(d["hello"])

when you launch it from other place you have different starting point for decoding path.

Grzegorz Bokota
  • 1,736
  • 11
  • 19