Building installable tar.gz/whl with poetry using local dependency

Question

There is following structure in my python project:

├───pyproject.toml
└───mypackage
    │
    ├───lib
    │       localdep-0.2.0-py3-none-any.whl
    │       localdep-0.2.0.tar.gz
    └───service
            app.py
            home.py
            modules.py

I need to build mypackage using poetry and local dependency localdep from mypackage/lib/localdep-0.2.0... to be able to install mypackage just using simple pip install mypackage-0.1.0.tar.gz command without any additional files. I've tried to use path and file specifiers in pyproject.toml however I continuously get following error:

ERROR: Could not find a version that satisfies the requirement localdep(from mypackage==0.1.0) (from versions: none)

Current version of my pyproject.toml:

[build-system]
requires = [ "poetry>=0.12",]
build-backend = "poetry.masonry.api"

[tool.poetry]
name = "myproject"
version = "0.1.0"
description = "Simple demo project."
authors = ["Some Author"]
license = "MPL 2.0"

[tool.poetry.dependencies]
python = "3.7.3"
localdep = {file = "mypackage/lib/localdep-0.2.0-py3-none-any.whl"}

Does anyone know how to pass local dependency to pyproject.toml so poetry build will be able to package it in the correct way?

Answer 1

The use case of poetry's local dependency syntax is different from what you need, so it can't solve your problem here.

From the usage examples in those docs you can see that the path points to packages that are not part of the package itself, they always leave the root first, like this: ../some/different/location. The whole construct is only useful during development, and its logic will be run with poetry install, not poetry build.

What you want is to bundle the local dependency together with your project so that pip will know during a deployment of your project's .whl where to pull the local dependency from. But since pyproject.toml does not get packaged together with the wheel metadata, the information where to get dependencies from is no longer available, so it can't work. A built package only knows what dependencies it has, not where to get them from. This philosophy might be a bit unusual coming from other languages where it is not unusual to bundle all dependencies together with your code.

So even if you are able to build your package to include another wheel, which by default doesn't work because setuptools only includes .py files in the sdist/bdist by default, there is no way for pip to know that the dependency is reachable. I see four options to solve your problem in a way that python supports.

1. Use a version of your localdep that is on pyPI, or upload it there. But if that were a possibility, you would have probably not asked this question.
2. If localdep is under your control and is only used by mypackage, write it to be a simple submodule of mypackage instead - read, the initial decision to make localdep its own package was overengineering, which may or may not be true.
3. Use a way to vendor localdep that python understands. Take this guide for an in-depth explanation with all the considerations and pitfalls, or this hacky post if you want to get it to work without needing to really understand why or how.
4. Deploy your package as a "wheelhouse".

---

What is a wheelhouse?

The wheelhouse part needs a little bit more text, but it might be closest to what you initially had in mind. In this approach, you don't need to include localdep in your project, and the whole mypackage/lib folder should best be deleted. localdep only needs to be installed into your python interpreter, which you can ensure by running pip freeze and checking the output for localdep's name and version.

That same output will then be used to build said wheelhouse with pip wheel -w wheelhouse $(pip freeze). If you don't want to include dev dependencies, delete your current virtualenv and set it up and enter it before running the wheel command with poetry install; poetry shell again.

For completeness' sake, you can build dist/myproject.whl with poetry build and throw it in there as well, then you can just use this wheelhouse to install your package wherever you like by running python -m pip install wheelhouse/* with whichever python you want.

Why use pip and not poetry for that?

Poetry is a dependency manager for developing packages, as such it doesn't deal with deployment issues as much and only targets them tangentially. This is usually fine, because pip, as we saw, is quite capable in that regard. pip is not that convenient during development, which is why poetry is nice, but poetry does not completely replace pip.