2

I have a Python package that uses data files stored in the src/data directory. I can successfully install the package using pip install . but using pip install -e . does not work because the data folder is not found. How can I perform an editable install of this package?

mypackage
├── README.md
├── examples
│   └── example.py
├── pyproject.toml
├── src
│   ├── data
│   │   └── fruits.csv
│   └── mypackage
│       ├── __init__.py
│       ├── adder.py
│       └── fruit_reader.py
└── tests
    ├── test_adder.py
    └── test_read_fruits.py

The contents of the pyproject.toml file is shown below. Notice the src/data folder is included as mypackage/data during the build process.

# pyproject.toml

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[project]
name = "mypackage"
version = "0.1"
authors = [
  { name="Example Author", email="author@example.com" },
]
description = "A small example package"
readme = "README.md"
requires-python = ">=3.9"
classifiers = [
  "Programming Language :: Python :: 3",
  "License :: OSI Approved :: MIT License",
  "Operating System :: OS Independent",
]

[tool.hatch.build.force-include]
"src/data" = "mypackage/data"

The read_fruits() function reads the CSV file located in the data folder as shown here.

# fruit_reader.py

import pandas as pd
from pathlib import Path

def read_fruits():
    """
    Read fruit data from CSV file.
    """
    path = Path(__file__).parent / 'data/fruits.csv'
    print(f'File path is\n{path}')

    df = pd.read_csv(path)
    print(f'Pandas dataframe is\n{df}')
wigging
  • 8,492
  • 12
  • 75
  • 117
  • Use `importlib.resources` instead of `__path__` when reading package data. -- Anyway; this is a question you should ask to hatch maintainers directly. -- Editable support (PEP660) is kind of new (so is hatch itself actually), so expect some rough edges in non typical use cases. And your case seems definitley non-typical because you want editable support with a path rewrite, I believe this is hard to implement; it never worked in pre-PEP660 setuptools for example. – sinoroc Aug 24 '22 at 09:29
  • @sinoroc Can you submit an answer that demonstrates using `importlib.resources`? Also, if you have a better project structure that I should use for my example then please demonstrate that in your answer. I am also open to using setuptools instead of hatch if you think it would work better for my example. – wigging Aug 24 '22 at 13:23
  • I recommend [this answer about reading package data](https://stackoverflow.com/a/58941536) which exposes multiple techniques with pros and cons. If I understood correctly `pkgutil` will get deprecated soon-ish (no panic, it will most likely stay a valid solution for quite some time still), that is why I rather recommend `importlib.resources`. -- I am not sure migrating to _setuptools_ would be obviously better. I think the "path rewrite in editable install" could be an issue with any build back-end. If it is important for you, then try multiple back-ends until you find one that fits. – sinoroc Aug 24 '22 at 18:48

0 Answers0