One entry point in WORKSPACE for managing various python project requirements

Question

My Bazel project structure is like this

├── BUILD
├── WORKSPACE
├── example.py
├── reqs.bzl
└── requirements.txt

I have created my WORKSPACE like this

load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

http_archive(
    name = "rules_python",
    sha256 = "a30abdfc7126d497a7698c29c46ea9901c6392d6ed315171a6df5ce433aa4502",
    strip_prefix = "rules_python-0.6.0",
    url = "https://github.com/bazelbuild/rules_python/archive/0.6.0.tar.gz",
)

load("reqs.bzl","load_reqs")
load_reqs()

I did that as I want to have one entry point in WORKSPACE from where I can manage dependencies for various python based projects. my reqs.bzl looks like this

load("@rules_python//python:pip.bzl", "pip_parse")
load("@rules_python//python:pip.bzl", "compile_pip_requirements")

def load_reqs():
    compile_pip_requirements(
        name = "pyreqs",
        requirements_in = "//:requirements.txt",
        visibility = ["/visbibility:public"]
    )

My BUILD file looks like

load("@pyreqs//:requirements.bzl", "requirement")

py_binary(
   name="example",
   srcs=["example.py"],
   deps=[
      requirement("absl-py"),
      requirement("numpy"),
      requirement("scipy"),
      requirement("matplotlib")
   ],
   visibility=["//visibility:public"]
)

This above setup is obviously failing to build my example.

I tried compile_pip_requirement as I couldnt pip_parse, load and invoke install_deps() from the load_reqs() function of reqs.bzl

Appreciate any input from the community on how this has to be done correctly.

PS: Still climbing that Bazel learning curve and trying to figure out best practices. The primary motivation behind above project setup is, I have to add lot of python targets and I don't want to have a large WORKSPACE file and frequently make edits to it.

requirements.txt

absl-py==1.4.0
cycler==0.11.0
kiwisolver==1.4.4
numpy==1.24.2
pillow
matplotlib==3.3.4
python-dateutil==2.8.2
pyparsing==3.0.9
pytz==2022.7.1
scipy==1.10.1
six==1.16.0
requests

bazel build //:example gives error

ERROR: Traceback (most recent call last):
        File "/home/ubuntu/bazel-tests/python-ex-2/WORKSPACE", line 11, column 10, in <toplevel>
                load_reqs()
        File "/home/ubuntu/bazel-tests/python-ex-2/reqs.bzl", line 5, column 29, in load_reqs
                compile_pip_requirements(
        File "/home/ubuntu/.cache/bazel/_bazel_ubuntu/7d424a81034067ed3c4b3321d48bbdb4/external/rules_python/python/pip_install/requirements.bzl", line 40, column 21, in compile_pip_requirements
                native.filegroup(
Error in filegroup: filegroup cannot be in the WORKSPACE file (used by //external:pyreqs)
ERROR: Error computing the main repository mapping: error loading package 'external': Package 'external' contains errors

Try clean and rebuild the project. make sure all the external dependencies are properly downloaded or not. Try adding --verbose_failures and --verbose_explanations to understand more clearly the logs — SG_Bazel, Mar 21 '23 at 07:13

score 0 · Answer 1 · answered Mar 24 '23 at 09:31

compile_pip_requirement is a rule that should be used in BUILD files, not WORKSPACE file.

Seems there is no way to load a .bzl file other than load(). new_local_repository doesn't work because it won't evaluate WORKSPACE in external workspace.

If you can accept one single load() and install_deps, maybe you can write a repository rule that merge all requirements.txt into one file, then call pip_parse, load() and install_deps once in your workspace root.

Otherwise, you can only figure out how pip_parse work and write your own.

One entry point in WORKSPACE for managing various python project requirements

1 Answers1