How do I make a bazel `sh_binary` target depend on other binary targets?

Question

I have set up bazel to build a number of CLI tools that perform various database maintenance tasks. Each one is a py_binary or cc_binary target that is called from the command line with the path to some data file: it processes that file and stores the results in a database.

Now, I need to create a dependent package that contains data files and shell scripts that call these CLI tools to perform application-specific database operations.

However, there doesn't seem to be a way to depend on the existing py_binary or cc_binary targets from a new package that only contains sh_binary targets and data files. Trying to do so results in an error like:

ERROR: /workspace/shbin/BUILD.bazel:5:12: in deps attribute of sh_binary rule //shbin:run: py_binary rule '//pybin:counter' is misplaced here (expected sh_library)

Is there a way to call/depend on an existing bazel binary target from a shell script using sh_binary?

I have implemented a full example here: https://github.com/psigen/bazel-mixed-binaries

---

Notes:

I cannot use py_library and cc_library instead of py_binary and cc_binary. This is because (a) I need to call mixes of the two languages to process my data files and (b) these tools are from an upstream repository where they are already designed as CLI tools.

I also cannot put all the data files into the CLI tool packages -- there are multiple application-specific packages and they cannot be mixed.

Answer 1

You can either create a genrule to run these tools as part of the build, or create a sh_binary that depends on the tools via the data attribute and runs them them.

The genrule approach

This is the easier way and lets you run the tools as part of the build.

genrule(
    name = "foo",
    tools = [
        "//tool_a:py",
        "//tool_b:cc",
    ],
    srcs = [
        "//source:file1",
        ":file2",
    ],
    outs = [
        "output_file1",
        "output_file2",
    ],
    cmd = "$(location //tool_a:py) --input=$(location //source:file1) --output=$(location output_file1) && $(location //tool_b:cc) < $(location :file2) > $(location output_file2)",
)

The sh_binary approach

This is more complicated, but lets you run the sh_binary either as part of the build (if it is in a genrule.tools, similar to the previous approach) or after the build (from under bazel-bin).

In the sh_binary you have to data-depend on the tools:

sh_binary(
    name = "foo",
    srcs = ["my_shbin.sh"],
    data = [
        "//tool_a:py",
        "//tool_b:cc",
    ],
)

Then, in the sh_binary you have to use the so-called "Bash runfiles library" built into Bazel to look up the runtime-path of the binaries. This library's documentation is in its source file.

The idea is:

1. the sh_binary has to depend on a specific target
2. you have to copy-paste some boilerplate code to the top of the sh_binary (reason is described here)
3. then you can use the rlocation function to look up the runtime-path of the binaries

For example your my_shbin.sh may look like this:

#!/bin/bash
# --- begin runfiles.bash initialization ---
...
# --- end runfiles.bash initialization ---

path=$(rlocation "__main__/tool_a/py")
if [[ ! -f "${path:-}" ]]; then
  echo >&2 "ERROR: could not look up the Python tool path"
  exit 1
fi
$path --input=$1 --output=$2

The __main__ in the rlocation path argument is the name of the workspace. Since your WORKSPACE file does not have a "workspace" rule in, which would define the workspace's name, Bazel will use the default workspace name, which is __main__.

Answer 2

A clean way to do this is to use args and $(location):

Contents of BUILD:

py_binary(
    name = "counter",
    srcs = ["counter.py"],
    main = "counter.py",
)

sh_binary(
    name = "run",
    srcs = ["run.sh"],
    data = [":counter"],
    args = ["$(location :counter)"],
)

Contents of counter.py (your tool):

print("This is the counter tool.")

Contents of run.sh (your bash script):

#!/bin/bash
set -eEuo pipefail

counter="$1"
shift

echo "This is the bash script, about to call the counter tool."
"$counter"

And here's a demo showing the bash script calling the Python tool:

$ bazel run //example:run 2>/dev/null
This is the bash script, about to call the counter tool.
This is the counter tool.

It's also worth mentioning this note (from the docs):

The arguments are not passed when you run the target outside of bazel (for example, by manually executing the binary in bazel-bin/).

Answer 3

An easier approach for me is to add the cc_binary as a dependency in the data section. In prefix/BUILD

cc_binary(name = "foo", ...)
sh_test(name = "foo_test", srcs = ["foo_test.sh"], data = [":foo"])

Inside foo_test.sh, the working directory is different, so you need to find the right prefix for the binary

#! /usr/bin/env bash

executable=prefix/foo

$executable ...